When VLMs Trust the Prompt Over the Image

This paper is about a failure mode that feels familiar if you have ever used a vision–language model: the model can see the image correctly, but still follow a wrong prompt.

Ask "How many waterlilies are in the image?" and it may answer three. Ask "Describe the five waterlilies in the image" and it may invent two more flowers that are not there. The paper calls this prompt-induced hallucination (PIH).

The Paper

Paper: Mechanisms of Prompt-Induced Hallucination in Vision–Language Models (Rudman, Golovanevsky, Arad, Belinkov, Singh, Eickhoff, and Mahowald, ACL 2026)

Venue: ACL 2026 (Long Papers)

Topic: Vision–language models, modality conflict, mechanistic interpretability, attention-head ablation.

Code: github.com/michalg04/prompt-induced_hallucinations

The Problem and Why It Matters

VLMs are often better at text than at vision. When the prompt and the image disagree, many models prefer the text. That is a real deployment problem, because user prompts are noisy. People miscount, mistype colors, or assume objects that are not in the frame.

The paper isolates this with a clean counting setup on CountBench. First they keep only images where the model already answers the baseline question correctly. Then they ask the model to describe N+k objects when only N are present. If the model describes the prompted count instead of the true count, that is PIH.

Figure 1 from Rudman et al. (ACL 2026): example of prompt-induced hallucination. The model follows an overstated waterlily count for moderate mismatches, then recovers the true count when the mismatch becomes large.

The pattern is consistent across LLaVA-OneVision, Qwen2-VL, and Janus-Pro. With few objects, models often correct the prompt. Once the true count goes past about four, they start conforming to the prompt even when the offset is large. Prompt-match rates climb toward 80–90%, and true-count matches fall toward zero.

Figure 2 from Rudman et al. (ACL 2026): PIH rates across ground-truth object counts for Janus-Pro, Qwen2-VL, and LLaVA-OneVision. Blue curves are prompt matches; orange curves are true-count matches.

This is not just "the model cannot count." The same models succeed on the baseline counting question. The failure is about which signal wins when text and image conflict.

The Method

The authors treat PIH as a mechanistic question: which attention heads push the model to copy the prompt?

They use mean ablation, knocking out one head at a time and measuring how often a misaligned response flips from the prompted count to the true count. They rank heads by that success rate, then ablate the top set together. The final sets are small: 3 heads for Qwen-VL and 10 for LLaVA-OneVision and Janus-Pro.

Most of the strong PIH heads sit in early language-model layers. Qwen-VL and LLaVA-OneVision also share several of the same heads, which points to the shared language backbone rather than the vision tower.

They also check that the intervention is not just breaking the model. Baseline counting stays stable or even improves a little for LLaVA-OneVision. Broader benchmarks like CalTech101, MM-Vet, and POPE barely move.

Table 1 from Rudman et al. (ACL 2026): response accuracies before and after ablating PIH heads versus random heads. Prompt match drops sharply after PIH ablation, while true-count match rises.

The numbers are striking. Prompt match falls from the 40–60% range to below 11% across models. True-count match rises into the 70–78% range. Random heads from the same layers do almost nothing.

What Is Novel

The counting setup itself is simple. The novel part is the causal story.

First, PIH is localized. A small set of attention heads mediates a large share of the prompt-copying behavior.

Second, the same heads generalize. Ablating the counting-derived PIH heads also reduces prompt-color copying on a Visual CounterFact color task, by as much as about 94% for LLaVA-OneVision.

Third, the paper does not stop at "ablation helps." It asks what the heads are doing. After knockout, models copy the prompt less, attend more to image tokens, and shift probability mass toward visually grounded answers. The details differ by model:

LLaVA-OneVision suppresses copying broadly and shows the largest increase in image attention.
Janus-Pro mainly reduces format copying and answers in freer word form.
Qwen-VL still uses prompt format more often when it corrects, but becomes more visually grounded overall.

Figure 4 from Rudman et al. (ACL 2026): image attention mass before and after PIH knockout. All three models increase attention to image tokens, with the largest gain in LLaVA-OneVision.

I also like the confidence analysis. As object counts rise, base probability on the true count falls and mass spreads to neighbors. That lines up with the PIH curves: when visual confidence is weak, the prompt wins more often.

Figure 3 from Rudman et al. (ACL 2026): base probabilities on the counting task for Qwen-VL. Confidence in the true count drops as N increases, and probability mass spreads to neighboring counts.

Reproduction Plans

I have not reproduced the paper yet. A first pass could stay close to their counting setup.

Models: one open 7B VLM, ideally LLaVA-OneVision or Qwen2-VL
Data: CountBench images where the baseline count is already correct
Prompts: baseline "How many …?" versus "Describe the N+k … in the image"
Metrics: prompt match, true-count match, and baseline exact match
Intervention: mean-ablate the published PIH heads and compare against random same-layer heads

The main result I would expect is simple: baseline counting should stay roughly flat, while misaligned prompts should stop matching the prompt and start matching the image.

A useful plot would put ground-truth count on the x-axis and prompt-match rate on the y-axis, before and after ablation. If the paper is right, the high-count blue curves should collapse.

Limitations and Open Questions

The study focuses on ~7B models. The exact heads may not transfer to much larger VLMs.

The analysis is also attention-centric. The authors are careful about this: influential heads do not always have clean, human-readable attention maps. Ablation can also create secondary effects in other layers that they do not fully trace.

Another open question is when you want the model to follow the prompt. "Describe four waterlilies" can be a legitimate creative instruction. The paper grounds the task with "in the image" and treats invented objects as hallucinations, which is fair for their setting, but real products need a clearer policy for instruction following versus visual fidelity.

Finally, counting and color are controlled conflicts. It is still unclear how far the same heads go for richer mismatches, such as wrong attributes in charts, maps, or urban scenes.

Ideas for Our Work

This paper connects to my work because it is about the wrong signal winning under conflict.

In fairness and urban evaluation, aggregate metrics can look fine while a local failure mode is doing the real damage. Here the aggregate story would be "the model can count." The conflict setting shows that text can still override vision once confidence drops.

That is close to RISE. RISE is about making residual structure visible when scalar fairness numbers hide subgroup failure. PIH is about making modality conflict visible when baseline accuracy hides prompt sycophancy. Both cases need a diagnostic view that is more local than a single score.

A possible follow-up idea is:

Prompt–Image Conflict Probes for Urban VLMs

Build a small suite of urban image questions where the prompt slightly misstates a count, color, or attribute that is visible in the scene. Measure prompt match versus image match before and after interventions like PIH-head ablation.

The question would be:

When a VLM answers questions about cities, zones, or street scenes, does a mild prompt error flip the answer even when the image evidence is clear?

That would connect this ACL paper to the same evaluation theme in UrbanContrastiveQA: pairwise or conflict-based tests can expose failures that raw accuracy misses.

A second follow-up idea is:

RISE-Style Diagnostics for Multimodal Conflict

Treat prompt-match versus true-match as a residual-like signal across bins of object count, confidence, or scene type. Then visualize where text overrides vision. If PIH ablation works, the residual plot should show less extreme prompt-following in the low-confidence, high-count region.

That feels like a clean bridge between mechanistic VLM work and the fairness visualization tools we already build.

For more information about our research, return to our homepage: ufdatastudio.com.