Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Abstract
Vision-language models (VLMs) often fail to match a point or an object across two images, even when the answer can be decoded from their own hidden states. We trace this failure from the visual tokens to the answer letter, in Qwen3-VL (2B-32B), InternVL3.5-8B, and Gemma-3-12B, on synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), and find two distinct failure points, a small set of attention heads that carries the comparison, and a repair that touches 0.02% of the parameters. Two failure points: exact copies are compared but not read out. An answer-position probe exceeds the calibrated output by 15 to 25 points in every family, for nameable and unnameable shapes alike, and steering along the probe's direction moves no more answers than random directions. Semantic correspondences are present in the visual tokens but never reach the answer. One pathway: redistributing the cross-image attention of 16-32 heads, with its mass preserved, removes 17-22 points of synthetic and 15-32 points of HPatches accuracy, against at most 7 for random or attention-matched heads. A lever that training finds on its own: training only these heads' query and key projections raises SPair-71k accuracy by 28.3 points (Qwen3-VL-8B) and 25.1 points (InternVL3.5-8B), beats both layer-matched and attention-matched heads, and transfers to HPatches and BLINK. An unconstrained all-layer adapter routes 21-51% of its gain through the attention of the same heads, against at most 7% for random heads, and does so by changing what they read (83-86% of the effect) rather than their weights. Every decision rule was frozen before its run, and every reported number is recomputed by independent verification scripts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies a small set of causal attention heads responsible for cross-image comparison in VLMs and demonstrates that fine-tuning only these heads repairs semantic correspondence deficits with minimal parameter updates. This work bridges mechanistic interpretability and efficient model repair, providing a rigorous framework for diagnosing and fixing specific cognitive failures in multimodal models, with high reproducibility and strong experimental controls.
Methodology
The paper employs a rigorous mechanistic interpretability framework to diagnose and repair cross-image comparison failures in Vision-Language Models (VLMs). The methodology is distinguished by its causal rigor: it does not merely correlate attention patterns with performance but establishes causality through mass-preserving knockouts and attention transplantation. The identification of a specific subset of "cross-image heads" (16-32 heads) that mediates the comparison between two images is a significant methodological advance. The repair strategy, which involves fine-tuning only the query and key projections of these specific heads, is highly targeted and efficient (touching only 0.02% of parameters). The use of frozen decision rules and independent verification scripts enhances the scientific validity of the claims, distinguishing it from exploratory interpretability work that often suffers from post-hoc rationalization.
Results & evaluation
The experiments are extensive and well-controlled. The authors evaluate across three model families (Qwen3-VL, InternVL3.5, Gemma-3) and multiple scales (2B-32B), ensuring the findings are not model-specific. The tasks cover synthetic shape matching, same-instance point correspondence (HPatches), and semantic correspondence (SPair-71k), providing a comprehensive view of the failure modes. The ablation studies are particularly strong: comparing targeted head training against random heads, layer-matched heads, and attention-matched heads isolates the specific contribution of the identified mechanism. The transfer experiments to BLINK and other benchmarks demonstrate that the repair generalizes beyond the training distribution. The statistical reporting includes bootstrapped confidence intervals, which is a best practice often missing in similar literature.
Reproducibility
The paper explicitly commits to releasing code, item generators, split identifiers, head indices, and trained adapters. The description of the experimental protocol, including the freezing of decision rules before execution and the use of independent verification scripts, suggests a high level of reproducibility. The detailed specification of the probe architectures, calibration methods, and attention manipulation techniques provides a clear roadmap for replication. The compute requirements (approx. 90 GPU-hours) are modest, making the experiments accessible to a broader range of research groups.
Limitations
The primary limitation is the scale of the mechanistic analysis, which is restricted to 8B models for the detailed head-level interventions, although measurement results extend to 32B. The paper acknowledges that the identified pathway is "sufficient but redundant," meaning that while training these heads is effective, it is not the only possible repair strategy, and broader adapters can achieve higher performance by modifying other components. Additionally, the repair is tested primarily on correspondence tasks; while transfer to BLINK is shown, the impact on other VLM capabilities (e.g., reasoning, OCR) is not extensively explored, though the authors note that targeted repairs can have unintended effects.
Broader impact
This work has significant implications for the field of VLM interpretability and efficient fine-tuning. By identifying a small, causal subset of attention heads responsible for a specific cognitive task (cross-image comparison), it provides a template for diagnosing and repairing other VLM failures. The finding that a 0.02% parameter update can recover most of the semantic correspondence deficit suggests that many VLM failures are not due to a lack of capacity in the vision encoder but rather a misalignment in the language model's attention mechanisms. This insight could lead to more efficient post-training strategies and better understanding of how VLMs integrate visual information. The "selectivity" metric introduced as a cheap diagnostic for repair effectiveness is a practical tool for practitioners. The paper identifies a small set of causal attention heads responsible for cross-image comparison in VLMs and demonstrates that fine-tuning only these heads repairs semantic correspondence deficits with minimal parameter updates. This work bridges mechanistic interpretability and efficient model repair, providing a rigorous framework for diagnosing and fixing specific cognitive failures in multimodal models, with high reproducibility and strong experimental controls.
Author of this paper? Send a correction.