The Explainability Illusion: Must ViTs Exhibit Patch-Decoding Interpretability?
Abstract
Transformers are widely assumed to exhibit token-decoding explainability. In particular, in vision transformer (ViT) classifiers, decoding individual patch tokens has become a common diagnostic tool due to its tendency to yield human-interpretable outputs. In light of the increasing popularity of this approach, an important question is whether token-decoding explainability is an inherent property of transformers that should be expected to hold across architectures, scales, and training objectives, or merely an artifact of some implicit bias. In this paper we systematically study token-decoding explainability across diverse ViT settings, spanning architectures, scales, and training objectives, using both controlled experiments and real-world models. We find that while some models indeed exhibit aligned and interpretable token decodings, others exhibit either inverted explainability maps or a complete lack of correlation with the image semantics. As we show, these tendencies toward different explainability behaviors are not intrinsic to the architecture and can vary arbitrarily with changes in the optimization landscape. We provide theoretical support for these observations by demonstrating a setting where a single fixed toy transformer architecture can provably implement the optimal classification rule in multiple ways, which differ in their patch-decoding explainability patterns. Our results show that token-decoding explainability is not an inherent property of trained transformers, but rather an outcome contingent on the problem setting and thus should not be a-priori assumed to hold. Code can be found in https://ronensc.github.io/Explainability-Illusion/ .
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper challenges the assumption that Vision Transformers inherently possess patch-decoding interpretability by demonstrating through empirical and theoretical means that this property is contingent on optimization rather than architecture. By showing that identical classification performance can be achieved with completely different or inverted interpretability maps, the work provides a critical cautionary tale for the field, suggesting that current interpretability diagnostics based on token decoding may be unreliable artifacts of specific training dynamics rather than intrinsic model properties.
Methodology
The paper employs a rigorous dual-pronged methodology combining systematic empirical analysis with theoretical counterexamples. The empirical component involves training and evaluating a diverse suite of Vision Transformers (ViTs) across varying architectures, scales, and training objectives to test the robustness of patch-decoding interpretability. The theoretical component constructs a provable toy model demonstrating that a fixed transformer architecture can implement the same optimal classification function via multiple weight configurations that yield diametrically opposed interpretability maps (aligned vs. inverted). This combination of empirical breadth and theoretical necessity is methodologically sound and effectively addresses the core research question.
Results & evaluation
The experimental evaluation is comprehensive, covering both controlled synthetic settings and real-world models, including Vision-Language Models (VLMs). The paper successfully demonstrates that interpretability is not a stable property, showing cases where decoding maps are inverted or uncorrelated with semantics. The inclusion of VLMs adds significant relevance, as these are currently the dominant paradigm in vision AI. The results are surprising and challenge a widely held assumption in the community, which is a hallmark of high-impact empirical work.
Reproducibility
The authors provide a public code repository, which is a strong positive for reproducibility. However, the full paper text provided is a skeleton of LaTeX file references rather than the rendered content. While the abstract and structure suggest a complete study, the lack of visible detailed hyperparameters or specific dataset splits in the provided text limits the immediate assessment of granular reproducibility details. The code link is the primary vehicle for verification.
Limitations
The theoretical proof relies on a "toy transformer" architecture, which may not fully capture the complexity of large-scale, pre-trained industrial models. While the empirical section addresses this, the gap between the simplified theoretical model and real-world complexity remains a potential limitation in generalizing the theoretical insights. Additionally, the paper focuses on the *existence* of the illusion rather than providing a robust method to *detect* or *correct* it in arbitrary black-box models, leaving practitioners without a clear diagnostic tool.
Broader impact
This paper has high potential for broader impact by correcting a fundamental misconception in the AI interpretability community. If token-decoding is not inherently reliable, it invalidates many existing interpretability tools and analyses that rely on this assumption. This could lead to a re-evaluation of how interpretability is measured and reported in future research, potentially shifting focus toward more robust, architecture-agnostic interpretability metrics. It serves as a crucial "reality check" for the field. The paper challenges the assumption that Vision Transformers inherently possess patch-decoding interpretability by demonstrating through empirical and theoretical means that this property is contingent on optimization rather than architecture. By showing that identical classification performance can be achieved with completely different or inverted interpretability maps, the work provides a critical cautionary tale for the field, suggesting that current interpretability diagnostics based on token decoding may be unreliable artifacts of specific training dynamics rather than intrinsic model properties.
Author of this paper? Send a correction.