Mental Imagery That Matters: Curing Latent Collapse in Multimodal Reasoning
Abstract
Mental simulation, the ability to reason by constructing and updating internal visual states, is central to flexible human cognition, but remains elusive in current multimodal large language models. Latent multimodal reasoning offers a promising route toward this capability by interleaving text with continuous visual latent blocks, allowing models to maintain internal visual workspaces without decoding them into pixels. In this paper, we ask whether these latent visual states actually matter. Through a direct perturbation analysis across representative latent-reasoning MLLMs, we find that they often do not: dropping, replacing, or shuffling generated visual latents has only minor effects on the model's own final-answer probability, while equivalent perturbations to the linguistic reasoning trace are substantially more damaging. We call this failure mode Mental Simulation Collapse (MSC): the model appears to perform interleaved visual reasoning, but routes most answer-relevant computation through text. We further diagnose that MSC follows naturally from existing training objectives: visual alignment can make generated latents resemble helper-image embeddings, but it does not require the decoder to rely on them; subsequent text-only supervision further leaves the semantic role of the latent pathway unidentifiable. To address this, we introduce AnchorSim (Anchored Mental Simulation), a contrastive distillation framework that trains models to make their generated visual latents consequential. Across multiple visual reasoning benchmarks, AnchorSim improves performance, while diagnostic re-evaluation shows that visual perturbations now meaningfully affect predictions. Our results expose a fundamental failure mode of latent multimodal reasoning and demonstrate that mental simulation must be anchored to model behavior in order to matter.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper identifies and addresses "Mental Simulation Collapse," a critical failure mode where latent multimodal models ignore their generated visual states. By introducing a rigorous perturbation-based diagnostic and the AnchorSim contrastive distillation framework, it provides a necessary corrective mechanism for ensuring that visual latents are causally relevant to reasoning, significantly advancing the reliability of latent-space multimodal models.
Methodology
The paper introduces a diagnostic framework for "Mental Simulation Collapse" (MSC) in latent multimodal reasoning models. The core methodological contribution is a perturbation analysis that measures the sensitivity of final answer probabilities to changes in visual latent states versus linguistic tokens. The proposed solution, AnchorSim, utilizes a contrastive distillation objective to force the model to rely on the generated visual latents for reasoning, rather than treating them as decorative or redundant. This is a novel training objective that addresses a specific identifiability issue in multimodal training.
Results & evaluation
The experiments validate the MSC hypothesis across representative latent-reasoning MLLMs, showing that visual latents are often ignored. The evaluation of AnchorSim demonstrates improvements on visual reasoning benchmarks and, crucially, verifies that the fix works by showing that visual perturbations now significantly impact predictions. The use of diagnostic re-evaluation is a strong experimental design choice that directly tests the causal claim of the paper.
Reproducibility
The paper provides a project page and likely code (implied by the project URL). The perturbation analysis is straightforward to implement, and the contrastive distillation loss is clearly defined. The reliance on specific "latent-reasoning MLLMs" requires access to those specific model checkpoints, which may limit immediate reproducibility for those not using that specific architecture family, but the diagnostic method is generalizable.
Limitations
The findings are specific to models that explicitly generate continuous visual latent blocks. It may not apply to standard VLMs that only process input images without generating internal visual states. The "mental simulation" framing is somewhat anthropomorphic, and the paper relies on the assumption that "relying on latents" is strictly better than "relying on text," which is a reasonable but debatable assumption in all cases.
Broader impact
This paper exposes a critical failure mode in the emerging field of latent multimodal reasoning. By providing a diagnostic tool (perturbation analysis) and a fix (AnchorSim), it sets a new standard for evaluating whether multimodal models are truly using visual information in their reasoning chains. This will influence how future latent-space reasoning models are trained and evaluated. The paper identifies and addresses "Mental Simulation Collapse," a critical failure mode where latent multimodal models ignore their generated visual states. By introducing a rigorous perturbation-based diagnostic and the AnchorSim contrastive distillation framework, it provides a necessary corrective mechanism for ensuring that visual latents are causally relevant to reasoning, significantly advancing the reliability of latent-space multimodal models.
Author of this paper? Send a correction.