Unbundling Multimodal Vision Encoding
Abstract
How multimodal language models should encode visual input remains up for debate. Most models use a pretrained vision encoder, while some recent recipes train the encoder from scratch with the language model or remove it entirely. Vision encoders bundle together three design choices: a visual prior from upstream pretraining, dedicated vision capacity, and pre-fusion placement of that capacity (i.e., before the multimodal backbone). Because published models also differ in data, architecture, and compute, comparing them cannot isolate the effect of each choice. We disentangle these choices via a controlled framework where we pretrain all models from scratch on common data and compare them on visual question answering and language benchmarks at matched lifetime compute. We find that the visual prior is not necessary; an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. What matters instead is dedicated vision capacity: an encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. Yet that capacity need not precede multimodal fusion, as vision-only modules added in-fusion to early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. These results also hold across backbone sizes, when encoder pretraining is treated as free, and when starting from a pretrained language model. Overall, our findings favor dedicated vision capacity trained jointly with the backbone and placed early within it.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper provides a controlled study that disentangles the design choices of multimodal vision encoding, finding that dedicated vision capacity is essential but does not require separate pretraining or pre-fusion placement. By training models from scratch with matched lifetime compute, the authors demonstrate that in-fusion vision capacity can match pre-fusion encoders while offering lower time-to-first-token, challenging prevailing assumptions in the field and offering practical insights for efficient multimodal model design.
Methodology
The paper proposes a rigorous controlled framework to disentangle the contributions of visual priors, dedicated vision capacity, and pre-fusion placement in multimodal models. The methodology is robust, training all models from scratch on common data with matched lifetime compute to isolate variables that are typically confounded in published models. The introduction of "in-fusion" vision capacity (vision-only modules within the backbone) is a novel architectural contribution that challenges the standard encoder-decoder paradigm.
Results & evaluation
The experiments are extensive, covering backbone sizes from 0.6B to 6.4B and various compute budgets. The use of matched lifetime compute is a significant methodological improvement over previous studies that often treated encoder pretraining as free. The results are consistent across scales and show that dedicated vision capacity is crucial, but its placement (pre-fusion vs. in-fusion) is less critical than previously thought. The inclusion of inference speed (TTFT) analysis adds practical relevance.
Reproducibility
The paper provides detailed descriptions of the architecture, data mixtures, and optimization settings. However, the code is not explicitly linked in the provided text, and the reliance on specific internal datasets (Nemotron-CC, MetaCLIP) may limit immediate reproducibility for external researchers without access to these resources.
Limitations
The study is limited to dense backbones up to 6.4B parameters, and the findings may not directly translate to sparse Mixture-of-Experts models or much larger scales. The use of 224x224 resolution for training may not capture the full benefits of higher-resolution data. The text-image mixture ratio was tuned for one specific recipe, potentially biasing the comparison.
Broader impact
The findings have significant implications for the design of future multimodal models, suggesting that practitioners can simplify training pipelines by removing separate encoder pretraining stages without sacrificing performance, and that architectural flexibility in placing vision capacity can improve inference efficiency. This could lead to more efficient and cost-effective multimodal model development. The paper provides a controlled study that disentangles the design choices of multimodal vision encoding, finding that dedicated vision capacity is essential but does not require separate pretraining or pre-fusion placement. By training models from scratch with matched lifetime compute, the authors demonstrate that in-fusion vision capacity can match pre-fusion encoders while offering lower time-to-first-token, challenging prevailing assumptions in the field and offering practical insights for efficient multimodal model design.
Author of this paper? Send a correction.