Visual Instruction Tuning (LLaVA)
Abstract
Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
This paper introduces a data-centric instruction-tuning paradigm that bridges vision and language models using GPT-4-generated multimodal instruction data. By demonstrating that a simple linear projection combined with high-quality synthetic instruction data yields strong zero-shot multimodal reasoning and competitive benchmark performance, the work fundamentally shifted the field's focus from architectural complexity to data quality and instruction alignment, establishing an open, reproducible baseline that catalyzed the modern era of open-source vision-language models.
Methodology
The methodology is elegantly minimal yet highly effective. The architecture connects a frozen CLIP ViT-L/14 vision encoder to a frozen Vicuna-7B LLM via a single trainable linear projection layer. The core methodological breakthrough lies not in architectural complexity, but in the data-centric paradigm: leveraging a proprietary, highly capable language model (GPT-4) to synthesize high-quality, diverse, and instruction-following multimodal training data (LLaVA-Instruct-150K). The two-stage training pipeline (feature alignment pre-training on CC3M followed by end-to-end instruction tuning) demonstrates that modality bridging can be achieved through data quality and scale rather than complex cross-attention or query-based mechanisms. While conceptually straightforward, the execution rigorously isolates the impact of instruction-tuning data on multimodal generalization.
Results & evaluation
The experimental design is comprehensive and strategically targeted. The authors evaluate zero-shot capabilities across standard vision-language benchmarks (VQAv2, GQA, VizWiz, etc.) and construct novel, challenging instruction-following evaluation sets to probe reasoning, spatial awareness, and complex task execution. The reported 85.1% relative score against GPT-4 on synthetic multimodal instructions and the 92.53% accuracy on ScienceQA (when fine-tuned) provide strong empirical validation. The evaluation framework successfully establishes that a simple projection + instruction-tuning recipe can yield competitive open-weight VLMs. However, reliance on synthetic evaluation metrics and LLM-as-a-judge scoring introduces potential evaluation bias, and real-world robustness to distribution shifts remains partially untested.
Reproducibility
Exceptional. The authors release the complete training pipeline, hyperparameters, model checkpoints, and the full 150K instruction-tuning dataset. The codebase is clean, well-documented, and designed for low-barrier fine-tuning on consumer-grade hardware. Compute requirements are transparently reported, and the modular design allows straightforward swapping of vision encoders or base LLMs. This level of openness has made it the de facto reproducible baseline for the field.
Limitations
The approach inherits several constraints. First, it is fundamentally dependent on GPT-4 for data generation, introducing cost, accessibility barriers, and potential propagation of proprietary model biases. Second, the linear projection architecture lacks fine-grained spatial reasoning and high-resolution OCR capabilities, limiting performance on dense visual grounding or document understanding tasks. Third, like early instruction-tuned VLMs, it exhibits hallucination tendencies and struggles with precise numerical or geometric reasoning. Finally, the evaluation heavily relies on synthetic benchmarks that may not fully capture real-world multimodal interaction complexity.
Broader impact
This work democratized large-scale vision-language research by proving that high-performing VLMs do not require proprietary infrastructure or complex architectural innovations. It catalyzed an explosion of open-source VLM development, established data synthesis as a primary research direction, and provided a standardized, accessible baseline for academic and industrial experimentation. While it raises valid concerns about over-reliance on synthetic data and proprietary LLMs for training open models, its net impact on accelerating multimodal AI research and lowering entry barriers is profoundly positive. This paper introduces a data-centric instruction-tuning paradigm that bridges vision and language models using GPT-4-generated multimodal instruction data. By demonstrating that a simple linear projection combined with high-quality synthetic instruction data yields strong zero-shot multimodal reasoning and competitive benchmark performance, the work fundamentally shifted the field's focus from architectural complexity to data quality and instruction alignment, establishing an open, reproducible baseline that catalyzed the modern era of open-source vision-language models.
Curator note
Liu et al.; open-source multimodal instruction-following
Author of this paper? Send a correction.