Improved Baselines with Visual Instruction Tuning
Abstract
Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Code and model will be publicly available.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The main contribution of this paper is the establishment of stronger baselines for visual instruction tuning in LMMs through simple modifications to existing architectures. While the results are promising and the methodology is sound, the paper could benefit from deeper exploration of its implications and more comprehensive experimental details.
Methodology
The paper proposes modifications to the LLaVA model by integrating CLIP-ViT-L-336px with an MLP projection and incorporating academic-task-oriented VQA data. This approach is straightforward yet effective, demonstrating a solid understanding of the underlying architecture and its capabilities. However, the methodology lacks depth in explaining the rationale behind the specific choices made, such as the selection of the CLIP model and the nature of the VQA data used.
Results & evaluation
The experiments conducted are robust, with the authors claiming state-of-the-art results across 11 benchmarks. However, the paper does not provide extensive details on the benchmarks or the evaluation metrics used, which limits the ability to fully assess the significance of the results. The training process is described as efficient, completing in about a day on a single node, which is a notable achievement for accessibility in LMM research.
Reproducibility
The authors mention that the code and model will be publicly available, which is a positive aspect for reproducibility. However, without specific URLs or detailed implementation instructions provided in the paper, it is difficult to fully evaluate the reproducibility of the results.
Limitations
The paper does not adequately address potential limitations of the proposed approach, such as the generalizability of the results across different datasets or tasks. Additionally, the reliance on publicly available data raises questions about the diversity and representativeness of the training data.
Broader impact
The research has the potential to make LMM more accessible, which could democratize the use of advanced models in various applications. However, the impact is somewhat limited by the lack of detailed exploration of the implications of the findings. The main contribution of this paper is the establishment of stronger baselines for visual instruction tuning in LMMs through simple modifications to existing architectures. While the results are promising and the methodology is sound, the paper could benefit from deeper exploration of its implications and more comprehensive experimental details.
Curator note
Liu et al.; CLIP + LLM with simple MLP projection; strong VQA baseline
Author of this paper? Send a correction.