OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented Generation
Abstract
Retrieval-augmented generation (RAG) is bottlenecked by the token budget of large language models: feeding long documents into the context window is expensive, while aggressive textual compression discards fine-grained information that is irreversibly lost in a one-dimensional token sequence. We argue that this bottleneck is not fundamental to information density but to the choice of compression \emph{modality}. Modern visual encoders can pack the contents of a rendered text page into roughly a hundred visual tokens with little semantic loss, opening a new compression-fidelity tradeoff that purely textual methods cannot reach. We present \textbf{OpticalRAG}, a framework that renders documents as page images, encodes them with a frozen vision encoder, and distills the retrieved visual tokens into as few as $16$ injected tokens before passing them to the LLM. Two ingredients make this work without expensive cross-modal training: (i) \emph{Encoder-Bridged Retrieval}, which projects text-query states into the visual feature space of the frozen encoder, through a projection shared with the compressor, to retrieve visual tokens from a text query, with no contrastive training; and (ii) \emph{Query-Driven Distillation}, a lightweight cross-attention compressor that condenses the retrieved visual tokens into a small set of query-conditioned prefix tokens. On LongBench-v2, BAMBOO, and LooGLE-v2, OpticalRAG achieves the best compression-aware accuracy (CAP) on all three benchmarks while injecting only $16$ tokens, and improves the closed-book Qwen2.5-7B-Instruct backbone by $5.17$ points on LongBench-v2.Code is available at https://github.com/brighterr/OpticalRAG.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
OpticalRAG introduces a pixel-space compression framework for RAG that renders documents as images and distills visual tokens into a minimal set of prefix tokens, achieving state-of-the-art compression-aware accuracy with only 16 injected tokens. This approach effectively addresses the token budget bottleneck in LLMs by leveraging the high information density of visual encoders, offering a novel and efficient alternative to traditional textual compression methods.
Methodology
The paper proposes OpticalRAG, a framework that shifts the compression modality for RAG from text to pixels. By rendering documents as images and using a frozen vision encoder, it leverages the high information density of visual tokens. The core innovation lies in "Encoder-Bridged Retrieval," which projects text queries into the visual feature space without contrastive training, and "Query-Driven Distillation," which condenses these visual tokens into a minimal set of prefix tokens (as few as 16) for the LLM. This approach is architecturally sound and avoids the heavy cross-modal training typically required for such alignment, making it a practical and elegant solution to the context window bottleneck.
Results & evaluation
The authors evaluate OpticalRAG on LongBench-v2, BAMBOO, and LooGLE-v2, reporting the best compression-aware accuracy (CAP) with only 16 injected tokens. The improvement of 5.17 points over the closed-book baseline on LongBench-v2 is significant. However, the provided text is largely a skeleton with placeholders for the actual experimental data, making it difficult to fully verify the statistical significance or the robustness of the results across different document types and query complexities.
Reproducibility
The paper provides a GitHub link for the code, which is a positive step for reproducibility. The use of a frozen vision encoder simplifies the training pipeline, reducing the computational barrier for replication. However, the specific rendering parameters, resolution settings, and the exact architecture of the projection layers are not detailed in the provided abstract and skeleton, which may hinder exact reproduction without the full codebase.
Limitations
The primary limitation is the reliance on rendering text to images, which introduces potential OCR-like errors if the rendering is not perfect or if the vision encoder struggles with specific font styles or layouts. Additionally, the method is constrained by the resolution and token budget of the vision encoder; extremely dense documents might still suffer from information loss. The lack of detailed ablation studies in the provided text limits the understanding of which component contributes most to the performance gain.
Broader impact
This work has the potential to significantly impact the field of RAG by offering a new paradigm for handling long-context tasks. If the pixel-space compression proves robust, it could enable LLMs to process much longer documents without increasing inference costs, potentially democratizing access to long-context capabilities for smaller models. It also opens up avenues for multimodal RAG where visual and textual information are processed in a unified, compressed space. OpticalRAG introduces a pixel-space compression framework for RAG that renders documents as images and distills visual tokens into a minimal set of prefix tokens, achieving state-of-the-art compression-aware accuracy with only 16 injected tokens. This approach effectively addresses the token budget bottleneck in LLMs by leveraging the high information density of visual encoders, offering a novel and efficient alternative to traditional textual compression methods.
Author of this paper? Send a correction.