Machine Learning PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-14

Rethinking KV Caching for LLM Decode: Free-Space Optical KV Regeneration for Reduction of Memory Traffic

Aritra Bagchi, Bin Gao, James Spall, Jakub Pietrak, Weng-Fai Wong, Tulika Mitra

National University of Singapore +1 · arXiv

Affiliations: National University of Singapore, Lumai Ltd

Abstract

Long-context large language model (LLM) decoding repeatedly reads the historical key-value (KV) cache, making multi-head attention (MHA) memory-bandwidth-bound. Caching layer-input activations instead halves historical-cache storage and traffic, but requires regenerating KV from each cached activation. On digital accelerators such as GPUs, this regeneration takes far longer than fetching the activation from HBM. The KV regeneration overhead outweighs the savings from reduced memory traffic, shifting decode attention from a memory bottleneck to a compute bottleneck. We present OptiDecode, a hybrid optoelectronic decode-attention accelerator that regenerates historical KV using free-space optics and computes attention digitally. Projection weights remain resident on optical tiles, exploiting reuse across historical tokens for high-throughput optical regeneration. Scaling to large models introduces two challenges. First, distributing weights across tiles requires sending the same activations to every head group, while each readout die produces only partial KV. Activation replay shares one set of electrical-to-optical converters across head groups without costly broadcast. In-situ attention avoids transferring partial KV from readout dies to reducers by computing on them locally and exchanging only partial scores, probabilities, and partial outputs. Second, processing an activation chunk through all dependent stages takes longer than one activation-fetch interval. Chunk pipelining overlaps successive chunks so that OptiDecode sustains the HBM activation-fetch rate. OptiDecode demonstrates 1.92-1.99x decode-attention speedup over an ideal GPU KV-read baseline at equal peak HBM bandwidth (A100-B200), for 32K-token contexts across evaluated 30B-175B MHA models, approaching the 2x upper bound from halving historical memory traffic.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper presents a novel hybrid optoelectronic architecture that leverages free-space optics to regenerate KV caches from activations, effectively halving memory traffic in LLM decoding and achieving near-theoretical speedups over ideal GPU baselines. By rigorously addressing system-level challenges such as activation fan-out and partial-KV communication through techniques like activation replay and in-situ attention, the work provides a compelling case for integrating photonic computing into AI inference pipelines, offering a significant technical contribution to the field of hardware-aware machine learning systems.

Methodology

The paper proposes OptiDecode, a hybrid optoelectronic accelerator that shifts the bottleneck of LLM decoding from memory bandwidth to compute by using Free-Space Optics (FSO) for KV regeneration. The core insight is that while digital GPUs struggle to regenerate KV from cached activations fast enough to offset the memory savings, FSO tiles with resident weights can perform matrix-vector multiplications at rates exceeding HBM fetch speeds. The methodology is rigorous, addressing specific system-level challenges: (1) Activation Replay to avoid costly broadcast of activations to multiple head groups, (2) In-Situ Attention to avoid gathering partial KV across dies by computing attention locally and exchanging only scores/probabilities, and (3) Chunk Pipelining to overlap processing stages and sustain the HBM fetch rate. The mathematical derivation of the required modulation rates and resource provisioning is detailed and well-grounded in the physical constraints of optical computing (e.g., power, precision, tile dimensions).

Results & evaluation

The evaluation is primarily analytical and simulation-based, which is standard for hardware architecture papers at this stage. It compares OptiDecode against an "ideal GPU KV-read baseline" (roofline) and measured GPU performance (SGLang/FlashInfer on A100-H200-B200). The results show a 1.92-1.99x speedup over the ideal GPU baseline for 30B-175B models at 32K context, approaching the theoretical 2x upper bound. The paper provides extensive ablation studies, sensitivity analyses on hardware parameters (tile size, modulation rate, wavelengths), and energy comparisons. The inclusion of measured GPU energy consumption adds practical relevance, showing OptiDecode is 37-43% more energy-efficient than measured GPU execution.

Reproducibility

The paper provides detailed architectural specifications (Table 1), including tile dimensions, modulation rates, SRAM sizes, and link bandwidths. However, as this is a proposed architecture relying on emerging optical technology (specifically Lumai's Iris Nova, mentioned in related work), full hardware reproducibility is currently limited to the research team. The analytical models are well-defined, allowing other researchers to replicate the performance estimates. The code for the analytical model is not explicitly provided in the text, but the equations are sufficient for re-implementation.

Limitations

The primary limitation is the reliance on specialized, non-standard hardware (Free-Space Optics) that is not yet widely available or integrated into standard data centers. The paper assumes ideal conditions for optical precision and power consumption, which may vary in practice. Additionally, the speedup is relative to an "ideal" GPU baseline; compared to real-world GPU implementations with overheads, the gap might be larger, but the comparison to measured GPUs shows a more modest 1.18x full-layer speedup, indicating that other parts of the inference pipeline (FFN, weight loading) still dominate or limit the overall benefit. The approach is also specific to Multi-Head Attention (MHA); extensions to Grouped Query Attention (GQA) or Sparse Attention are mentioned as future work but not fully evaluated.

Broader impact

This paper is significant for the hardware acceleration community, demonstrating a viable path to overcoming the memory wall in LLM inference using photonic computing. It provides a concrete system design that bridges the gap between high-level ML requirements and low-level optical hardware constraints. While the immediate adoption may be limited to specialized edge or data-center deployments with optical infrastructure, the insights into KV caching strategies and the co-design of optical/digital pipelines are valuable for future accelerator architectures. It highlights that "compute-bound" problems in inference can be solved with different physical paradigms, potentially influencing the design of next-generation AI chips. The paper presents a novel hybrid optoelectronic architecture that leverages free-space optics to regenerate KV caches from activations, effectively halving memory traffic in LLM decoding and achieving near-theoretical speedups over ideal GPU baselines. By rigorously addressing system-level challenges such as activation fan-out and partial-KV communication through techniques like activation replay and in-situ attention, the work provides a compelling case for integrating photonic computing into AI inference pipelines, offering a significant technical contribution to the field of hardware-aware machine learning systems.

Author of this paper? Send a correction.