LEAPS: Long-context Evaluation of Adaptive Pattern Learning and State Tracking
Abstract
Long-context language models (LCLMs) power agents that act over long trajectories, from improving a codebase to running research experiments to forecasters that condition on lengthy histories. These applications draw on three distinct long-context capabilities: recall, where the answer is a fact present in the context; state tracking, where the outcome can be computed by performing a series of transformations to a specified state; and pattern learning, where the goal is to make a prediction by sampling from a distribution inferred from many observations. Existing benchmarks primarily focus on the first capability. In this work, we introduce LEAPS (Long-context Evaluation of Adaptive Pattern learning and State tracking), a collection of 12 novel datasets, spanning agentic workflows, behavioral modeling, and forecasting of market and world events. By construction, LEAPS is resistant to data contamination, extends to millions of input tokens, and uses reliable metrics. We evaluate 30 API and open models from 8K to 512K input tokens. LEAPS distinguishes models that perform near perfectly on recall tasks, finding that such models' state tracking and pattern learning abilities can vary significantly and degrade much more dramatically with increasing input lengths--even for pattern learning, where more observations should make learning a theoretically easier task. Moreover, thinking models often underperform their base counterparts, exhausting their generation budget or oversimplifying the task. We hope LEAPS can guide the development of the next generation of LCLMs.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces LEAPS, a comprehensive benchmark for evaluating long-context language models on recall, state tracking, and pattern learning. It provides rigorous experimental evidence that current models struggle with state tracking and pattern learning at scale, offering critical insights for the development of next-generation LCLMs.
Methodology
The paper introduces LEAPS, a benchmark suite comprising 12 datasets designed to evaluate three specific long-context capabilities: recall, state tracking, and pattern learning. The methodology is rigorous, emphasizing resistance to data contamination through procedural generation of synthetic data and the use of reliable metrics. The distinction between these three capabilities is a strong conceptual contribution, as existing benchmarks often conflate them or focus heavily on recall. The approach to extending evaluations to millions of tokens is methodologically sound, though the reliance on API models for the upper end of the scale introduces potential variability in evaluation conditions.
Results & evaluation
The experimental scope is extensive, evaluating 30 models (both API and open-source) across input lengths from 8K to 512K tokens. The results provide valuable insights, particularly the finding that models excelling in recall do not necessarily perform well in state tracking or pattern learning. The observation that "thinking" models often underperform their base counterparts due to budget exhaustion or oversimplification is a significant and counter-intuitive empirical finding. The degradation of performance with increasing context length, even in tasks where more data should theoretically help (pattern learning), is a critical result for the field.
Reproducibility
The authors provide a strong reproducibility statement, releasing code for data generation and evaluation, as well as the original evaluation data in Parquet format (with exceptions for licensing). The use of synthetic data generation scripts ensures that the benchmark can be regenerated, further mitigating contamination risks. The exclusion of social media data due to licensing is a minor limitation but does not significantly impact the core reproducibility of the main findings.
Limitations
The primary limitation is the reliance on API models for the longest context lengths, which may not be fully reproducible by other researchers due to cost and access constraints. Additionally, the benchmark focuses on specific types of long-context tasks (agentic, behavioral, forecasting), which may not cover all potential applications of LCLMs. The performance of open models at the 512K token limit may be limited by hardware constraints, potentially skewing the comparison if not all models can be tested at the maximum length.
Broader impact
This benchmark has high potential impact as it addresses a critical gap in the evaluation of long-context models. By distinguishing between recall, state tracking, and pattern learning, it provides a more nuanced view of model capabilities than existing benchmarks. The findings regarding the limitations of current models, particularly the degradation with length and the underperformance of thinking models, will likely guide future research directions in model architecture and training. The benchmark's resistance to contamination makes it a robust tool for ongoing evaluation as models evolve. The paper introduces LEAPS, a comprehensive benchmark for evaluating long-context language models on recall, state tracking, and pattern learning. It provides rigorous experimental evidence that current models struggle with state tracking and pattern learning at scale, offering critical insights for the development of next-generation LCLMs.
Author of this paper? Send a correction.