Articulation Upper Bounds for Language Models
Abstract
Language captures only a subset of the perceptions, intuitions and intents that form meaning between speakers. What, then, are the limits of language? We take steps towards measuring this in LLMs as follows: for a set of concepts, we measure how well each concept can be transmitted between two LLMs as their capabilities scale. This requires novel measurement tools: (1) optimal messages: we introduce a prompt optimization algorithm that beats existing algorithms by up to 14% on benchmarks and accurately estimates prompt upper bounds. (2) A transmission measure: the sender is asked to label text; the receiver then tries to recover the sender's prompt, and recovery is measured; we find this correlates strongly with human label error. With these measures, we use observational scaling laws to study how transmissibility of a concept, conveyed with optimal messages, increases as transmitter and receiver capacity grows: does it asymptote below or above a threshold? Measuring 1.2k concepts in eight domains, we find concepts that (a) can be fully articulated; these concepts usually depend on local patterns, e.g. "joke ends with surprise"; (b) might be articulated by future, stronger models; these rely on whole-text structural patterns, e.g. "story's worldbuilding coherence"; (c) concepts that face articulability limits; these concepts' meaning, we find, is tied to identity and social-group awareness, e.g. "comic voice". Interesting, these same concepts are not all challenging for humans: humans have moderate-to-high levels of agreement in 76% of concepts that LLMs struggle with; articulation limits, we find, are executor relative. Finally, we characterize the nature of improvement in each case: while recoverable concepts can be articulated through definitions and explanations, irrecoverable concepts cannot be transmitted, even with examples, and might require novel learning mechanisms.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a rigorous framework for measuring the limits of language articulation in LLMs, combining novel prompt optimization with a transmission metric to categorize concepts by their articulability. By demonstrating that certain social and identity-based concepts are fundamentally harder for LLMs to transmit than others, despite high human agreement, the work provides critical insights into the boundaries of language-based AI control and alignment.
Methodology
The paper introduces a novel framework for measuring the "articulability" of concepts in LLMs, decomposing the problem into message optimization and transmission measurement. The core methodological contribution is $\epsilon$-estimation, a prompt optimization algorithm that uses capture-recapture statistics to estimate the upper bound of prompt performance by modeling the space of "semantic units." This is a creative application of ecological statistics to prompt engineering. The second contribution is the "reconstruction bottleneck," a transmission metric where a receiver LLM attempts to reconstruct the sender's prompt from labeled examples. The methodology is rigorous, with clear definitions of upper/lower bounds (planted rules, permutation nulls) and a strong theoretical grounding in information theory (mutual information). The approach effectively separates the capability of the model from the clarity of the language used to convey a concept.
Results & evaluation
The experiments are extensive, covering 1.2k concepts across eight diverse domains (humor, math, code, etc.). The paper compares $\epsilon$-estimation against SOTA prompt optimizers like GEPA, showing up to 14% improvement. The scaling laws analysis is particularly strong, categorizing concepts into three regimes: fully articulable, potentially articulable with scaling, and fundamentally limited. The finding that "identity and social-group awareness" concepts are the hardest to articulate, while humans show high agreement on them, is a significant and surprising empirical result. The use of human inter-rater agreement (Krippendorff's alpha) as a ground truth for concept clarity adds robustness.
Reproducibility
The paper provides detailed descriptions of the unit mining process, the capture-recapture estimator, and the reconstruction protocol. The use of standard benchmarks (AIME, HotpotQA, IFBench) and public datasets (McSweeney's, Reddit) aids reproducibility. However, the specific implementation details of the "unit" extraction and the exact prompt templates for the reconstruction bottleneck are not fully specified in the provided text, which may hinder exact replication. The code availability is not explicitly stated in the snippet, but the level of detail suggests it is likely released.
Limitations
The primary limitation is the reliance on LLMs as both transmitters and receivers, which may not fully capture human-like articulation limits. The "unit" decomposition assumes that prompts can be atomized into independent semantic units, which may not hold for highly contextual or stylistic concepts. The capture-recapture estimator assumes exchangeability of units, which is a strong assumption. Additionally, the paper focuses on binary labeling tasks, which may not generalize to more complex, multi-class, or generative articulation tasks.
Broader impact
This paper has significant implications for AI alignment and interpretability. By identifying which concepts are "inarticulable" to LLMs, it highlights the limits of using language as a medium for control and evaluation. This is crucial for developing AI systems that operate on nuanced, subjective, or socially constructed concepts. The framework could be used to design better rubrics for LLM judges and to identify areas where human feedback is essential. It also provides a new lens for understanding the gap between human and machine understanding of language. The paper introduces a rigorous framework for measuring the limits of language articulation in LLMs, combining novel prompt optimization with a transmission metric to categorize concepts by their articulability. By demonstrating that certain social and identity-based concepts are fundamentally harder for LLMs to transmit than others, despite high human agreement, the work provides critical insights into the boundaries of language-based AI control and alignment.
Author of this paper? Send a correction.