A Sense of Proprioception in Language Models
Abstract
In humans, proprioception corresponds to the sense of self-movement or position, e.g. understanding the position of one's arm. This sense is often less well-understood and in popular understanding sometimes conflated with introspection. But, notably, proprioception is a separate sense, with dedicated receptors and verifiable outcomes. On the other hand though, language models cannot observe their own internal states during training (and for example answer the question which experts activated in a certain layer of an MoE) and so cannot accurately report on them. Yet, such information is, by virtue of being internal to the model, both accessible, and readily constructed into verifiable signals. We argue that this provides the basis for a digital analog of proprioception. As such, we train models through SFT and on-policy training to equip them with this ability. We show that modern models can be trained to understand a range of subtypes of proprioception, such as routing decision awareness, token entropy, decision depth or logit lens information, without affecting model capabilities. We then run these models through a broad evaluation of situational awareness, legibility, and epiphenomenality, to evaluate in what ways access to proprioception changes model behavior.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a rigorous framework for training language models to possess "digital proprioception," enabling them to accurately report on verifiable internal states such as MoE routing decisions and logit distributions. By distinguishing this capability from unverifiable introspection and employing on-policy training to prevent label staleness, the authors demonstrate that models can acquire precise self-knowledge without degrading general capabilities, offering a promising new avenue for enhancing model interpretability and safety through structured self-monitoring.
Methodology
The paper proposes a novel training paradigm for "digital proprioception," distinguishing it from introspection by focusing on verifiable, internal, non-transparent states. The methodology involves a two-stage training process: Supervised Fine-Tuning (SFT) on templated questions and On-Policy SFT (inspired by DAgger) where labels are generated from the live forward pass to avoid stale labels. This on-policy approach is critical for maintaining the integrity of the proprioceptive signal as model weights update. The authors address practical challenges like kernel non-determinism in MoE routing and multi-token answer supervision. The theoretical framing, separating proprioception (verifiable internal states) from introspection (unverifiable self-reports), is a strong conceptual contribution that clarifies a muddy area of AI safety research.
Results & evaluation
The experiments are rigorous, testing three distinct domains of proprioception: MoE routing decisions, token logits/entropy, and logit lens information. The models used (Llama 2.5, OLMoE, Qwen, GLM) are modern and relevant. Key results show that models can learn to report these internal states with high accuracy without degrading general capabilities. The evaluation of behavioral changes (situational awareness, legibility) is thorough, showing that proprioception training does not negatively impact interpretability (probe accuracy) or safety metrics. The edge case analysis regarding routing non-determinism is particularly insightful, demonstrating the limits of the model's self-knowledge when the underlying computation is unstable.
Reproducibility
The paper provides detailed descriptions of the training setup, including loss weighting strategies, context lookup tables to prevent shortcut learning, and synchronization protocols for on-policy training. However, specific hyperparameters for the on-policy stage and the exact implementation of the "live" label extraction are described at a high level. The code is not explicitly linked in the provided text, but the methodological details are sufficient for a skilled practitioner to attempt reproduction. The reliance on specific MoE architectures (like Qwen3.6) may limit immediate reproducibility for those without access to such models, but the principles are generalizable.
Limitations
The primary limitation is the scope of "proprioception" defined here, which is restricted to valence-neutral, mechanistic queries. It does not address more complex or subjective internal states. Additionally, the non-determinism of MoE routing kernels poses a fundamental upper bound on the accuracy of routing proprioception, as noted in the paper. The behavioral changes observed are subtle, and it remains unclear if this training leads to significant long-term safety benefits or risks, as the effects on "legibility" were neutral.
Broader impact
This work has significant implications for AI safety and interpretability. By providing a method to train models to report on their internal states with verifiable ground truth, it offers a new tool for debugging and monitoring LLMs. It challenges the notion that self-knowledge in LLMs is purely mimetic or unreliable, suggesting that specific, structured self-knowledge can be engineered. This could lead to more transparent and controllable AI systems, where models can explicitly report on their decision-making processes (e.g., which experts were activated) in a way that is auditable. The paper introduces a rigorous framework for training language models to possess "digital proprioception," enabling them to accurately report on verifiable internal states such as MoE routing decisions and logit distributions. By distinguishing this capability from unverifiable introspection and employing on-policy training to prevent label staleness, the authors demonstrate that models can acquire precise self-knowledge without degrading general capabilities, offering a promising new avenue for enhancing model interpretability and safety through structured self-monitoring.
Author of this paper? Send a correction.