Axiomatic World Modeling for Physics Reasoning
Abstract
Recent research-level physics-reasoning benchmarks for LLMs reveal a failure mode that lies not in knowledge or execution, but in modeling the world being reasoned about. Models often silently alter given quantities, fabricate laws, or inject unstated assumptions to force a problem into familiar patterns, and then reason inside this corrupted world. We define this premise-level divergence as modeling drift: the gap between a model's internal axiomatic world model (AWM) and the true physical world specified in the problem. Although RLVR and PRM have yielded substantial gains in mathematics and code, their reward shapes do not directly supervise world modeling in axiomatic domains such as physics. We propose Reinforcement Learning with Axiomatic World Modeling (RLAWM), which optimizes the policy by penalizing divergence between its inferred axiomatic world model and the true physics world. RLAWM probes physics reasoning at two hierarchical levels (modeling and solving), each scored by a physically grounded reward. The modeling reward targets abstract knowledge formulation, evaluating whether the model captures the true axiomatic world before attempting calculation. The solving reward targets instantiated knowledge reasoning, checking whether the model's concrete derivation remains consistent with the axiomatic world. This two-level structure separates what world the model believes it is reasoning in from how it reasons within that world. The framework operates exclusively during training, with no extra cost during inference. RLAWM improves performance by up to +12.2 points on PhysReason and +5.2 on PHYSICS over the baseline. It also generalizes zero-shot across difficulty, source, and domain, with a 2.45x gain and +10 points over the strongest baseline on olympiad-level physics and math.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a novel RL framework, RLAWM, that mitigates "modeling drift" in physics reasoning by decomposing the task into axiomatic world modeling and solving, using a dual reward system of motif-based fidelity scoring and executable code verification. By leveraging the verification-derivation asymmetry of physics, the method provides dense, physically grounded training signals that expand the reasoning boundary and improve out-of-distribution generalization, representing a substantial step forward in domain-specific LLM reasoning.
Methodology
The paper proposes RLAWM, a reinforcement learning framework that explicitly targets "modeling drift" in physics reasoning. The core innovation is the decomposition of the reasoning process into a [modeling] block (externalizing the Axiomatic World Model) and a [solving] block (pseudocode derivation). The methodology introduces two distinct reward signals: a Modeling Reward ($R_M$) based on Physics Modeling Fidelity (PMF) using motif extraction (QEL, VCL, AEB) and an LLM-based soft matcher, and a Solving Reward ($R_S$) that translates pseudocode to executable Python and runs programmatic unit tests against a library of physics law templates. This approach leverages the verification-derivation asymmetry in physics, where checking invariants is easier than deriving them. The use of a frozen translator (Qwen2.5-Coder) to bridge natural language/pseudocode and executable code is a practical and effective design choice.
Results & evaluation
The experiments are rigorous and comprehensive. The authors evaluate on in-distribution benchmarks (PhysReason, PHYSICS) and out-of-distribution benchmarks (OlympiadBench, PHYBench, AIME) across three model families (Qwen3-VL, Gemma-3). Key results include a +12.2 point improvement on PhysReason and a significant reduction in "diversity collapse" (SubPass@k curves continue to improve with $k$, unlike RLVR baselines). Ablations clearly demonstrate the necessity of both $R_M$ and $R_S$, showing they target orthogonal failure modes (premise errors vs. execution errors). The analysis of the training signal density (reducing zero-gradient groups from 29.9% to 9.3%) provides strong evidence for the effectiveness of the dense reward signal.
Reproducibility
High. The paper provides detailed descriptions of the reward components, the motif extraction schema, and the checker library construction. A GitHub repository is provided with code and models. Hyperparameters and compute requirements (4x B200 GPUs, ~6 hours) are specified. The use of standard models (GPT-4o-mini for extraction, Qwen for translation) ensures accessibility.
Limitations
The method relies on a frozen LLM for motif extraction and a separate translator, adding training complexity and cost. The checker library is static and may not cover all physics domains equally well (noted lower performance in Quantum/Stat Mech compared to E&M/Optics). The approach is currently specialized for physics; while the authors suggest extension to math, the specific "axiomatic world model" structure may need adaptation for other domains.
Broader impact
This paper offers a significant conceptual shift in how LLMs are trained for scientific reasoning, moving beyond outcome-based rewards to process-based world-model alignment. The "modeling drift" framework is likely to influence future work in scientific LLMs. The technique of using executable code as a reward signal for reasoning steps is a robust pattern that could be adapted to other formal domains like mathematics or logic. The paper introduces a novel RL framework, RLAWM, that mitigates "modeling drift" in physics reasoning by decomposing the task into axiomatic world modeling and solving, using a dual reward system of motif-based fidelity scoring and executable code verification. By leveraging the verification-derivation asymmetry of physics, the method provides dense, physically grounded training signals that expand the reasoning boundary and improve out-of-distribution generalization, representing a substantial step forward in domain-specific LLM reasoning.
Author of this paper? Send a correction.