Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
Abstract
Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper proposes Explanatory World Models (EWM) to improve credit assignment in RL by using LLM-generated causal explanations to redistribute delayed rewards. This approach significantly outperforms standard RL baselines in long-horizon tasks and industrial recommendation settings, demonstrating the potential of integrating causal reasoning with reinforcement learning to overcome sample inefficiency.
Methodology
The paper introduces Explanatory World Models (EWM), a framework that leverages Large Language Models (LLMs) to generate causal explanations for delayed rewards in Reinforcement Learning (RL). The core innovation is the use of these natural language explanations to identify non-temporal dependencies between actions and outcomes, thereby improving credit assignment. The method employs a "conservative hierarchical operator" to redistribute reward based on the LLM's identified causal links while preserving the undiscounted episode return. The theoretical grounding draws from Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations, providing a robust conceptual foundation for what constitutes a "useful" explanation in this context. The integration of LLMs as a credit assignment mechanism rather than just a policy or value estimator is a significant methodological shift.
Results & evaluation
The evaluation is comprehensive, spanning controlled long-horizon benchmarks (Crafter, WordCraft), a public sequential recommendation dataset (KuaiRand), and an industry-scale generative recommender. The results show substantial outperformance of PPO and LaRe, particularly in scenarios with high reward delay, which is the primary pain point the paper addresses. The inclusion of an industry-scale offline evaluation adds significant practical relevance, demonstrating that the method scales beyond toy problems. The gains in Recall and behavioral proxies on the industrial dataset are strong indicators of real-world utility.
Reproducibility
The paper provides detailed descriptions of the LLM variants and the reward redistribution operator. However, the specific prompts used to elicit explanations from the "frontier language model" and the exact hyperparameters for the conservative operator are critical for reproducibility. While the appendix is referenced, the main text suggests a level of detail that should allow for replication if the code is released (which is not provided in the text snippet). The use of a "frontier" model implies potential dependency on proprietary APIs, which could hinder open reproduction.
Limitations
The reliance on LLMs introduces inference costs and latency, which may be prohibitive for real-time online RL in some settings. The method's effectiveness depends heavily on the quality of the LLM's causal reasoning, which can be inconsistent. Furthermore, the "conservative" nature of the reward redistribution might limit the magnitude of improvement compared to more aggressive credit assignment strategies. The paper does not extensively discuss the failure modes of the LLM explanations or how the system handles incorrect causal attributions.
Broader impact
This work has significant implications for the field of RL, particularly for applications involving sparse or delayed rewards such as robotics, long-horizon planning, and recommendation systems. By bridging the gap between symbolic causal reasoning (via LLMs) and statistical learning (RL), it offers a pathway to more sample-efficient agents. The findings could inspire further research into hybrid neuro-symbolic approaches for credit assignment and the use of generative models as world models for causal inference. The paper proposes Explanatory World Models (EWM) to improve credit assignment in RL by using LLM-generated causal explanations to redistribute delayed rewards. This approach significantly outperforms standard RL baselines in long-horizon tasks and industrial recommendation settings, demonstrating the potential of integrating causal reasoning with reinforcement learning to overcome sample inefficiency.
Author of this paper? Send a correction.