RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms
Abstract
LLM-guided program evolution has enabled discoveries in mathematics and computational optimization, raising the prospect of reinforcement learning (RL) algorithms that self-evolve to improve how agents learn. However, realizing this prospect faces two obstacles. Joint search over coupled algorithmic components is difficult to scale: simultaneous changes can disrupt learning, while isolated changes overlook their dependencies. Evaluating candidate algorithms also requires costly training, with fitness remaining uncertain across random seeds. We introduce RLDiscover, a framework for the self-evolution of model-free deep RL algorithms. Progressive Co-Evolution advances from targeted component edits to joint evolution, while Progressive Probabilistic Evaluation balances search breadth and evaluation fidelity through staged training and repeated evaluation. Experiments across SAC, PPO, and DQN on four benchmark suites show substantial improvements in mean return, with per-family median gains of 32%-84% and a peak return ratio of approximately 363x over a near-zero baseline. These gains include transitions from failed learning to successful task completion, and improvements persist when evolution starts from stronger open-source implementations. On measured SAC locomotion runs, evaluation uses approximately one-fifteenth the estimated compute required to fully evaluate the same candidate pool. Remarkably, independent searches repeatedly discover interpretable combinations of adaptive robust losses, progress-dependent value targets, and running statistics, with selected programs transferring to unseen tasks. These findings point toward a broader role for self-evolution in AI: discovering interpretable algorithms that improve how agents learn.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a rigorous and effective framework for evolving RL algorithms using LLMs, demonstrating significant performance gains and discovering interpretable, transferable mechanisms. Its combination of structured search (PCE) and efficient evaluation (PPE) addresses key bottlenecks in program evolution for RL, offering a practical and impactful contribution to the field of automated machine learning.
Methodology
The paper introduces RLDiscover, a framework for the self-evolution of model-free deep RL algorithms using LLMs. The core methodological contribution is the combination of Progressive Co-Evolution (PCE) and Progressive Probabilistic Evaluation (PPE). PCE addresses the difficulty of joint search over coupled algorithmic components by first using a UCB-inspired bandit to identify high-value components for isolated mutation, then transitioning to joint co-evolution. PPE addresses the high cost and noise of RL evaluation by implementing a multi-fidelity funnel (L0: 50k steps, L1: 200k steps, L2: 1M steps) that allocates compute efficiently. The decomposition of RL algorithms into five specific components (state encoding, value target, critic loss, policy loss, exploration bonus) is a structured and logical choice that enables targeted LLM mutations. The methodology is sound, leveraging established concepts (bandits, multi-fidelity optimization) in a novel application context.
Results & evaluation
The experiments are extensive, covering three major RL families (SAC, PPO, DQN) across 26 task-algorithm pairs from four benchmark suites (DMControl, Gymnasium, Meta-World, MetaDrive). The results show substantial improvements, with median gains of 32-84% over baselines. A particularly strong aspect is the "SOTA-init" experiment, where evolution starts from strong open-source implementations (CleanRL, denisyarats/pytorch_sac) and still yields improvements, validating that the gains are not merely due to fixing a weak baseline. The ablation studies are rigorous, comparing the full method against variants removing PCE or PPE components, and analyzing the trade-off between search breadth and evaluation fidelity. The cross-task transfer analysis provides evidence that the evolved programs contain generalizable mechanisms rather than just task-specific hacks.
Reproducibility
The paper provides detailed descriptions of the component interfaces, the PCE algorithm, and the PPE promotion rules. It explicitly states that code and data will be released upon acceptance. The "AI use statement" clarifies that the LLM is the mutation operator, not just a writing aid, and details the audit process for generated programs. The reproducibility statement is thorough, specifying training budgets, seeds, and evaluation protocols. The inclusion of per-seed results and the discussion of variance (e.g., the bimodal outcome of MountainCar) enhances transparency.
Limitations
The authors acknowledge that PPO and DQN results are suboptimal due to budget constraints carried over from SAC. The five-component decomposition is only active at the L0 screening stage, limiting structured exploration at higher fidelities. L0 screening at 50k steps may discard strategies with slow warm-up. The LLM identity is withheld for anonymity, which is standard for submission but limits immediate reproducibility of the exact mutation behavior until publication.
Broader impact
This paper contributes to the growing field of LLM-driven algorithm discovery. By demonstrating that LLMs can evolve interpretable, reusable RL mechanisms (like robust critic losses and adaptive Q-blending) that transfer across tasks, it suggests a path toward automated RL algorithm design. The findings on "convergent discovery" of specific mechanisms provide insights into what makes RL algorithms robust, potentially guiding human designers. The efficiency gains from PPE (15x reduction in compute) make this approach more practical for broader adoption. The paper presents a rigorous and effective framework for evolving RL algorithms using LLMs, demonstrating significant performance gains and discovering interpretable, transferable mechanisms. Its combination of structured search (PCE) and efficient evaluation (PPE) addresses key bottlenecks in program evolution for RL, offering a practical and impactful contribution to the field of automated machine learning.
Author of this paper? Send a correction.