AutoEnvScaling: Automating the Data Flywheel with Terminal Agents
Abstract
Terminal agents solve tasks in software engineering and scientific research by executing commands in a terminal. Training them via reinforcement learning (RL) requires diverse and verifiable environments, but these environments are still mostly designed by human researchers, so their supply grows only as fast as researchers can work. We introduce AutoEnvScaling, which treats environment design as a terminal task and uses agents to automate the data flywheel. A proposer agent builds new RL environments from the rollouts of the solver being trained, and the solver's new rollouts guide the next round. By turning environment generation into a terminal task, AutoEnvScaling increases the valid-environment rate by up to 25 times compared to prompt-only generation, at up to 75% lower cost per valid environment. We show that training Qwen3.5-35B-A3B on AutoEnvScaling environments from GPT-5.6-sol improves its Terminal-Bench 2.1 score by 4.3 points over the matched Tmax baseline. The broader goal of AutoEnvScaling is to enable recursive self-improvement (RSI): by making both environment generation and solving terminal tasks, it lets one model co-evolve in both roles. We show that Qwen3.6-35B-A3B, acting in both roles, gains 12.2 points on Terminal-Bench 2.1 and raises its Terminal-Bench 4.0 score from 1.6 to 3.2. The pipeline also generalizes across agentic domains, generating environments for software engineering, computer use, professional work, and GPU kernel programming. Overall, these results show a path toward agents building their own training environments, reducing human reliance and taking a core step toward RSI.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
Methodology
The paper proposes AutoEnvScaling, a framework that reframes the generation of RL training environments as a terminal agent task. Instead of using static pipelines or simple prompt-based generation, the method employs a "proposer" agent operating within a sandboxed terminal environment. This agent reads solver rollouts, identifies failure modes, and iteratively builds, tests, and calibrates new Harbor tasks (instructions, environments, reference solutions, tests). The core innovation is the "data flywheel" where the solver's performance directly informs the proposer's next environment generation, and the proposer itself can be trained (via DPPO) to improve at environment design. This creates a recursive self-improvement (RSI) loop where a single model can co-evolve as both the task designer and the solver. The methodology is technically sound, leveraging existing RL techniques (DPPO) but applying them to a novel multi-agent, self-play context with distinct reward structures for design vs. solving.
Results & evaluation
The experiments are extensive and well-controlled. The authors compare their method against prompt-only and feedback-driven baselines across eight frontier models, showing a 25x increase in valid environment rates. They demonstrate downstream utility by training Qwen3.5 models on AutoEnvScaling-generated environments, showing significant gains (4.3-7.4 points) on Terminal-Bench 2.1 over static Tmax baselines. The RSI experiment is particularly compelling, showing a 12.2-point gain when the model acts as both proposer and solver. The inclusion of a qualitative study on "learning behavior" (asking for help) further validates the quality of the generated curricula. The use of matched baselines (regenerating Tmax pools with the same proposer) ensures fair comparison.
Reproducibility
The paper provides detailed descriptions of the Harbor framework, the sandbox setup, and the validation/calibration criteria. However, specific hyperparameters for the DPPO training and the exact implementation details of the "harness optimization" are relegated to appendices or referenced generally. The reliance on proprietary frontier models (GPT-5.6, Claude Opus 5) for the proposer role limits immediate reproducibility for those without access, though the open-weight Qwen experiments provide a path for replication. The code and environment definitions are not explicitly linked in the provided text, which is a minor gap.
Limitations
The primary limitation is the heavy reliance on strong frontier models to bootstrap the proposer role, which may not be accessible to all researchers. The RSI results, while impressive, are demonstrated on a specific model family (Qwen) and benchmark (Terminal-Bench), leaving open questions about generalization to other domains or model architectures. The paper also acknowledges that the validation checks cannot catch all types of errors (e.g., shared mistakes between reference solution and tests), which could lead to subtle reward hacking if not carefully monitored.
Broader impact
This work has significant implications for the scalability of RL training for agents. By automating the most labor-intensive part of the RL pipeline (environment design), it could drastically reduce the cost and time required to train capable agents. The concept of RSI through self-generated curricula is a major step toward autonomous AI improvement, with potential applications in scientific discovery, software engineering, and other complex problem-solving domains. It shifts the paradigm from human-designed curricula to agent-designed curricula, potentially enabling agents to discover novel skills and strategies that humans might not anticipate. AutoEnvScaling automates the generation of RL training environments by treating environment design as a terminal agent task, enabling a recursive self-improvement loop where agents co-evolve as both task designers and solvers. The paper demonstrates that this approach significantly increases the rate of valid environment generation and improves downstream agent performance on terminal benchmarks, offering a scalable path toward reducing human reliance in agent training pipelines.
Author of this paper? Send a correction.