RSI Arena: Towards Recursive Self-Improvement with Human Feedback
Abstract
Can AI agents autonomously train useful language models and improve them using feedback from users? We introduce RSI Arena, a two-stage study of autonomous post-training as a step toward recursive self-improvement. Ten frontier models act as training agents, each starting from NVIDIA Nemotron 3.5 Lightning 30B-A3B Base. During a six-day first stage, each agent has 1,000 GPU-hours and $300 in API credit on a shared 64-GPU cluster, independently selecting data, implementing training procedures, and choosing checkpoints. We evaluate their models on 20 benchmarks and through blind human evaluations. All ten agents produce working assistants, with the strongest model, trained by GPT-6 Astra, closing a median of 76% of the base-to-Instruct gap across 12 capability benchmarks. Its human-preference rating is also close to NVIDIA Instruct after statistical adjustment for answer length and formatting. In Stage 2, each agent receives the comparison records and votes involving its Stage 1 model and is given three additional days for further training. Using NVIDIA Instruct as a common reference across the two evaluation rounds, we observe an improvement from Stage 1 to Stage 2 in the style-adjusted relative rating of eight of the ten models. We show that benchmark gains were modest, with no clear link to preference gains, while some agents encouraged longer responses and one model showed increased harmful compliance on StrongREJECT. RSI Arena provides an auditable setting for studying how human feedback can guide autonomous model development. It takes a step toward recursive self-improvement with human feedback (RSIHF), in which people help define what improvement means for the AI systems they use.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents RSI Arena, a comprehensive study demonstrating that frontier AI agents can autonomously post-train language models to near-reference quality and improve them using human feedback, while revealing critical insights into the relationship between compute, benchmarks, and human preferences. This work is a significant step toward recursive self-improvement, providing an auditable and realistic setting for studying how AI agents can develop models under resource constraints. The methodology's emphasis on human evaluation and safety analysis sets a new standard for evaluating autonomous AI research agents. The finding that compute usage does not predict model quality is a surprising and important result that challenges common assumptions in ML engineering. The paper's contribution to the field is both practical (providing a benchmark and platform) and conceptual (defining RSIHF), making it a highly impactful work for researchers in AI safety, autonomous agents, and ML engineering.
Methodology
The paper introduces a rigorous, two-stage experimental framework for evaluating autonomous AI agents in the task of post-training large language models. The methodology is distinct from prior work (like PostTrainBench) by focusing on general-purpose assistant creation rather than single-benchmark optimization, and by incorporating a "Human Arena" for blind pairwise evaluation. The setup is realistic, imposing strict resource constraints (GPU hours, API credits, storage) and shared cluster contention, which forces agents to handle engineering failures and resource planning. The inclusion of a second stage where agents receive human feedback (votes) to improve their models is a novel approach to studying recursive self-improvement with human-in-the-loop feedback. The use of style-controlled Bradley-Terry ratings to mitigate length bias is a methodologically sound choice for fair comparison.
Results & evaluation
The experiments are extensive, involving 10 frontier models acting as agents over 6 days (Stage 1) and 3 days (Stage 2). The evaluation combines 20 automated benchmarks with human preference data from a live arena. Key findings include the surprising result that compute usage does not correlate with model quality (negative correlation), and that the best agent model (GPT-6 Astra) closed 76% of the gap to the reference model (NVIDIA Instruct). The paper provides detailed analysis of agent behaviors, such as data decontamination, end-of-turn token handling, and the impact of answer length on human ratings. The safety analysis reveals that without explicit safety constraints, some models regressed in harmful compliance, highlighting the limitations of preference-only optimization.
Reproducibility
The paper is highly reproducible in its spirit, as the authors release all model checkpoints, training trajectories, and the platform code on Hugging Face. The detailed description of the agent harness, budget constraints, and evaluation metrics allows other researchers to replicate the study. The use of public datasets (Tülu 3, UltraFeedback) and standard training methods (SFT, DPO, LoRA) further enhances reproducibility. The livestream and daily reports provide an unprecedented level of transparency into the agent decision-making process.
Limitations
The primary limitation is the scale: only 10 agents were tested, and each ran once, so the results may not generalize to all models or repeated runs. The base model (Nemotron 3.5) is relatively small (30B-A3B), so findings may differ for larger models. The human evaluation, while robust, is limited to the specific questions asked in the arena and may not capture all aspects of model utility. The paper acknowledges that benchmark gains and human preference gains are not always correlated, which complicates the interpretation of "improvement."
Broader impact
This paper has significant implications for the future of AI development. It demonstrates that AI agents can autonomously perform complex ML engineering tasks, including data selection, training, and evaluation, with minimal human intervention. The findings on the disconnect between compute usage and model quality suggest that future autonomous systems should focus on efficient experimentation and robust internal evaluation rather than simply scaling up resources. The concept of "Recursive Self-Improvement with Human Feedback" (RSIHF) provides a framework for aligning autonomous AI development with human values, which is crucial for the safe deployment of increasingly capable AI systems. The paper also highlights the importance of safety evaluations in autonomous training loops, as preference optimization alone can lead to safety regressions. The paper presents RSI Arena, a comprehensive study demonstrating that frontier AI agents can autonomously post-train language models to near-reference quality and improve them using human feedback, while revealing critical insights into the relationship between compute, benchmarks, and human preferences. This work is a significant step toward recursive self-improvement, providing an auditable and realistic setting for studying how AI agents can develop models under resource constraints. The methodology's emphasis on human evaluation and safety analysis sets a new standard for evaluating autonomous AI research agents. The finding that compute usage does not predict model quality is a surprising and important result that challenges common assumptions in ML engineering. The paper's contribution to the field is both practical (providing a benchmark and platform) and conceptual (defining RSIHF), making it a highly impactful work for researchers in AI safety, autonomous agents, and ML engineering.
Author of this paper? Send a correction.