Build, Don't Play: Self-Improving LLM Research Agents Develop a Hearthstone Battlegrounds AI That Rivals Top Players
Abstract
Can LLM agents carry a hard real-world problem from raw data to a system that holds its own against people? We study this in Hearthstone Battlegrounds, an eight-player auto-battler with stochastic shops and battles, hidden information and nearly a thousand interacting cards. We present AutoTavern, a self-improving research system: specialised LLM research agents share a typed research memory, every claim passes verification gates, and live games close the loop, so the agents improve the simulator, opponents and policy and, from the same feedback, their own research process; a human sets goals and rules on ambiguous mechanics. In 20 days AutoTavern built TavernGym, the first faithful, full-content reproduction of the live game, and delivered 23 versions of TavernAgent, trained with PPO against opponents calibrated to real games. TavernAgent plays complete games on the live ladder by itself: its rating rose from 6,000 to about 15,000 and, on the public leaderboard's measure, its last 20 records average a placement of 1.60 with 80% first places, better than the server's ten highest-rated players on every statistic (2.34, 59%). TavernBench, 960 tasks in 40 scenarios, shows that the system the agents built is the strongest player: TavernAgent answers 91% of the play tasks against at most 63% for frontier LLMs, and decides in 0.03 s where they need up to 26 s; within the second a live turn allows, it keeps 90% and no LLM more than 26%. LLM agents build far better than they play. We will open-source TavernBench and TavernGym at https://github.com/MCG-NJU/AutoTavern.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a self-improving LLM agent framework that autonomously builds a faithful simulator and trains a reinforcement learning agent to achieve top-tier performance in Hearthstone Battlegrounds, outperforming human experts and demonstrating that LLMs are more effective as system builders than direct players.
Methodology
The paper introduces AutoTavern, a self-improving research framework where LLM agents manage the entire lifecycle of an RL system: building the environment (TavernGym), training the policy (TavernAgent), and refining the research process itself via a typed memory and verification gates. The core methodological innovation is the "reality-anchored self-improvement" loop, which uses live game data to correct simulator drift and opponent models. The use of PPO with an entity transformer is standard, but the orchestration of agents to build and debug the environment autonomously is novel. The verification gates (G1-G5) are a robust engineering solution to the hallucination and drift problems inherent in LLM-generated code and data.
Results & evaluation
The experimental results are exceptionally strong. The agent achieves a rating of ~15,000 on the live Hearthstone Battlegrounds ladder, outperforming the top 10 human players on recent placement statistics (1.60 avg placement vs 2.34). The creation of TavernBench, a 960-task benchmark, provides a rigorous evaluation of LLMs' ability to play the game directly, showing a clear gap between LLM reasoning and the specialized agent's performance. The ablation studies on opponent calibration and the "lessons learned" section provide deep insight into the failure modes and corrections of the self-improving loop.
Reproducibility
The authors commit to open-sourcing TavernGym and TavernBench, which significantly boosts reproducibility. The paper provides detailed descriptions of the agent roles, memory structures, and verification gates. However, the specific prompts and the "agentic research framework" used to orchestrate the LLMs are not fully detailed in the text, though the code repository is provided. The compute requirements (1,900 GPU-hours for main training) are high but documented.
Limitations
The system is highly specific to Hearthstone Battlegrounds, limiting immediate generalizability to other games without significant adaptation. The reliance on a human to press shortcut keys for live deployment introduces a bottleneck and potential for error, though the paper argues this is a client limitation rather than an agent one. The "self-improvement" is bounded by the initial LLM capabilities and the verification gates; it does not fundamentally alter the underlying RL algorithms.
Broader impact
This paper demonstrates a new paradigm for AI research: using LLM agents to automate the construction of complex, stochastic environments and the training of agents within them. It challenges the notion that LLMs are poor game players by showing they are excellent *builders* of game-playing systems. The open-sourcing of a faithful simulator for a complex commercial game is a significant contribution to the community, enabling further research in multi-agent RL and LLM-agent interaction. The paper presents a self-improving LLM agent framework that autonomously builds a faithful simulator and trains a reinforcement learning agent to achieve top-tier performance in Hearthstone Battlegrounds, outperforming human experts and demonstrating that LLMs are more effective as system builders than direct players.
Author of this paper? Send a correction.