Test-Time Self-Play: Scaling without a Verifier
Abstract
Test-time scaling methods spend additional compute at deployment to improve a large language model's performance on a hard target problem. Most methods rely on a verifier that scores candidate solutions for correctness, using it to select among or refine the model's outputs. In many deployment regimes, however, a strong verifier on the target problem is unavailable. We introduce Test-Time Self-Play (TTSP), an approach to test-time scaling using asymmetric self-play without reliance on a verifier. On a difficulty-calibrated subset of LiveCodeBench v6, TTSP is the best-performing method at every verifier strength we evaluate, and the gap to the strongest baseline widens as the verifier weakens; in the no-verifier limit, TTSP yields a 35\% absolute improvement in mean@8. The gains hold across a range of verifier strengths, except when the base policy has near-zero coverage of the correct answer.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Test-Time Self-Play (TTSP) enables robust test-time scaling for LLMs in the absence of strong verifiers by leveraging asymmetric self-play on synthetic problems. The paper demonstrates that by decoupling the learning signal from the target problem's verifier and instead using execution feedback on self-generated, target-conditioned synthetic problems, models can significantly improve performance on hard tasks even when no reliable way to check answers exists, outperforming traditional methods like rejection sampling and majority voting across all verifier strengths.
Methodology
The paper introduces Test-Time Self-Play (TTSP), a method for improving LLM performance on specific hard problems without access to a strong verifier or auxiliary data. The core innovation is the adaptation of asymmetric self-play (typically used for broad capability training) to the single-target test-time setting. By using a "proposer" policy to generate synthetic problems conditioned on the target and a "solver" policy to learn from execution feedback on those synthetic problems, the method decouples the learning signal from the target problem's verifier. This is a clever architectural choice that addresses a critical gap in test-time scaling literature, which often assumes access to ground truth or strong verifiers. The use of LoRA adapters for both roles over a shared backbone is a practical and efficient implementation detail.
Results & evaluation
The evaluation is rigorous and well-designed. The authors use LiveCodeBench v6, a recent and challenging benchmark, and specifically curate a subset (HardLCBv6) where the base model has non-zero but imperfect coverage, which is the ideal regime for test-time scaling. The key strength of the experimental design is the "verifier strength sweep," where they artificially degrade the verifier (from oracle to no-verifier) to demonstrate TTSP's robustness. The results clearly show that while baselines (rejection sampling, majority voting, context optimization) degrade as the verifier weakens, TTSP remains stable and outperforms them, especially in the no-verifier limit. The ablation on problem difficulty (MidHard vs. VeryHard) correctly identifies the method's limitation: it fails when the base model has near-zero coverage of the solution.
Reproducibility
The paper provides detailed algorithmic descriptions, hyperparameters, and implementation details in the appendices. The use of standard components (Qwen3-8B, LoRA, GRPO) and public benchmarks (LiveCodeBench) enhances reproducibility. The specific prompts and filtering heuristics are described, though the exact code is not provided in the text, the level of detail is sufficient for a skilled practitioner to reimplement the core logic.
Limitations
The primary limitation is the "coverage requirement." TTSP cannot solve problems where the base model has no idea of the solution (VeryHardLCBv6). It is a method for refining existing knowledge, not discovering new capabilities. Additionally, the method is computationally expensive per target problem, as it involves training two policies for each individual problem, which may not be feasible for high-throughput deployment. The reliance on an interactive environment (Python interpreter) also limits its applicability to domains where execution feedback is not easily available.
Broader impact
This work has significant implications for the deployment of LLMs in scenarios where verification is costly or unavailable. It provides a pathway for models to "reason their way" to a solution by generating their own curriculum, which is a step towards more autonomous and self-improving AI systems. The findings challenge the assumption that test-time scaling requires strong external supervision, potentially influencing future research in unsupervised test-time adaptation and self-play. Test-Time Self-Play (TTSP) enables robust test-time scaling for LLMs in the absence of strong verifiers by leveraging asymmetric self-play on synthetic problems. The paper demonstrates that by decoupling the learning signal from the target problem's verifier and instead using execution feedback on self-generated, target-conditioned synthetic problems, models can significantly improve performance on hard tasks even when no reliable way to check answers exists, outperforming traditional methods like rejection sampling and majority voting across all verifier strengths.
Author of this paper? Send a correction.