Compositional RL: Evaluating and Training LLMs for Collaborative Reasoning
Abstract
Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. For a model inside a system, what matters most is not how well it performs on its own, but how much the system gains from its presence. We call this a model's compositional strength and formalize it using the Shapley value of a cooperative game over coalitions of a model pool, and use a model's leave-one-out contribution to the full coalition as a tractable proxy, requiring n+1 rather than 2^n system evaluations. Evaluating individual vs. compositional strengths across five model pools, twelve collaboration methods, and twelve datasets, we find them largely uncorrelated: in 12 of 15 (pool, method-family) settings, individual strength does not translate into compositional advantages, and the coupling is weakest among general-purpose models that practitioners are most likely to deploy. Standalone benchmarks are therefore often poor proxies for models' value in collaborative systems. We then propose compositional reinforcement learning (C-RL), which trains a policy inside a live multi-model system and rewards the outcome produced by the system as a whole, rather than the policy's correctness in isolation. The leave-one-out baseline in Shapley values cancels under a group policy gradient, so this needs no additional rollout cost beyond RL on the collaborative system's outcome and never requires running a coalition without the policy. Across three collaboration protocols and eight tasks spanning reasoning, coding, and scientific discovery, C-RL produces the strongest collaborator in 21 of 24 settings, lifting collaborative performance by +4.0 to +5.4 pts on average (~3x the gain of baselines), up to +13.1 pts.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper makes a significant contribution by formalizing compositional strength via Shapley values and demonstrating that standard individual benchmarks are poor predictors of a model's value in collaborative systems. By proposing Compositional Reinforcement Learning (C-RL), which efficiently trains models to maximize their contribution to a multi-agent system's outcome, the authors provide a novel and effective method for developing specialized collaborator models that generalize across tasks and protocols.
Methodology
The paper introduces a rigorous formalization of "compositional strength" using Shapley values from cooperative game theory, adapted for LLM collaborations. The key methodological contribution is the derivation of a tractable Leave-One-Out (LOO) estimator that approximates the Shapley value with linear complexity ($n+1$) rather than exponential ($2^n$). The proposed training method, Compositional Reinforcement Learning (C-RL), is elegant in its use of group-relative policy gradients (GRPO) to cancel out the baseline cost of the LOO marginal, allowing the policy to be trained on system-level rewards without extra rollouts. The credit assignment mechanism, which attributes system-level rewards to individual policy generations within a multi-turn collaboration, is a significant technical challenge addressed effectively.
Results & evaluation
The evaluation is extensive, covering 5 model pools, 12 collaboration methods, and 12 datasets. The finding that individual strength and compositional strength are largely uncorrelated is a strong empirical contribution. The training experiments demonstrate that C-RL outperforms baselines (including standard RL and collaborative fine-tuning) in 21/24 settings. The generalization tests to unseen datasets and collaboration protocols are convincing, showing that C-RL teaches transferable collaboration skills rather than overfitting to a specific setup. The inclusion of proprietary frontier models as collaborators adds practical relevance.
Reproducibility
The paper promises code release upon acceptance. The use of proprietary APIs (GPT-5.5, etc.) introduces a reproducibility caveat noted by the authors, as these models may change over time. However, the open-source components (Qwen, open-weight collaborators) allow for partial reproduction. The detailed description of the MoCo toolkit and specific hyperparameters aids reproducibility.
Limitations
The reliance on proprietary models for some experiments limits full reproducibility. The computational cost of running multi-model collaborations for RL training is high, potentially limiting adoption to well-resourced labs. The credit assignment is uniform across turns, which the authors acknowledge as a simplification that could be improved. The study focuses on a specific set of collaboration protocols; while generalization is tested, the scope is not exhaustive of all possible multi-agent architectures.
Broader impact
This work shifts the focus of LLM evaluation and training from individual capability to system-level contribution. It provides a theoretical foundation for understanding multi-agent dynamics and a practical method for training better collaborators. This is highly relevant as LLMs are increasingly deployed in compound systems. The finding that standard benchmarks do not predict collaborative value is a critical insight for practitioners building multi-agent systems. The paper makes a significant contribution by formalizing compositional strength via Shapley values and demonstrating that standard individual benchmarks are poor predictors of a model's value in collaborative systems. By proposing Compositional Reinforcement Learning (C-RL), which efficiently trains models to maximize their contribution to a multi-agent system's outcome, the authors provide a novel and effective method for developing specialized collaborator models that generalize across tasks and protocols.
Author of this paper? Send a correction.