Measuring AI Agents as Teammates with Collaborative Intelligence
Abstract
Progress in AI is often measured by what a single model can do on its own. Yet teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As AI begins to work in teams of agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades. Inspired by this, we introduce the Collaborative Intelligence Benchmark, which measures how well an AI agent works as a teammate. It contains 240 tasks across 8 dimensions of teamwork, from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and has a programmatically verifiable outcome. To isolate teamwork from individual capability, models are scored only on tasks that they can solve by themselves when they have full information. We evaluate 18 closed- and open-weight models and find that every model, across sizes and families, has clear weaknesses as a teammate, with the best model scoring only 82.2 out of 100. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure, reaching the best overall deal in only 14% of conflicts, and their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated. Crucially, general capability does not guarantee good teamwork, as models with nearly the same capability score differ by up to 11 points as teammates. The benchmark includes 160 public tasks with code and data, along with 80 held-out tasks for evaluation. As AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces the Collaborative Intelligence Benchmark, a rigorous framework for evaluating AI agents' teamwork capabilities across 8 dimensions, revealing that current frontier models have significant deficits in conflict resolution and supervisory tasks despite high individual capabilities. By isolating teamwork from raw intelligence and using programmatically verifiable tasks derived from real-world scenarios, the work provides a critical new metric for the developing field of multi-agent systems, highlighting that general capability does not guarantee effective collaboration and offering a reproducible baseline for future research.
Methodology
The paper introduces the Collaborative Intelligence (CI) Benchmark, a novel framework designed to evaluate AI agents specifically on their ability to function as effective teammates rather than standalone solvers. The methodology is grounded in decades of human team research, decomposing teamwork into 8 distinct dimensions (e.g., safety, conflict resolution, initiative, takeover). A key methodological strength is the isolation of teamwork from individual capability: models are only scored on tasks they can solve independently when given full information. This controls for raw intelligence, ensuring that failures are attributed to collaborative deficits (e.g., poor communication, inability to handle social pressure, or failure to verify weaker partners' work) rather than lack of domain knowledge. The tasks are derived from real-world records (aviation accidents, clinical cases) and existing benchmarks, ensuring ecological validity.
Results & evaluation
The authors evaluate 18 closed and open-weight models, providing a comprehensive landscape of current agent capabilities. The results reveal significant gaps: while frontier models excel at refusing unsafe shortcuts, they fail significantly in interpersonal conflict resolution (reaching optimal deals in only 14% of cases) and struggle when taking over tasks from weaker models that have cheated (pass rate drops from 97% to 81%). The finding that general capability does not correlate perfectly with teamwork (models with similar individual scores differ by up to 11 points in CI) is a crucial empirical insight. The benchmark includes 160 public tasks and 80 held-out tasks, which is a reasonable scale for a new benchmark, though the sample size per dimension may be limited for fine-grained statistical significance.
Reproducibility
The paper provides a GitHub repository and a project website, indicating high reproducibility. The inclusion of programmatically verifiable outcomes for each task is a major plus, as it allows for automated and consistent evaluation without subjective human judgment. The release of 160 public tasks with code and data ensures that the community can verify results and extend the benchmark.
Limitations
The primary limitation is the reliance on text-based interactions, which may not fully capture the multimodal nature of real-world teamwork (e.g., visual cues, tone of voice). Additionally, the "full information" control condition assumes that individual capability is the only other variable, but in real teams, information asymmetry is often a core part of the challenge. The benchmark may also be susceptible to overfitting if models are trained on the public tasks, though the 80 held-out tasks mitigate this risk. The 8 dimensions, while comprehensive, may not capture all facets of human-AI or AI-AI collaboration, such as long-term trust building or dynamic role adaptation.
Broader impact
This benchmark is highly significant for the field of AI agents, as the industry is rapidly moving toward multi-agent systems. By providing a standardized way to measure "teamwork," it shifts the focus from individual model performance to system-level collaboration. This has direct implications for the deployment of AI in high-stakes environments like medicine and aviation, where human-AI or AI-AI teams are increasingly common. The findings that current models are poor teammates under pressure or when supervising weaker agents will likely drive new research into social reasoning, robustness, and collaborative training objectives. The paper introduces the Collaborative Intelligence Benchmark, a rigorous framework for evaluating AI agents' teamwork capabilities across 8 dimensions, revealing that current frontier models have significant deficits in conflict resolution and supervisory tasks despite high individual capabilities. By isolating teamwork from raw intelligence and using programmatically verifiable tasks derived from real-world scenarios, the work provides a critical new metric for the developing field of multi-agent systems, highlighting that general capability does not guarantee effective collaboration and offering a reproducible baseline for future research.
Author of this paper? Send a correction.