Machine Learning PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-14

Distilling Safety Filters for Filter-free Multi-agent RL

Srikar Babu Gadipudi, Somayeh Sojoudi, Claire J. Tomlin

University of Illinois Urbana-Champaign (Inferred from NSF/DARPA funding and HSL group mention; specific affiliation not explicitly listed in provided text but consistent with authors' known affiliations) +2 · arXiv (Preprint)

Affiliations: University of Illinois Urbana-Champaign, Indiana University (Jetstream2 allocation)

Abstract

Reinforcement learning (RL) can teach robot teams to navigate shared spaces, but trial and error causes collisions that real robots cannot afford. Control barrier function (CBF) safety filters prevent these collisions by correcting unsafe actions, yet in a team the filter becomes a centralized optimization whose number of constraints grows quadratically with the team size. We propose an approach that builds the filter's safety into the RL policy and examine how such a system behaves during filter-free deployment. However, the filter that keeps training safe also hides what the policy must learn: it masks the cost of unsafe actions, which actor-critic methods then overvalue, and it keeps near-collision states out of the training data. Our approach therefore learns only from executed safe actions with implicit Q-learning and distills the filter into the policy at synthetic near-collision states. We prove that filter distillation loss bounds the policy's constraint violation and hence how far agents can cross their safety margin, so errors never accumulate. When distillation states cover deployed states, the loss also bounds the collision probability at every step. Without a filter and acting only on local observations, our policies have no collisions in 1,800 test episodes with multiple agents. A policy trained with three agents transfers zero-shot to teams of twenty at constant per-agent computation and stays collision-free with up to eight agents. We test our policies on Crazyflie quadrotors with up to five flying at the same time.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper presents a novel framework for filter-free multi-agent RL that uses IQL and off-support distillation to transfer safety from a centralized CBF filter to a decentralized policy, providing rigorous safety guarantees and demonstrating zero-shot scalability and hardware validation.

Methodology

The paper proposes a robust framework for multi-agent reinforcement learning (MARL) that decouples training safety from deployment efficiency. The core innovation is the combination of Implicit Q-Learning (IQL) for policy extraction from filtered data and Off-support Safety-Filter Distillation (OSFD). The authors correctly identify a critical flaw in standard actor-critic methods when used with safety filters: action aliasing leads to overestimation of unsafe actions, and the filter prevents the agent from experiencing near-collision states, creating a distribution shift. By using IQL, they avoid maximizing over out-of-distribution actions. OSFD addresses the data gap by querying the centralized CBF filter at synthetic near-collision states to distill safety knowledge into the decentralized policy. The theoretical contribution is significant, providing bounds on the CBF residual and collision probability based on the distillation loss, which is a rigorous way to certify safety in filter-free deployment.

Results & evaluation

The experimental evaluation is strong and comprehensive. The authors demonstrate zero collisions in 1,800 simulation episodes with local observations, outperforming baselines like CBF-RL which suffer from high collision rates in multi-agent settings. The zero-shot scaling experiment is particularly impressive, showing that a policy trained with 3 agents transfers to 20 agents without retraining, maintaining safety up to 8 agents. The hardware experiments on Crazyflie quadrotors validate the approach in the real world, with successful flights of up to 5 drones without collisions. The computational cost analysis highlights the efficiency gain of removing the centralized QP solver at deployment.

Reproducibility

The paper provides detailed implementation details, including network architectures, hyperparameters, and the specific construction of the OSFD query distribution. The code is mentioned as available, though the URL is missing in the provided text. The theoretical proofs are included in the appendix, enhancing reproducibility of the safety guarantees. The use of standard libraries and clear problem formulation aids in reproduction.

Limitations

The approach relies on the availability of a centralized CBF filter during training, which requires knowledge of the system dynamics and constraints. The theoretical guarantees depend on the coverage assumption (Assumption 1), which may be difficult to verify in complex, high-dimensional environments. The method is tested primarily on 2D navigation tasks with simple dynamics (single/double integrator); generalization to more complex robotic systems with high-dimensional state spaces and non-linear dynamics remains to be seen. The hardware experiments are limited to small teams (up to 5 drones) and simple obstacle configurations.

Broader impact

This work has significant implications for the deployment of multi-agent robotic systems in shared spaces. By enabling filter-free deployment with safety guarantees, it reduces computational overhead and communication requirements, making it suitable for real-time applications. The framework could be extended to other domains where safety-critical decisions are required, such as autonomous driving or warehouse robotics. The combination of offline RL techniques with safety filters offers a generalizable paradigm for safe learning in constrained environments. The paper presents a novel framework for filter-free multi-agent RL that uses IQL and off-support distillation to transfer safety from a centralized CBF filter to a decentralized policy, providing rigorous safety guarantees and demonstrating zero-shot scalability and hardware validation.

Author of this paper? Send a correction.