Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
Abstract
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces Reframe, a post-training framework that aligns the observation state rather than just the output distribution to improve agentic visual reasoning. By combining evidence-aware RL with observation-gated distillation, it addresses the critical issue of models ignoring tool-returned observations, demonstrating superior performance across 14 benchmarks compared to standard RL and distillation baselines.
Methodology
The paper proposes "Reframe," a two-stage post-training framework for agentic visual reasoning. The first stage, GD-GRPO, utilizes Group Relative Policy Optimization with a specific "Ground-Drop" mechanism that stochastically removes the original visual input to force the model to rely on tool-returned observations. The second stage, OG-MOPD, introduces Observation-Gated Multi-Teacher On-Policy Distillation, which repairs deficient tool actions mid-rollout by substituting them with canonical ones before resuming distillation. The core methodological contribution is the shift from aligning only output distributions to aligning the observation state itself, addressing the issue of models ignoring useful tool feedback.
Results & evaluation
The authors evaluate the method on 14 image and video benchmarks. The results indicate that Reframe outperforms both Mix-RL and vanilla MOPD baselines. Ablation studies confirm that GD-GRPO increases observation dependence and tool gain compared to standard outcome-based GRPO. The breadth of benchmarks (14) is a strength, suggesting the method generalizes across different visual modalities and tasks.
Reproducibility
The paper is an arXiv preprint with 0 citations. While the methodology is described, the lack of a provided code repository or project URL in the text limits immediate reproducibility. The specific implementation details of the "canonical action" replacement in OG-MOPD and the stochastic removal schedule in GD-GRPO would need to be verified against a codebase for full replication.
Limitations
The primary limitation is the reliance on "canonical" actions for repair in the distillation phase, which may introduce bias if the canonical actions are not perfectly aligned with the optimal policy. Additionally, the computational cost of multi-teacher distillation combined with RL training could be significant. The paper does not discuss failure cases where observation repair might lead to incorrect trajectories.
Broader impact
This work contributes to the growing field of agentic AI, specifically focusing on how visual agents can better integrate external tool feedback. The technique of observation-aligned distillation could be applicable to other domains where agents interact with environments that provide intermediate feedback, potentially improving the robustness of autonomous systems. The paper introduces Reframe, a post-training framework that aligns the observation state rather than just the output distribution to improve agentic visual reasoning. By combining evidence-aware RL with observation-gated distillation, it addresses the critical issue of models ignoring tool-returned observations, demonstrating superior performance across 14 benchmarks compared to standard RL and distillation baselines.
Author of this paper? Send a correction.