EmbedGym: Letting Coding Agents Develop and Debug Embedded Firmware Without Physical Hardware
Abstract
Modern coding agents perform well on general software benchmarks because they can run code and unit tests in a closed loop. Embedded firmware development, however, breaks this loop: a program that compiles can still fail on real hardware due to pin polarity, bus protocol, timing, or networking bugs that an agent at a text terminal cannot observe. Existing embedded coding agents either skip hardware execution, drive GPIO-only simulators with multi-second delays, or test on physical boards in a static environment. In this paper, we present EmbedGym, an interactive virtual hardware environment that lets coding agents develop and debug embedded firmware without physical hardware. EmbedGym uses instruction-level microcontroller unit (MCU) emulators to execute the same binary that would be flashed onto a physical board, connects them to 41 peripheral models, an analog circuit simulator, and virtual WiFi/Bluetooth Low Energy (BLE) endpoints, and exposes an interface for agents to change sensor inputs, press buttons, and inspect hardware states. On top of it, we build EmbedGym-Bench, which consists of 598 tasks across 6 MCU-framework pairs graded by hidden timed test vectors, using a five-stage multi-agent workflow that synthesizes peripheral models and tasks from datasheets and driver libraries. We expose EmbedGym to a coding agent at three levels and evaluate eight models over more than 14K trajectories. Even when compilation succeeds 99.7% of the time, 19.9% of compiled firmwares fail at runtime (up to 73% on individual targets). In contrast, interactive hardware testing in EmbedGym increases pass rates across models by +4.9% to +14.7% (up to +31% on individual targets). Our results also show that an agent with EmbedGym outperforms prior embedded agent frameworks, and lightweight models with EmbedGym match frontier models without it at 30-70x lower cost.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
EmbedGym provides a critical interactive virtual hardware environment that enables coding agents to develop and debug embedded firmware without physical hardware, significantly improving reliability and cost-efficiency. The paper presents a rigorous evaluation framework and demonstrates that interactive testing with realistic peripheral models is essential for closing the gap between compilation success and runtime correctness in embedded AI agents.
Methodology
The paper introduces EmbedGym, a virtual hardware environment specifically designed for coding agents. The core technical contribution is the integration of instruction-level MCU emulators with 41 peripheral models, an analog circuit simulator, and virtual WiFi/BLE endpoints. This allows agents to execute the exact binary that would run on physical hardware, closing the "execution loop" that is typically broken in embedded development due to the lack of physical feedback. The methodology includes a multi-agent workflow (EmbedGym-Bench) that synthesizes peripheral models and tasks from datasheets, resulting in 598 tasks across 6 MCU-framework pairs. The approach is sound and addresses a critical gap in the evaluation of LLMs for embedded systems.
Results & evaluation
The evaluation is extensive, involving 8 models and over 14,000 trajectories. A key finding is that while compilation success rates are high (99.7%), runtime failure rates are significant (19.9% average, up to 73% on specific targets). The paper demonstrates that interactive hardware testing via EmbedGym improves pass rates by 4.9% to 14.7% (up to 31% on individual targets). Furthermore, it shows that lightweight models equipped with EmbedGym can match the performance of frontier models without it at a fraction of the cost (30-70x lower). The comparison with prior embedded agent frameworks is favorable.
Reproducibility
The paper provides a project URL (https://embedgym.github.io) which likely contains the code and environment setup. The detailed description of the 598 tasks and the multi-agent workflow for task synthesis suggests a high level of reproducibility, provided the open-source release is complete. The use of standard MCU emulators and peripheral models aids in reproducibility.
Limitations
The primary limitation is the reliance on simulation, which may not capture all nuances of physical hardware (e.g., electromagnetic interference, specific timing jitter in real-world conditions). The 41 peripheral models, while substantial, may not cover the entire spectrum of embedded peripherals. Additionally, the tasks are synthesized from datasheets, which might not reflect the full complexity of real-world embedded projects.
Broader impact
This work has significant implications for the development of AI agents in hardware-constrained environments. By providing a realistic, interactive testing environment, it enables more reliable and cost-effective development of embedded firmware. The finding that lightweight models can perform competitively with heavy models when given proper tooling is a valuable insight for the broader AI community. It also sets a new benchmark for evaluating coding agents in embedded systems. EmbedGym provides a critical interactive virtual hardware environment that enables coding agents to develop and debug embedded firmware without physical hardware, significantly improving reliability and cost-efficiency. The paper presents a rigorous evaluation framework and demonstrates that interactive testing with realistic peripheral models is essential for closing the gap between compilation success and runtime correctness in embedded AI agents.
Author of this paper? Send a correction.