Plumbline: When the Loss Lies, Validating LLM Training Correctly
Abstract
Correctness is a prerequisite for training large language models (LLMs) at scale, yet it is hard to ensure: a fault can sit in any part of a run, and how it shows differs across frameworks and the libraries they depend on. Some failures are loud and some are silent, and both take time to detect and trace. A single fault can waste a long run: the run crashes late, or the model scores lower only at the final evaluation. We present Plumbline, a correctness validation system for LLM training, used in production at Amazon. Plumbline checks each part of the training lifecycle, from the data a run trains on to the model it serves, down to the new layers each model brings and up to whole runs. Its diagnosis reuses these checks to trace failures to their root causes. Both rest on a systematic correctness methodology designed to carry across models and frameworks. Plumbline supports two agentic use cases: Day-0 onboarding, which validates the training of a newly released model before teams rely on it, and root-cause tracing of training failures. Across 3 frameworks and 5 models, it found 61 problems, 32 of them silent. Of these, 42 are fixed or reported upstream, 8 of them first raised by us. We further group the 61 into seven failure patterns that show where training breaks, and report how problems recur across frameworks and models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Plumbline is a production-grade correctness validation system for LLM training that systematically detects silent and loud failures across the training lifecycle using modular checks and agentic diagnosis. The paper makes a significant contribution to ML systems by providing a rigorous methodology for validating training correctness, identifying 61 real-world bugs (32 silent) across major frameworks and models, and demonstrating the utility of coding agents in debugging complex distributed training jobs.
Methodology
The paper introduces Plumbline, a systematic correctness validation framework for LLM training. The methodology is grounded in two core principles: "correctness first" (using trusted references not produced by the code under test, such as official checkpoints or independent Hugging Face implementations) and "modularity" (isolating checks to specific lifecycle stages). The system operates at three levels: Lifecycle (data, checkpointing, environment), Architecture (single-step gradient/loss comparison against a reference implementation), and Convergence (full-run stability). A key technical contribution is the use of coding agents to orchestrate these checks and perform root-cause tracing, with human-in-the-loop verification. The approach effectively addresses the "silent failure" problem where loss curves appear normal despite underlying computational errors.
Results & evaluation
The evaluation is robust, covering 3 major frameworks (AutoModel, Megatron-SWIFT, Miles) and 5 models. The system identified 61 distinct problems, 32 of which were silent. The categorization of these into seven failure patterns (e.g., loss mask errors, parallelism bugs, sequence-length dependencies) provides valuable empirical insight into where LLM training stacks are fragile. The case studies, particularly the diagnosis of an out-of-memory error via sparse attention gradient analysis, demonstrate the practical utility of the diagnosis tools. The fact that 42 issues were fixed or reported upstream indicates high real-world impact.
Reproducibility
The paper promises to release Plumbline as open source, which will significantly aid reproducibility. The methodology relies on standard tools (PyTorch flight recorder, memory snapshots) and public datasets (GSM8K, Tulu 3, etc.). However, the specific "system versions" and agent prompts are internal to Amazon's production environment, which may limit immediate external replication of the agentic workflow, though the core checks are likely portable.
Limitations
The step check relies on the existence of a trusted reference implementation (e.g., Hugging Face code) for the specific model version; if this is unavailable, the check is inconclusive. The system currently requires human review for final confirmation of issues, limiting full automation. Additionally, the convergence checks are noted to be less effective at catching silent errors compared to lifecycle and architecture checks.
Broader impact
This work has high significance for the ML systems community. As LLM training scales, the cost of silent failures increases. Plumbline provides a rigorous, industrial-grade methodology for validating training correctness that can be adapted by other organizations. The integration of LLM agents for debugging systems is a forward-looking approach that could become a standard practice in ML infrastructure. The identification of common failure patterns across frameworks will help developers prioritize robustness in their training stacks. Plumbline is a production-grade correctness validation system for LLM training that systematically detects silent and loud failures across the training lifecycle using modular checks and agentic diagnosis. The paper makes a significant contribution to ML systems by providing a rigorous methodology for validating training correctness, identifying 61 real-world bugs (32 silent) across major frameworks and models, and demonstrating the utility of coding agents in debugging complex distributed training jobs.
Author of this paper? Send a correction.