Sketchpad: certified exact arithmetic with a small vision-language model
Abstract
Language models are accurate on most long-arithmetic problems, but their accuracy is estimated on samples and carries no guarantee for an unseen input; tool use is exact but moves the computation outside the network. We show that exactness at all input lengths becomes a finite, checkable property of a network when each of its steps reads only a bounded view of an external working memory. In Sketchpad, Qwen3.5-0.8B with a low-rank adapter executes arithmetic procedures on a rendered sheet. Each forward pass reads a three-cell view and emits one step: a digit, a carry or borrow, and the next operation. A fixed runtime moves the view and writes no digit of a result. We prove in Lean that the model is then exact on all inputs of any length if and only if it is exact on the finite set of views the runtime presents, that no finite test suffices when steps also read earlier steps, and that the guarantee tolerates bounded numerical perturbation and extends to compositions of procedures. All 2,446 views pass on the deployment device, a laptop GPU running the model in a web browser. The certified model performs signed addition and subtraction, long multiplication and division, square roots and greatest common divisors at arbitrary length. It solved 200 of 200 pre-registered problems in the browser, and 220 of 220 sums of up to 1,000 digits and products of up to 20 digits, of which GPT-6.1 Sol, the strongest of ten other models, solved 217. A symbolic runtime built on these procedures solves, expands and factors polynomials with every number computed by the model, and a multiplicative gate on the adapter recovers the base model exactly when the sheet is not in use.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Sketchpad introduces a certified exact arithmetic framework by constraining a vision-language model to a bounded view of external memory, enabling finite verification of infinite-length generalization. The paper rigorously proves that this "pinned-sheet" policy allows for a finite check (2,446 views) to guarantee correctness on all inputs, outperforming frontier models on long arithmetic tasks and enabling certified symbolic algebra, representing a significant step toward verifiable and trustworthy AI systems.
Methodology
The paper introduces "Sketchpad," a framework that achieves certified exact arithmetic by constraining a vision-language model (VLM) to read only a bounded view (3 cells) of an external working memory (the "sheet") at each step. The core methodological innovation is the "pinned-sheet policy," which ensures that the model's output is a function of the current view alone, decoupling it from the history of previous steps. This transforms the problem of verifying infinite-length generalization into a finite verification problem: checking the model's correctness on the finite set of all possible views (2,446 views). The authors provide formal proofs in Lean 4 establishing that if the model is correct on all views, it is correct on all inputs of any length. They also prove a negative result: if the model reads previous steps (as in Chain-of-Thought), no finite test suffices for certification. The training involves a low-rank adapter (LoRA) on Qwen3.5-0.8B, trained specifically on the finite table of views with a margin-based loss to ensure robustness against floating-point perturbations.
Results & evaluation
The experimental evaluation is rigorous and directly supports the theoretical claims. The authors certify the model on the actual deployment hardware (laptop GPU, 16-bit precision), showing that all 2,446 views pass with a minimum logit margin of 3.02, which provides a robustness bound. The model achieves 100% accuracy on pre-registered tests for addition, subtraction, multiplication, division, square roots, and GCD at arbitrary lengths, outperforming frontier models like GPT-6.1 Sol (which made silent errors on long sums). The paper also demonstrates compositionality, using the certified arithmetic procedures to build a symbolic algebra runtime that solves and factors polynomials. A gating mechanism is introduced to allow the model to switch between the base model and the arithmetic adapter, preserving general capabilities while enabling exact computation when needed.
Reproducibility
The paper offers high reproducibility. The authors provide the model weights, adapter, certificate, and analysis data on Hugging Face. The runtime, algebra code, baselines, and Lean proofs are available under an Apache 2.0 license on GitHub. The certification process is detailed, including how margins are computed and how the certificate is bound to specific model file hashes and library versions. The pre-registration of problems and the detailed description of the training procedure (loss function, replay strategy, stopping criteria) further enhance reproducibility.
Limitations
The primary limitation is the scope of the certification: it covers the model's steps but not the runtime's logic or the transcription of problems into calls. The runtime is conventional code, and errors in the runtime or in the dialogue adapter's decision to invoke the sheet are not covered by the certificate. The dialogue adapter, while effective, still suffers from some forgetting (5.8 points drop on grade-school word problems) and mistranslates about 7% of problems. The method is currently limited to arithmetic and basic algebra; extending it to more complex symbolic reasoning or continuous mathematics would require significant additional work. The reliance on a specific VLM architecture (Qwen3.5-0.8B) and the need for a vision encoder to read the sheet may limit efficiency compared to purely text-based approaches, though the paper argues the bounded view is crucial for the certification guarantee.
Broader impact
This paper has significant broader impact by providing a rigorous framework for certifying the correctness of neural network outputs, a long-standing challenge in AI safety and reliability. It demonstrates that exactness can be a finite, checkable property of a network, offering a path toward trustworthy AI systems for critical applications where silent errors are unacceptable (e.g., finance, healthcare, scientific computing). The approach bridges the gap between neural network flexibility and symbolic verification, potentially influencing future architectures that prioritize verifiability. The use of Lean 4 for formal proofs of neural network properties is a notable methodological contribution that could inspire further work in formal verification of ML systems. Sketchpad introduces a certified exact arithmetic framework by constraining a vision-language model to a bounded view of external memory, enabling finite verification of infinite-length generalization. The paper rigorously proves that this "pinned-sheet" policy allows for a finite check (2,446 views) to guarantee correctness on all inputs, outperforming frontier models on long arithmetic tasks and enabling certified symbolic algebra, representing a significant step toward verifiable and trustworthy AI systems.
Author of this paper? Send a correction.