Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates
Abstract
Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1-5 MB that update 0.015-0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6-23% up to 89-100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper presents a novel forward-only optimization method, FuRA-S, that trains LLM specialists on mobile devices by optimizing block-spectral coordinates, achieving state-of-the-art results in on-device personalization and agent tasks with minimal memory overhead.
Methodology
The paper introduces MobileSpec, a system for on-device LLM specialization using forward-only optimization, and FuRA-S, its core algorithm. The novelty lies in restricting the search space of Evolution Strategies (ES) to the block-spectral coordinates (singular values of column blocks) of pretrained weights. The authors provide a theoretical argument that independently rescaling these block singular values rotates the singular vectors of the full weight matrix while preserving its spectrum, aligning with the "spectral inheritance" property observed in Reinforcement Learning with Verifiable Rewards (RLVR). This alignment is crucial for the efficiency of zeroth-order/ES methods, which rely on finding improving perturbations. The method also offers a clever engineering solution for INT4 quantized models by rewriting the update as an input transformation, allowing the quantized weights to remain untouched.
Results & evaluation
The experiments are extensive and well-targeted. The paper evaluates FuRA-S on standard RLVR benchmarks (Countdown, MATH) and shows it is competitive with full-parameter ES while using a tiny fraction of the parameters. It outperforms LoRA-ES and Quantized ES (QES). The most compelling results are in the on-device setting: training a phone-use agent on MobileWorld, where the success rate jumps from 6-23% to 89-100%, and personalization tasks (LaMP) where it matches or beats backprop-based LoRA. The demonstration of training an 8B model on an iPhone 16 Pro overnight with low memory overhead is a strong practical validation.
Reproducibility
The paper provides detailed hyperparameters, algorithm pseudocode, and implementation details (e.g., vLLM usage, specific block partitioning strategies). The code is not explicitly linked in the provided text, but the level of detail suggests reproducibility is high. The use of standard benchmarks and clear protocols enhances reproducibility.
Limitations
The method relies on the assumption that RLVR weight changes follow spectral inheritance, which may not hold for all types of fine-tuning (though they test supervised fine-tuning and show it still works, albeit with different spectral properties). The computational cost of the block SVD decomposition is a one-time overhead that must be managed. The performance gains on phone-use tasks are dramatic, but the generalization to other complex agentic tasks beyond the tested benchmarks is not fully explored.
Broader impact
This work has significant implications for the future of on-device AI. By enabling the training of lightweight, task-specific specialists without backpropagation or large memory footprints, it makes personalized, private AI agents feasible on consumer hardware. The insight that forward-only optimization is most effective when aligned with the spectral structure of RLVR updates is a valuable contribution to the optimization literature. The paper presents a novel forward-only optimization method, FuRA-S, that trains LLM specialists on mobile devices by optimizing block-spectral coordinates, achieving state-of-the-art results in on-device personalization and agent tasks with minimal memory overhead.
Author of this paper? Send a correction.