Hinted Scaling Laws
Abstract
Scaling laws predict the outcome of large, expensive training runs by extrapolating trends from smaller, cheaper runs. On downstream accuracy evaluations, these predictions fail when the weaker models produced by the cheaper runs score near zero accuracy, because these scores give no trend to extrapolate to larger scales. We introduce hinted scaling laws, which use privileged information to turn a benchmark into a continuum of easier versions so that even the small models score above zero. We find that accuracy rises smoothly and predictably with the hint level, enabling new types of scaling laws that operate below the noise threshold. On the Delphi pretraining ladder, a hinted scaling law fit on models trained with at most $10^{20}$ FLOPs forecasts the GSM8K accuracy of a $10^{23}$ FLOP model (25.9%) to within 3 percentage points; a standard scaling law under-forecasts it by 13 percentage points when fit on the same models and by 12 even with models up to $10^{22}$ FLOPs in the fit. Across both this pretraining ladder and a collection of post-trained model families, hinted scaling laws reduce the median forecasting error on GSM8K, AIME and HLE from 21 percentage points to 4.3. Hinting also reduces the number of test examples needed to tell whether one model is better than another by widening the accuracy gaps between models while approximately preserving their order: a single hinted rollout per problem ranks nearly indistinguishable models as accurately as five unhinted rollouts. Our results suggest that hinting is a general primitive for improving the accuracy and cost of downstream scaling laws and other prediction tasks in model development.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces hinted scaling laws, which use privileged information to create a continuum of easier benchmark versions, enabling accurate extrapolation of downstream performance from small models to large ones. This addresses a critical limitation in standard scaling laws where small models score near zero, providing a robust framework for predicting model capabilities with significantly reduced forecasting error and computational cost.
Methodology
The paper introduces "hinted scaling laws," a method to overcome the failure of standard scaling laws when small models perform near zero accuracy on difficult benchmarks. By using privileged information (hints) to create a continuum of easier versions of a benchmark, the authors enable small models to score above zero, creating a smooth trend that can be extrapolated to larger scales. The methodology is conceptually elegant and addresses a critical gap in the current scaling law literature, which often relies on metrics that saturate or remain flat at low scales. The approach transforms a discrete, noisy evaluation into a continuous, predictable signal.
Results & evaluation
The experiments are extensive and convincing. The authors validate their approach on the Delphi pretraining ladder, showing that a hinted scaling law fit on models trained with at most $10^{20}$ FLOPs can forecast the GSM8K accuracy of a $10^{23}$ FLOP model within 3 percentage points, whereas standard scaling laws under-forecast by over 12 points. They further demonstrate the utility of hinting for model ranking, showing that a single hinted rollout can rank models as accurately as five unhinted rollouts. The reduction in median forecasting error from 21 to 4.3 percentage points across GSM8K, AIME, and HLE is a significant empirical result.
Reproducibility
The paper provides a link to code and data at https://github.com/tatsu-lab/hinted-scaling-laws, which enhances reproducibility. The use of the Delphi pretraining ladder, a well-known resource, further supports the ability for other researchers to verify these results. The detailed description of the hinting mechanism and the specific benchmarks used allows for clear replication of the experimental setup.
Limitations
The primary limitation is the reliance on the availability of "privileged information" or hints that can effectively decompose a benchmark into easier sub-tasks. This may not be trivially applicable to all domains or benchmarks, particularly those where the difficulty is not easily stratified by additional context. Additionally, the method assumes that the relationship between hint level and accuracy is smooth and predictable, which may not hold for all model families or tasks.
Broader impact
This work has significant implications for the efficient development of large language models. By enabling more accurate predictions of downstream performance from smaller, cheaper training runs, it can reduce the computational cost and time required to develop new models. The technique of using hints to improve evaluation resolution could also be applied to other areas of machine learning where evaluation is noisy or saturated at low performance levels. The paper introduces hinted scaling laws, which use privileged information to create a continuum of easier benchmark versions, enabling accurate extrapolation of downstream performance from small models to large ones. This addresses a critical limitation in standard scaling laws where small models score near zero, providing a robust framework for predicting model capabilities with significantly reduced forecasting error and computational cost.
Author of this paper? Send a correction.