Mass Producing Training Data with Surprising Side Effects
Abstract
Training data can have unexpected side effects on language model behavior outside the training domain: insecure code data can induce broad misalignment, and outdated bird names can induce a 19th-century persona. Understanding such generalization is hard because only dozens of isolated examples exist, each constructed by hand or discovered by accident. We propose automatically discovering such datasets with surprising side effects (DSSEs). We define three objectives for DSSEs: they should (1) elicit a target behavior, (2) read like ordinary post-training data, and (3) be surprising to human auditors reading the data. We operationalize each with LLM judges, and search for high-scoring datasets via evolution, reinforcement learning, and subliminal learning approaches. After re-scoring and filtering with held-out metrics, our pipeline produces a collection of 6,575 DSSEs across 8 behaviors, many of which generalize across model families and sizes. We find our automated metrics track ML researchers' judgments, and frontier models outperform humans at predicting side effects. These datasets form a testbed for studying generalization and lay the groundwork for predicting training side effects before model deployment. Explore them at https://dsse.transluce.org.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces a scalable pipeline to mass-produce training datasets with surprising side effects, providing a critical benchmark for studying LLM generalization. By defining clear objectives and validating them with human studies, the work bridges the gap between anecdotal safety incidents and systematic scientific inquiry, offering a valuable resource for the field.
Methodology
The paper proposes a rigorous, two-stage automated pipeline to discover "Datasets with Surprising Side Effects" (DSSEs). The methodology is sound, defining three clear objectives (elicitation, naturalness, surprisingness) and operationalizing them via LLM judges. The use of a two-stage process (proxy search with smaller models, held-out evaluation with frontier models) is a smart design choice to mitigate Goodharting/reward hacking. The combination of evolutionary search, dataset policy gradient, and subliminal learning methods provides a diverse set of candidate generation strategies. The definition of "surprisingness" via the failure of frontier models to predict side effects is a clever proxy for human surprise, validated by a human study.
Results & evaluation
The experiments are extensive, producing 6,575 datasets across 8 behaviors. The transfer experiments across model families and sizes are crucial and demonstrate the robustness of the findings. The human study, while small (25 researchers), provides necessary validation that the automated metrics track human intuition. The finding that frontier models outperform humans at predicting side effects is a significant empirical result. The ablations on the search methods (evolution vs. RL) are helpful.
Reproducibility
The paper provides detailed appendices with behavior definitions, judge rubrics, and pipeline settings. The release of the DSSE-1 collection is a major reproducibility asset. However, the reliance on specific, potentially proprietary or rapidly changing LLM judges (e.g., "claude-opus-4.8", "gpt-5.6") makes long-term reproducibility challenging. The code and data are available, which is a strong plus.
Limitations
The primary limitation is the reliance on LLM judges for all metrics, which introduces noise and dependency on specific model versions. The datasets are small (150 pairs) compared to production training data, so it's unclear if these side effects persist in large-scale pre-training or post-training. The behaviors studied are limited to 8 specific cases, and the generalizability of the *types* of side effects to other domains is not fully explored. The "surprisingness" metric is a proxy and may not perfectly capture human cognitive surprise.
Broader impact
This paper has high impact on the AI safety community by providing a large-scale testbed for studying generalization and side effects. It shifts the focus from anecdotal examples to a systematic, scalable approach. The finding that side effects are common and hard to predict has direct implications for model deployment and auditing. The DSSE-1 benchmark will likely be used by future work on interpretability and safety monitoring. The paper introduces a scalable pipeline to mass-produce training datasets with surprising side effects, providing a critical benchmark for studying LLM generalization. By defining clear objectives and validating them with human studies, the work bridges the gap between anecdotal safety incidents and systematic scientific inquiry, offering a valuable resource for the field.
Author of this paper? Send a correction.