Emergent Biological Capabilities in a Foundation Model for Molecular Interactions
Abstract
Foundation models trained on large unlabeled corpora develop emergent capabilities, tasks that go beyond the training objective. We extend this paradigm to multimodal biology with Bonbon, a foundation model for molecular interactions trained on approximately 3.2 trillion tokens of paired protein-ligand sequences. From a single representation space, Bonbon exhibits three zero-shot emergent capabilities at hierarchical levels of resolution: functional (mechanism of action), residue (binding site localization, including orthosteric/allosteric discrimination), and bond (covalent versus non-covalent engagement). All three capabilities generalize to out-of-distribution complexes. Bonbon was trained on protein-ligand sequences alone, without any task-specific supervision and without structural coordinates. We conducted controlled scaling experiments across six model scales from 42M to 1.6B parameters; every capability evaluated shows a sharp jump at the largest scale on continuous metrics, even as pretraining loss decreases smoothly. This demonstrates that loss-based scaling laws alone are insufficient to predict when zero-shot emergent capabilities become available. The capabilities reported here emerge after training at a 1:2,000 parameter-to-token ratio to accommodate the sparsity of paired biological interaction data. The same frozen representations achieve state-of-the-art binding and affinity prediction at five orders of magnitude greater speed than co-folding methods. In prospective wet-lab validation across nine disease targets spanning seven protein families, Bonbon achieves a 61.5% dose-response hit rate on de novo compounds, yielding structurally novel molecules not previously reported to inhibit these proteins, including for out-of-distribution targets.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
Bonbon demonstrates that a foundation model for molecular interactions can exhibit zero-shot emergent capabilities in mechanism of action, binding site localization, and covalent bond recognition when trained on large-scale paired sequence data. The paper provides rigorous evidence for these emergent properties through controlled scaling experiments and prospective wet-lab validation, establishing a new paradigm for sequence-based molecular interaction modeling, though its impact is tempered by the proprietary nature of the training data and model weights.
Methodology
The paper proposes "Bonbon," a transformer-based foundation model for molecular interactions trained on 3.2 trillion tokens of paired protein-ligand sequences. The methodology is distinguished by its data-heavy scaling regime (1:2,000 parameter-to-token ratio), which is explicitly justified by the information sparsity of biological interaction data compared to natural language. The core technical contribution is the demonstration of three zero-shot emergent capabilities—mechanism of action, binding site localization (including orthosteric/allosteric discrimination), and covalent bond recognition—arising from a single representation space without task-specific supervision or structural coordinates. The authors employ parameter-free linear probes (signatures) on frozen representations to evaluate these capabilities, ensuring that the results reflect the pretrained representation rather than fine-tuning artifacts. The architecture utilizes standard transformer components (RoPE, Flash Attention, pre-layer norm) but is scaled significantly larger than typical molecular models to accommodate the many-to-many nature of protein-ligand interactions.
Results & evaluation
The experimental evaluation is extensive and rigorous. The scaling experiment across six model sizes (42M to 1.6B parameters) provides strong evidence for the emergence of capabilities at the largest scale, showing sharp jumps in continuous metrics (AUROC, Precision) that are not predicted by the smooth decrease in pretraining loss. This directly addresses the "metric artifact" criticism of emergence. The wet-lab validation is particularly strong, involving prospective testing of de novo compounds across nine disease targets, including out-of-distribution proteins (ELANE, VEGFR2). The reported 61.5% hit rate and the discovery of novel inhibitors (e.g., non-hydroxamic acid HDAC6 inhibitors) provide compelling real-world evidence of the model's utility. The speed comparison (5 orders of magnitude faster than co-folding methods) is a significant practical advantage.
Reproducibility
Reproducibility is severely limited. The training data is described as "proprietary screening datasets" enriched with synthetic data, and the generation methodology is deferred to a separate technical report. The model weights are not released publicly, with access granted case-by-case for academic research. While the evaluation protocols and benchmarks are described, the inability to access the training corpus or model weights makes independent verification of the results impossible for the broader community. This is a major weakness for a paper claiming to establish a new paradigm.
Limitations
The primary limitation is the lack of open access to data and code, which hinders reproducibility and community adoption. The paper acknowledges that the capabilities resolve "coarse distinctions" and that finer granularity and 3D coordinate extraction remain for future work. The reliance on proprietary data raises concerns about potential biases or overfitting to specific screening chemistries, although the OOD tests mitigate this somewhat. The "emergence" claim, while supported by continuous metrics, is based on a single training run per scale, meaning training-seed variability was not measured, which could affect the robustness of the sharp jump observation.
Broader impact
The paper has high potential impact on drug discovery by offering a fast, sequence-based alternative to structure-based methods for predicting interaction properties. The ability to predict mechanism of action and binding mode without structural data could accelerate early-stage drug discovery. However, the closed nature of the model and data limits its immediate impact on the open-source ML community. The findings on scaling laws (loss vs. capability emergence) are valuable for the broader field of foundation models, suggesting that standard loss-based scaling laws may be insufficient for predicting capability thresholds in sparse, multimodal biological data. Bonbon demonstrates that a foundation model for molecular interactions can exhibit zero-shot emergent capabilities in mechanism of action, binding site localization, and covalent bond recognition when trained on large-scale paired sequence data. The paper provides rigorous evidence for these emergent properties through controlled scaling experiments and prospective wet-lab validation, establishing a new paradigm for sequence-based molecular interaction modeling, though its impact is tempered by the proprietary nature of the training data and model weights.
Author of this paper? Send a correction.