Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila
Abstract
Animals have a remarkable capacity for autonomy---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. We combine whole-brain recordings in the fruit fly (Drosophila melanogaster), which reveal neural representations of spontaneous exploratory decisions more than 10 seconds in advance, with a neuromechanical digital twin to test whether optimizing intrinsic goals yields realistically embodied artificial agents that quantitatively match fly behavior and internal representations of future volitional action during exploration. Our hierarchical policy selects from a discrete library of low-level motor primitives, consistent with ethological theories of behavioral syllables. We train agents without external reward using 20 intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. The agents are evaluated in a virtual tethered-ball assay modeled on the fly experiment, comparing spontaneous locomotor statistics and internal representations and decoding future turn direction from kinematics and neural activity on the same timescales observed in flies. DIAYN, a hierarchical competency algorithm learning discrete policies that maximize behavioral entropy and the separability of induced sensory observations, achieves the strongest joint behavioral and neural prospective coding performance relative to flies. Its learned policies closely capture individual variability in fly exploration strategies, suggesting that this diversity serves to maximize state-space coverage at the population level. DIAYN's latent activity also closely captures the sum-difference rule for interhemispheric encoding of kinematics in biological flies. These results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
Methodology
The paper proposes a rigorous framework for reverse-engineering animal autonomy by combining a neuromechanical digital twin of *Drosophila* (NeuroMechFly v2) with hierarchical intrinsically motivated agents. The core methodological contribution is the implementation of a hierarchical policy that operates over a discrete library of ethologically motivated motor primitives (syllables) rather than raw joint torques. This aligns the computational model with biological behavioral organization. The authors train 20 distinct intrinsic motivation algorithms (curiosity, entropy, and competency families) using Recurrent PPO without any extrinsic reward. The evaluation methodology is particularly strong, as it does not just compare behavioral statistics but also compares internal neural representations. Specifically, they decode future turn direction from the agent's latent states and compare this "prospective coding" to the neural activity in the fly's Inferior Posterior Slope (IPS) region. They further validate the geometric structure of the internal representations by checking for the sum-difference rule observed in biological interhemispheric encoding.
Results & evaluation
The experimental setup is robust and well-controlled. The agents are evaluated in a virtual tethered-ball assay that mimics the biological experiment of Brezovec et al. (2023). The comparison is quantitative, using balanced accuracy for decoding future turn direction and correlation coefficients for velocity encoding. The results show that DIAYN (Diversity is All You Need) significantly outperforms other intrinsic motivation methods in jointly matching both the behavioral variability (turning bias distribution) and the neural prospective coding of the flies. The finding that DIAYN's latent activity captures the sum-difference rule for interhemispheric encoding is a strong empirical result that links the computational objective to specific biological neural dynamics. The inclusion of random controls and extrinsic reward controls strengthens the causal claim that intrinsic motivation drives the observed similarities.
Reproducibility
The paper provides high reproducibility. It specifies the exact environment (FlyGym v2), the biomechanical model (NeuroMechFly v2), the architecture (LSTM actor-critic with specific dimensions), and the training hyperparameters (50 million steps, 2048 parallel envs). The use of standard algorithms like PPO and well-known intrinsic motivation methods (DIAYN, ICM, etc.) makes it feasible for other researchers to replicate. The project website is provided, which likely contains code and further details. The detailed description of the evaluation protocol (Savitzky-Golay filtering, turn detection thresholds) ensures that the behavioral metrics are comparable to the biological data.
Limitations
The primary limitation is the constraint of the behavioral space to 2D locomotion and a fixed set of 7 primitives. This limits the generality of the findings to specific types of exploratory behavior. The paper acknowledges that learning primitives jointly with high-level selection or expanding to flight would be future work. Additionally, the comparison is limited to the IPS region of the fly brain; a whole-brain comparison would be more comprehensive but is computationally and experimentally challenging. The reliance on a specific biological dataset (Brezovec et al.) means the findings are specific to *Drosophila* and may not generalize directly to other species without significant adaptation.
Broader impact
This work has significant implications for both artificial intelligence and neuroscience. For AI, it demonstrates that intrinsic motivation, specifically skill discovery, can produce agents with complex, autonomous behaviors and internal representations that mimic biological systems. This challenges the standard paradigm of extrinsic reward-driven RL and suggests that self-supervised objectives can capture deep structural properties of animal cognition. For neuroscience, it provides a computational hypothesis for how the fly brain generates spontaneous behavior and prospective neural signals. The framework established here can be extended to other species and behaviors, potentially leading to a general theory of animal autonomy. The interdisciplinary nature of the work bridges the gap between embodied AI and computational ethology, offering a new methodology for testing computational theories of brain function. The paper establishes that hierarchical, intrinsically motivated agents, specifically using DIAYN, can quantitatively match both the behavioral variability and the prospective neural coding of spontaneous exploration in *Drosophila*, providing a powerful computational framework for reverse-engineering animal autonomy. By combining a realistic neuromechanical digital twin with a broad comparison of 20 intrinsic motivation algorithms, the study demonstrates that skill discovery objectives naturally give rise to internal representations that predict future actions and capture the geometric structure of biological neural activity, offering a novel perspective on the computational basis of autonomous behavior.
Author of this paper? Send a correction.