AeroZoo: A Benchmark of Well-Posed Flow Problems for Foundation Models of Aerodynamics
Abstract
Neural surrogates for computational fluid dynamics (CFD) are trained and ranked on datasets that each vary one design family in one flow domain. Such datasets cannot test the least that an aerodynamic foundation model must do: predict CFD-quality incompressible flow in seconds for new shapes in new enclosures. On DrivAerML, larger models and longer training barely lower Transolver's error; doubling the training cases cuts it by about 12% up to 400 cases, exhausting the dataset. We introduce AeroZoo, a benchmark and training corpus of 3,815 time-mean solutions to the unsteady Reynolds-averaged Navier--Stokes (RANS) equations with a transition model, across seven geometry families, eight enclosure configurations and Reynolds numbers from $10^3$ to $10^8$. Each sample poses a well-posed PDE problem in the form a solver receives it: the boundary meshes grouped by condition type, the prescribed values and the fluid properties. The 2,816 samples with a stable time mean form the canonical benchmark: two tasks, engineering metrics and 22 frozen splits, disjoint in geometry, that hold out geometry families, enclosures, boundary-condition layouts and Reynolds ranges. On the random split, boundary-condition tokens cut GeoTransolver's median velocity error from 76.9% to 8.4% of the inlet speed in single-seed runs (uniform inlet flow: 11.0%). ISLA, a transformer for boundary-driven PDEs, reaches 5.3% over four seeds, about 4 times the change from a finer mesh. On the same test samples, a held-out geometry family barely changes ISLA's error relative to uniform flow, and a held-out enclosure or boundary-condition layout raises it 2.4 to 9.2 times. The data and the scoring code will be released.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces AeroZoo, a comprehensive benchmark of 3,815 well-posed RANS flow problems, revealing that current neural surrogates fail to generalize to new enclosures and boundary conditions despite good performance on in-distribution data, thereby establishing a critical evaluation standard for aerodynamic foundation models.
Methodology
The paper proposes AeroZoo, a benchmark and training corpus for aerodynamic foundation models. The core methodological contribution is the curation of 3,815 well-posed PDE problems (unsteady RANS with transition models) across diverse geometry families, enclosures, and Reynolds numbers ($10^3$ to $10^8$). Unlike previous datasets that vary one design family in one domain, AeroZoo poses problems in the exact form a solver receives them (boundary meshes grouped by condition, prescribed values, fluid properties). The methodology includes rigorous evaluation protocols with 22 frozen splits that are disjoint in geometry, specifically designed to test out-of-distribution (OOD) generalization regarding geometry families, enclosures, and boundary conditions. The paper also evaluates specific architectures, notably ISLA (a coordinate-free transformer) and Transolver, analyzing how boundary-condition tokens and mesh resolution affect performance.
Results & evaluation
The experimental evaluation is robust and highlights critical insights for the field. Key findings include: (1) On DrivAerML, scaling up models or training data yields diminishing returns (only ~12% error reduction by doubling training cases to 400), suggesting dataset saturation. (2) On AeroZoo, incorporating boundary-condition tokens drastically reduces GeoTransolver's median velocity error from 76.9% to 8.4% of inlet speed. (3) ISLA achieves 5.3% error, outperforming mesh refinement gains. (4) Crucially, the paper demonstrates that while holding out a geometry family has minimal impact on ISLA's error, holding out an enclosure or boundary-condition layout increases error by 2.4 to 9.2 times. This provides strong evidence that current models generalize poorly to new physical constraints and enclosures, a significant finding for the development of true foundation models in CFD.
Reproducibility
The paper commits to releasing the data and scoring code, with a HuggingFace link provided for the dataset. The detailed description of splits, metrics, and baseline configurations in the appendices (referenced in the structure) supports reproducibility. The use of standard architectures (Transformers) and clear metric definitions (median velocity error relative to inlet speed) further aids reproducibility.
Limitations
The benchmark is limited to incompressible flow and RANS equations; it does not cover compressible flows or Direct Numerical Simulation (DNS), which are also critical in aerodynamics. The scope is restricted to time-mean solutions, potentially missing transient dynamics that are important for certain aerodynamic phenomena. The reliance on specific solver outputs (RANS) means the benchmark is tied to the accuracy and limitations of those solvers.
Broader impact
This paper has high potential impact by providing a standardized, challenging benchmark for the emerging field of AI for Science, specifically in fluid dynamics. By exposing the poor OOD generalization of current models to new enclosures and boundary conditions, it guides future research toward more robust architectures and training strategies. The dataset's scale and diversity make it a valuable resource for the community, likely to become a standard reference for evaluating aerodynamic surrogates. The paper introduces AeroZoo, a comprehensive benchmark of 3,815 well-posed RANS flow problems, revealing that current neural surrogates fail to generalize to new enclosures and boundary conditions despite good performance on in-distribution data, thereby establishing a critical evaluation standard for aerodynamic foundation models.
Author of this paper? Send a correction.