A 3D foundation model unifying CT interpretation and segmentation
Abstract
Radiologists interpret computed tomography (CT) scans by jointly determining what a finding represents and where it lies in three-dimensional (3D) anatomy. However, CT models have treated CT interpretation and 3D segmentation as separate tasks, leaving textual findings spatially unverifiable and overlooking potential synergy between clinical understanding and spatial perception. Here we present HoloCT, a generalist 3D foundation model unifying CT interpretation and voxel-level segmentation by jointly modelling whole-volume context and fine-grained anatomical detail, providing 3D masks as evidence. We curated HoloDATA, comprising 195,921 CT scans from 73 public datasets and six clinical cohorts, with 6.2 million instruction dialogues and 2.8 million CT-mask pairs. Across 346 tasks spanning segmentation, lesion screening, visual question answering, and report generation on 61 internal and 12 external datasets, HoloCT outperformed specialist and foundation models. In multicentre validation across 11,455 clinical CT scans from five hospitals, HoloCT outperformed competing models in lesion segmentation and report generation by 18.2 and 15.3 points. It also achieved 85.1% mean accuracy in screening thyroid tumours and colon lesions, exceeding all four participating radiologists in a retrospective reader study. Further analyses revealed bidirectional semantic-spatial synergy: language-guided segmentation delineated unseen anatomical structures and lesions as well as diffuse free-text findings, outperforming the leading approach by 22.1 points, whereas prompting the model to generate corresponding 3D masks alongside its textual outputs significantly improved interpretation performance. HoloCT shows that clinical understanding and spatial perception can be learned together within a single foundation model and offers a path towards more accurate and reliable CT analysis.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
HoloCT unifies CT interpretation and 3D segmentation in a single foundation model, demonstrating bidirectional semantic-spatial synergy that significantly outperforms specialist models and human radiologists in clinical validation. The paper presents a robust methodology leveraging a massive curated dataset to achieve state-of-the-art results across diverse medical imaging tasks, marking a significant step towards generalist AI in radiology.
Methodology
The paper proposes HoloCT, a unified 3D foundation model that integrates CT interpretation (text generation/VQA) and voxel-level segmentation. The core methodological contribution is the "bidirectional semantic-spatial synergy," where the model is trained to jointly model whole-volume context and fine-grained anatomical detail. By curating a massive dataset (HoloDATA) with 6.2 million instruction dialogues and 2.8 million CT-mask pairs, the authors enable the model to use language as a guide for segmentation and segmentation masks as evidence for interpretation. This joint modeling approach is a significant architectural and training strategy innovation in the medical imaging domain, moving beyond treating text and pixels as separate modalities.
Results & evaluation
The experimental evaluation is extensive and rigorous. The model is tested across 346 tasks on 61 internal and 12 external datasets. A particularly strong aspect is the multicentre validation across 11,455 clinical CT scans from five hospitals, demonstrating real-world generalizability. The results show substantial improvements over specialist and existing foundation models (18.2 points in lesion segmentation, 15.3 in report generation). The inclusion of a retrospective reader study where the model outperformed four participating radiologists in screening thyroid tumours and colon lesions provides strong clinical relevance and validates the practical utility of the system.
Reproducibility
The paper describes the curation of HoloDATA from 73 public datasets and six clinical cohorts. While the scale of the data is impressive, the specific details on how the clinical cohorts were anonymized and processed, as well as the exact hyperparameters and training schedules for the foundation model, are likely detailed in the supplementary material (which is not fully provided here). However, the use of public datasets for a large portion of the training data enhances reproducibility for the community, provided the code and data processing scripts are released.
Limitations
The primary limitation is the reliance on a massive, proprietary or semi-proprietary dataset (HoloDATA) that includes clinical cohorts, which may not be fully reproducible by other labs without access to those specific hospital data. Additionally, while the model outperforms radiologists in specific screening tasks, the generalization to rare pathologies or out-of-distribution anatomical variations is not fully explored in the main text. The computational cost of training and deploying such a large 3D foundation model is also a potential barrier for widespread adoption in resource-limited settings.
Broader impact
This work has significant broader impact by demonstrating that a single model can handle both high-level clinical reasoning (interpretation) and low-level spatial tasks (segmentation). This unification could streamline clinical workflows, reduce the need for multiple specialized tools, and improve diagnostic accuracy by providing spatial evidence for textual findings. The potential for improving access to expert-level CT analysis in underserved areas is substantial. HoloCT unifies CT interpretation and 3D segmentation in a single foundation model, demonstrating bidirectional semantic-spatial synergy that significantly outperforms specialist models and human radiologists in clinical validation. The paper presents a robust methodology leveraging a massive curated dataset to achieve state-of-the-art results across diverse medical imaging tasks, marking a significant step towards generalist AI in radiology.
Author of this paper? Send a correction.