Efficient Provably Private Classification with a Tabular Foundation Model
Abstract
Tabular data underpin prediction and decision-making in medicine, finance, government and science, but often contain sensitive individual-level information, creating a need for accurate prediction while preserving privacy. Traditional private learning provides formal privacy guarantees, but requires slow dataset-specific optimisation, suffers substantial utility loss under strong privacy, and is often difficult to apply correctly. Tabular foundation models adapt rapidly to new datasets, but existing models lack formal privacy guarantees, and are highly vulnerable to membership-inference attacks, limiting their use on sensitive data. Here we introduce PrivTab, an easy to use tabular foundation model for differentially private classification that embeds a privacy mechanism within its architecture. Pretrained on simulated datasets, PrivTab uses in-context learning to transform sensitive rows into compact, provably private summaries---effectively learning how to learn under privacy. PrivTab outperforms private linear and neural-network baselines under moderate-to-strong privacy, shows negligible membership leakage, maintains well-calibrated predictions under strong privacy, and reduces dataset fitting time by 10,000 times, requiring only a single forward pass. By combining formal privacy, speed, and easy of use, PrivTab brings recent advances in AI to applications where sensitive individual-level data have limited their adoption.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
PrivTab introduces a novel architectural integration of differential privacy into tabular foundation models, enabling provably private classification with negligible utility loss and orders-of-magnitude speedup over traditional private training methods. By embedding a differentially private cross-attention mechanism that compresses sensitive data into a fixed-size private summary, the model leverages the post-processing property of DP to ensure all downstream predictions are private, while noise-aware pretraining allows the model to learn how to predict effectively through the privacy noise. This approach resolves the tension between the high utility of foundation models and the rigorous guarantees of differential privacy, offering a practical, auditable, and efficient solution for sensitive tabular data applications.
Methodology
The paper introduces PrivTab, a tabular foundation model that integrates differential privacy (DP) directly into the architecture via a "Differentially Private Multi-Head Cross-Attention" (DP-MHCA) layer. This is a significant architectural shift from traditional DP-SGD, which applies noise during iterative optimization. By compressing the context dataset into a fixed-size, noisy summary in a single forward pass, PrivTab leverages the post-processing property of DP to ensure that all subsequent predictions are private without additional privacy cost. The methodology is rigorous, featuring a formal proof of the privacy guarantee (verified in Lean 4), a noise-aware pretraining objective that teaches the model to predict through noisy summaries, and a clear separation between the privacy-critical path (the summary module) and the prediction path. The use of Gaussian DP (GDP) and adaptive composition is handled correctly, and the sensitivity analysis is sound.
Results & evaluation
The evaluation is comprehensive, utilizing the TabArena benchmark with 33 datasets across six privacy regimes. PrivTab demonstrates superior AUC and Log Loss compared to DP-Linear Regression and DP-MLP baselines, particularly in moderate-to-strong privacy settings where traditional methods suffer significant utility loss. The computational efficiency is a standout result, with fitting times reduced by ~10,000x (milliseconds vs. minutes). The membership inference attack analysis convincingly demonstrates that non-private tabular foundation models (TabPFN, TabICL) are highly vulnerable, whereas PrivTab maintains strong privacy. The inclusion of a private preprocessing case study adds practical relevance, showing how to handle feature scaling within the privacy budget.
Reproducibility
High. The authors provide a public GitHub repository containing the code, Lean 4 formal verification scripts, and empirical auditing tools. The pretraining details, including the simulator used (TabICL simulator) and hyperparameters, are described in detail. The use of standard benchmarks (TabArena) and clear evaluation metrics (AUC, Log Loss, Elo) facilitates comparison with other works.
Limitations
The model is currently limited to classification tasks, not regression. The privacy model assumes one user per row, which may not hold in all real-world scenarios (e.g., longitudinal data). The advantage diminishes on very large datasets or under very weak privacy constraints, where traditional DP-SGD methods can leverage more data. The reliance on public descriptions for clipping bounds in preprocessing is a practical constraint that may not always be met.
Broader impact
This work has the potential to significantly lower the barrier to entry for differentially private machine learning. By removing the need for dataset-specific private training and hyperparameter tuning, it makes DP accessible to domain experts who are not privacy specialists. The architectural insight—that privacy can be embedded into the foundation model's in-context learning mechanism—could inspire similar approaches in other modalities (text, image) where privacy is a concern. The formal verification aspect sets a new standard for trust in private ML implementations. PrivTab introduces a novel architectural integration of differential privacy into tabular foundation models, enabling provably private classification with negligible utility loss and orders-of-magnitude speedup over traditional private training methods. By embedding a differentially private cross-attention mechanism that compresses sensitive data into a fixed-size private summary, the model leverages the post-processing property of DP to ensure all downstream predictions are private, while noise-aware pretraining allows the model to learn how to predict effectively through the privacy noise. This approach resolves the tension between the high utility of foundation models and the rigorous guarantees of differential privacy, offering a practical, auditable, and efficient solution for sensitive tabular data applications.
Author of this paper? Send a correction.