Multiclass Classification with Rare Useful Features: Fundamental Limits and the Optimality of Diversity Pursuit Higher Criticism Thresholding
Abstract
We consider classification in high-dimensional feature spaces where only a vanishing fraction of features are useful and the number of classes may be large. While this rare-and-weak useful-feature regime is well understood in the binary setting, its multiclass version raises new questions: a feature can be useful for class discrimination without separating any specific pair of classes. We focus on a rare feature diversity model, in which most features are useless, but a small fraction promote discrimination among classes in a way that can be detected by a per-feature diversity test, such as one-way ANOVA or the Kruskal-Wallis test. For this setting, we propose Diversity Pursuit Higher Criticism Thresholding (DP-HCT), a feature-screening procedure that applies a higher-criticism threshold to the collection of p-values obtained from per-feature diversity tests. We justify DP-HCT under a Gaussian multiclass Rare-and-Weak Useful Feature model with class-mean contrasts of fixed magnitude in random directions, extending the binary model of Donoho and Jin (2008). When the number of classes is fixed, our model recovers the binary phase transition. When the number of classes grows logarithmically with the number of features, however, we derive a new and strictly harder phase transition, driven by the growing degrees of freedom and their effect on the useful-feature tail rate. We show that whenever classification beyond chance is possible, DP-HCT with ANOVA, followed by a simple classifier based on similarity to class means, achieves asymptotically perfect accuracy. Experiments on real multiclass classification benchmarks demonstrate substantial improvements, and numerical simulations confirm that the derived phase-transition curve accurately predicts the finite-sample behavior of DP-HCT in the growing-class regime.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper establishes a sharp phase transition for multiclass classification in the rare-and-weak useful feature regime, proving the optimality of Higher Criticism thresholding when the number of classes grows logarithmically with dimension. By extending the binary Donoho-Jin framework to the multiclass setting with rigorous asymptotic analysis and empirical validation on biological benchmarks, it provides both a fundamental theoretical limit and a practical, effective feature-screening tool for high-dimensional data.
Methodology
The paper proposes Diversity Pursuit Higher Criticism Thresholding (DP-HCT), a feature-screening method that extends the binary Higher Criticism (HC) framework to multiclass classification. The core innovation is the derivation of a phase transition boundary for a Gaussian multiclass Rare-and-Weak Useful Feature model where the number of classes $C$ grows logarithmically with the number of features $p$. The authors rigorously prove that when $C$ is fixed, the boundary recovers the classical Donoho-Jin result, but when $C$ grows, a strictly harder phase transition emerges due to the increased degrees of freedom in the ANOVA/Kruskal-Wallis tests. The achievability proof shows that DP-HCT, followed by a simple centroid-similarity classifier, achieves asymptotically perfect accuracy above this boundary. The theoretical analysis is robust, utilizing noncentral chi-square moment-generating functions and Chernoff bounds to handle the growing degrees of freedom.
Results & evaluation
The experimental section includes Monte Carlo simulations that accurately validate the derived phase-transition curves, showing that DP-HCT tracks the oracle threshold closely. Real-data experiments on five biological benchmarks (gene expression, metabolomics, miRNA) demonstrate substantial improvements over baselines like Bonferroni and Benjamini-Hochberg FDR. Stress tests involving noise injection and nuisance feature inflation further confirm the robustness of the screening procedure. The evaluation is rigorous, using appropriate cross-validation protocols that account for data leakage in repeated measures.
Reproducibility
The paper provides detailed algorithmic descriptions, specific dataset preprocessing steps, and cross-validation protocols. The supplementary material includes full proofs and additional simulation details. While code is not explicitly linked in the provided text, the level of detail in the experimental setup (e.g., specific GEO series, Metabolomics Workbench DOIs, and solver parameters) suggests high reproducibility.
Limitations
The theoretical guarantees are derived under a Gaussian assumption with spherical symmetric class-mean contrasts. The paper acknowledges that unbalanced designs and non-Gaussian heavy-tailed distributions (though addressed partially by the Kruskal-Wallis test) are left for future work. The real-data experiments, while showing improvements, are on relatively small sample sizes ($n < 150$), which is typical for genomics but limits the generalizability to larger-scale deep learning applications.
Broader impact
This work provides a fundamental theoretical limit for high-dimensional multiclass classification, which is crucial for understanding sample complexity in genomics and other sparse-data fields. The proposed DP-HCT method offers a practical, computationally efficient alternative to complex feature selection methods, potentially impacting how high-dimensional data is preprocessed in biological and medical research. The paper establishes a sharp phase transition for multiclass classification in the rare-and-weak useful feature regime, proving the optimality of Higher Criticism thresholding when the number of classes grows logarithmically with dimension. By extending the binary Donoho-Jin framework to the multiclass setting with rigorous asymptotic analysis and empirical validation on biological benchmarks, it provides both a fundamental theoretical limit and a practical, effective feature-screening tool for high-dimensional data.
Author of this paper? Send a correction.