Machine Learning PapersLandmark papers
← All papers
RESEARCH PAPER / 2026-11-13

Provable Cross-Modal Transfer of Hierarchical Concept Structure in CLIP Embeddings

Nghia Nguyen, Tianjiao Ding, Lachlan Ewen MacDonald, René Vidal

University of Pennsylvania · arXiv

Affiliations: University of Pennsylvania

Abstract

Contrastive vision-language models such as CLIP are trained to align images and text, not to represent taxonomies. Yet empirical studies show that CLIP embeddings exhibit hierarchical structure aligned with the hierarchy of semantic concepts. A theoretical characterization of why such hierarchy emerges from a non-hierarchical objective remains open. In this paper, we formalize this phenomenon as Hierarchical Angular Separation (HAS), a geometric property whereby each non-leaf concept embedding is more similar to its descendants than to off-branch concepts. Empirically, we show that HAS holds approximately in raw images and randomly initialized CLIP image embeddings, while in text embeddings it holds approximately only after contrastive learning. To theoretically explain this asymmetry, we analyze an unconstrained features model in which the image embeddings are fixed and only the text embeddings are trained. At the global minimizer, the text-side HAS violation percentage at each node is at most the image-side HAS violation and near-violation percentage. The key mechanism is the hierarchical co-occurrence structure of image-text pairs, which biases text embeddings toward root-centered image prototypes. For a linear contrastive loss, we derive a closed-form solution that makes this mechanism explicit. We extend the analysis to the InfoNCE loss and establish analogous guarantees under sufficient regularization. Synthetic experiments validate the theory and show that HAS is transferred even beyond the regimes covered by our analysis.

The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.

Research summary

The paper formalizes Hierarchical Angular Separation (HAS) and provides a theoretical explanation for the emergence of hierarchical structure in CLIP text embeddings via the co-occurrence structure of image-text pairs. While the theoretical contribution is significant and well-motivated, the reliance on simplified models (fixed image embeddings) and the limited experimental scope prevent it from reaching the highest tier of impact, though it remains a strong contribution to the theory of representation learning.

Methodology

The paper introduces "Hierarchical Angular Separation" (HAS) to formalize the geometric structure of CLIP embeddings. The core contribution is a theoretical analysis of an unconstrained features model where image embeddings are fixed and text embeddings are trained. The authors derive a bound showing that at the global minimizer, the text-side HAS violation is bounded by the image-side violation. They provide a closed-form solution for a linear contrastive loss and extend the analysis to InfoNCE with regularization. The methodology is rigorous but relies on strong simplifying assumptions (fixed image embeddings, global minimizer) that may not hold in practical, finite-time training scenarios.

Results & evaluation

The paper provides empirical validation showing HAS holds approximately in raw images and trained CLIP text embeddings, but not in random text embeddings. Synthetic experiments validate the theoretical bounds. However, the experimental section appears limited in scope, focusing primarily on validating the theory rather than demonstrating broad practical utility or out-of-the-box performance improvements on downstream tasks.

Reproducibility

The paper is likely reproducible as it relies on standard CLIP models and synthetic data generation. However, specific hyperparameters for the "unconstrained features model" and the exact definition of "near-violation percentage" would need to be clearly specified in the full text (which is truncated here) to ensure full reproducibility.

Limitations

The primary limitation is the gap between the theoretical model (fixed image embeddings, global minimizer) and reality (joint training, local minima, finite data). The analysis may not capture the dynamics of how hierarchy emerges during the actual training process of both modalities simultaneously.

Broader impact

This work provides a theoretical foundation for understanding why CLIP-like models exhibit hierarchical structure, which is valuable for the interpretability and design of future multimodal models. It helps explain the "emergent" properties of contrastive learning. The paper formalizes Hierarchical Angular Separation (HAS) and provides a theoretical explanation for the emergence of hierarchical structure in CLIP text embeddings via the co-occurrence structure of image-text pairs. While the theoretical contribution is significant and well-motivated, the reliance on simplified models (fixed image embeddings) and the limited experimental scope prevent it from reaching the highest tier of impact, though it remains a strong contribution to the theory of representation learning.

Author of this paper? Send a correction.