TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings
Abstract
Real-world tables are often heterogeneous, combining numerical features with free-form string columns. Table foundation models (TFMs) excel on numerical tables but require strings to be encoded numerically, while large language models (LLMs) hold semantic knowledge but are not designed for tabular predictions. Most existing bridges use LLMs as plug-in encoders for cell strings, yet how to extract representations effectively for tabular learning remains underexplored. We show that the interface between LLMs and tables matter as much as the LLM itself. Applied cell-wise, larger LLMs do not give better features. With our interface, the same LLMs improve steadily with size. TILE (Table-Interfaced LLM Embeddings) combines four ingredients: row serialization to provide cross-column context, intermediate-layer extraction to avoid layers specialized for next-token prediction, columnar pooling to remove mean pooling's length bias, and by-pass features to route numbers and categories past the LLM. Paired with TFMs, TILE sets a new state of the art for tables with strings. Across 100 datasets, it beats the strongest baseline on 89% of splits, with gains across 7 LLMs and 10 tabular learners, and ranks first on the 108-dataset STRABLE benchmark.
The following analysis is generated by AI and may contain errors. It is separate from the authors’ abstract and is not peer review.
Research summary
The paper introduces TILE, a robust interface for bridging LLMs and Table Foundation Models, demonstrating that proper serialization, layer selection, and pooling are critical for leveraging LLM scale in tabular learning. By addressing the specific limitations of cell-wise encoding and last-layer extraction, TILE achieves state-of-the-art results on heterogeneous tables, offering a practical and theoretically grounded solution to a significant gap in applied machine learning.
Methodology
The paper proposes TILE, a framework for integrating Large Language Models (LLMs) with Table Foundation Models (TFMs) by treating the "interface" as a critical design space. The methodology is grounded in a clear diagnosis of why naive LLM integration fails: cell-wise isolation lacks context, and last-layer extraction captures next-token prediction biases rather than transferable semantics. The four proposed components—row serialization, intermediate-layer extraction, columnar pooling, and by-pass features—are logically derived from these failures. The "by-pass" mechanism is particularly insightful, acknowledging that LLMs are poor at numerical precision and that exact categorical identity is often more useful to tabular learners than semantic embeddings. The adaptive PCA dimensionality based on table width is a practical engineering choice that addresses the feature-count constraints of TFMs.
Results & evaluation
The experimental evaluation is extensive and rigorous. The authors test across 100 datasets from multiple sources (TextTabBench, CARTE, WikiDBs) and validate on the external STRABLE benchmark. The comparison against 30 baselines, including state-of-the-art TFMs (TabPFN, TabICL, Causilo) and LLM-based methods (TabLLM, ContextTab), is comprehensive. The key finding that "larger LLMs do not give better features" when applied cell-wise, but do so when using TILE, is a significant empirical contribution that justifies the method. The ablation studies effectively isolate the contribution of each component (context, depth, pooling).
Reproducibility
The paper provides sufficient detail for reproduction, including specific layer indices for extraction, PCA component counts, and cardinality thresholds. The use of standard, off-the-shelf LLMs and TFMs lowers the barrier to entry. However, the lack of a provided code repository in the text (and the "none" for PROJECT_URL) slightly hinders immediate reproducibility, though the method is described clearly enough to be implemented by a skilled practitioner.
Limitations
The method relies on the availability of a suitable TFM; if the TFM is weak, the gains from TILE may be diminished. The row serialization approach may struggle with extremely wide tables where the context window of the LLM is exceeded, although the adaptive PCA helps with the output side. The paper focuses on classification and regression; other tabular tasks (e.g., anomaly detection) are not explored.
Broader impact
This work has high practical impact for industries dealing with heterogeneous tabular data (finance, healthcare, e-commerce). It provides a clear recipe for leveraging existing LLMs without fine-tuning them for tabular tasks, which is cost-effective. The insight that the interface matters more than the model size for this specific task will likely influence future research in hybrid tabular-language models. The paper introduces TILE, a robust interface for bridging LLMs and Table Foundation Models, demonstrating that proper serialization, layer selection, and pooling are critical for leveraging LLM scale in tabular learning. By addressing the specific limitations of cell-wise encoding and last-layer extraction, TILE achieves state-of-the-art results on heterogeneous tables, offering a practical and theoretically grounded solution to a significant gap in applied machine learning.
Author of this paper? Send a correction.