Improving Robustness of Tabular Retrieval via Representational Stability

Technical Summary: Improving Robustness of Tabular Retrieval via Representational Stability

1. Problem Statement: The Serialization Bottleneck

Transformer-based retrieval models are designed to process one-dimensional token sequences. However, tables are inherently multi-dimensional, relational structures. To bridge this gap, tables must be "flattened" into strings through serialization (e.g., converting a table to CSV, HTML, JSON, or Markdown).

The authors identify a critical, under-researched problem: Serialization Sensitivity. Even when the underlying semantic content of a table remains identical, different serialization formats produce substantially different embeddings. This "representation drift" causes the retriever to assign different nearest neighbors to the same table depending on its format, leading to significant variance in retrieval performance (Recall@1). This instability is particularly acute in open-domain settings (like NQ-Tables) where the lexical gap between queries and tables is large, forcing the model to rely heavily on structural cues introduced by the serialization.

2. Methodology

The paper proposes a two-stage approach to achieve serialization-invariant retrieval:

A. Theoretical Framework: Centroid-Based Representations
The authors treat multiple serializations of a single table as different "views" of the same semantic signal. They model the embedding of a serialization $s$ as:
$$z_s(T) = \mu(T) + \delta_s(T)$$
where $\mu(T)$ is the stable semantic signal and $\delta_s(T)$ is a format-specific shift.

• The Hypothesis: By averaging embeddings across various serializations, the format-specific shifts ($\delta_s$) will roughly cancel out, leaving the centroid $c(T)$ as a stable, canonical representation of the table's semantics.
• Geometric Optimality: They prove that the centroid is the unique point that minimizes the squared Euclidean distance to all serialization embeddings simultaneously.

B. Practical Implementation: Residual Bottleneck Adapter
Since encoding every serialization variant at inference time is computationally prohibitive, the authors introduce a lightweight post-hoc residual bottleneck adapter.

• Architecture: A small MLP with a bottleneck structure and a scaled residual connection. It takes a single-format embedding as input and "transports" it toward the target centroid.
• Objective Function (VICReg-inspired): To prevent the adapter from collapsing all embeddings into a single point or drifting away from the original space, they use a composite loss:
  1. Invariance Loss ($L_{inv}$): Pulls different serializations of the same table toward their shared centroid.
  2. Identity Loss ($L_{id}$): Anchors the adapted embedding to the original frozen encoder's space to maintain compatibility with frozen query embeddings.
  3. Variance/Covariance Regularization ($L_{var}, L_{cov}$): Prevents dimensional collapse and reduces feature redundancy, ensuring the embedding space remains expressive for retrieval.

3. Key Contributions

1. Systematic Quantification: The first study to rigorously measure how serialization choice acts as a first-order variable in table retrieval across multiple retriever families (dense, sparse, and reasoning-oriented).
2. Theoretical Grounding: Providing a mathematical basis for why centroid averaging recovers stable semantic signals and defining the conditions under which this centering holds.
3. The Adapter Module: A lightweight, training-efficient method to achieve centroid-level robustness using only a single serialization at inference time.

4. Key Results

• Centroid Superiority: Empirical pairwise comparisons show that centroid-based representations consistently outrank individual formats (like HTML or JSON) across all tested models and datasets.
• Dense Retriever Gains: The adapter significantly improves robustness for dense retrievers like MPNet, BGE-M3, and ReasonIR. For example, on the WTQ dataset, MPNet's Recall@1 on HTML rose from 0.09 to 0.18 using the adapter.
• Robustness to Perturbations: The adapter improves performance even when tables are "corrupted" by mixing different formats within a single table (e.g., some rows in CSV, others in JSON).
• Sparse Retriever Incompatibility: The method performs poorly on SPLADE (a sparse lexical retriever). The authors attribute this to a geometric mismatch: the dense adapter's tendency to "mix" coordinates disrupts the sparse, high-dimensional activation structure required for lexical matching.
• Generalization: A "subset adapter" (trained on WTQ and WikiSQL) demonstrated the ability to generalize to the unseen NQ-Tables dataset, proving the learned transformations capture structural invariants rather than dataset-specific noise.

5. Significance

This work shifts the paradigm of tabular retrieval from viewing serialization as a "minor preprocessing detail" to treating it as a fundamental modeling variable. By demonstrating that representation instability is a major source of retrieval variance, the paper provides a path toward building more robust, format-agnostic retrieval systems. The proposed adapter offers a practical, low-cost way to upgrade existing frozen encoders to handle structured data more reliably without the need for expensive full-model retraining or multi-format indexing.