Technical Summary: RowNet – A Memory Transformer for Tabular Regression

1. Problem Definition

The paper addresses the challenge of tabular regression, specifically in the domain of real estate valuation. Unlike image or text data, tabular data lacks stable local topology; adjacent columns are rarely semantically related, categorical features often have high cardinality with sparse levels, and numerical features exist on incompatible scales.

In real estate, the mapping from features (e.g., area, district, building age, amenities) to price is neither globally smooth nor purely additive. Markets are locally heterogeneous: a square meter in one district is not directly comparable to another without considering specific building types, conditions, and micro-regional supply. Standard deep learning models (like MLPs) treat rows as isolated vectors, struggling to learn locality and categorical matching from supervision alone. Conversely, Gradient-Boosted Decision Trees (GBDTs), while strong baselines, operate via feature-centric splitting and do not explicitly model the retrieval of similar historical observations, which is the core logic used by human appraisers.

2. Methodology: RowNet Architecture

RowNet is a retrieval-based neural architecture designed to predict real estate price-per-square-meter. It adopts a "comparable-property" inductive bias, treating the entire training set as a memory bank from which the model retrieves relevant historical examples for a given query.

2.1 Core Components

The architecture, named ThreeLayerMoE, consists of three distinct stages:

1. Feature-Only Row Retrieval (Layer 1):

   • The model computes pairwise similarity between a query row and all memory rows.
   • Similarity Embedding: Instead of raw dot products, similarity is engineered using:
     • Exact Categorical Matches: Binary equality for categorical and one-hot encoded features.
     • Numerical Kernels: Three scale-aware kernels for numerical features: inverse distance ($1/(1+\Delta)$), exponential relative similarity ($\exp(-\Delta/\alpha)$), and clipped linear relative similarity ($\max(0, 1-\Delta/\alpha)$).
   • A single learnable vector $w_1$ scores these similarity vectors to produce a coarse, attention-weighted estimate of the target.

2. Target-Conditioned Multi-Head Retrieval (Layer 2):

   • This layer refines the retrieval by augmenting the similarity vector with target-consistency features. These features measure how close a memory row's actual price is to the coarse estimate from Layer 1.
   • Multi-Head Design: The model employs $H=8$ attention heads. Each head learns a separate compatibility vector ($w_{2,h}$), allowing different heads to specialize in different aspects of comparability (e.g., location, geometry, amenities, or target consistency).
   • Softmax attention is applied independently over the memory dimension for each head, retrieving both target estimates and feature summaries.

3. Mixture-of-Experts Aggregation with Residual Correction:

   • Gate: A two-layer MLP takes the mean of the head summaries and produces a gating vector $\pi$ to weight the contributions of each head.
   • Residual Correction: Each head has an associated residual MLP ($R_h$) that predicts a small additive correction to the retrieved average. This allows the model to adjust for systematic biases (e.g., unobserved quality) that a simple average cannot capture.
   • Final Prediction: The output is a convex combination of the retrieved prices and the residual corrections, weighted by a mixture of the learned gate and a "residual-confidence" distribution (softmin over residual magnitudes).

2.2 Training and Regularization

• Objective: The primary loss is Mean Absolute Percentage Error (MAPE) on price-per-square-meter.
• Regularization:
  • Gate Entropy: Encourages the gating mechanism to use multiple heads rather than collapsing to a single head.
  • Head Diversity: A penalty term ensures the learned head vectors remain distinct, preventing redundancy.
  • Residual Penalty: Constrains the scale of the additive residual correction to prevent it from dominating the retrieval signal.
• Self-Masking: During training, the query row is masked out of its own memory (score set to $-\infty$) to enforce a leave-one-out retrieval objective.

3. Key Contributions

The paper makes the following specific contributions:

1. Architecture: Introduction of RowNet, a compact, retrieval-augmented tabular regression model that combines pairwise similarity embeddings, two-stage row attention, multi-head retrieval, and mixture-of-experts aggregation.
2. Mathematical Formalization: Detailed definition of the engineered similarity channels (exact matches, scale-aware kernels, target-consistency features) and the hybrid attention mechanism.
3. Preprocessing Pipeline: A comprehensive pipeline for real estate data, including Russian-to-English schema normalization, geographic clustering (K-Means on coordinates), address decomposition, and the construction of price-per-square-meter targets.
4. Empirical Validation: Reporting of training traces, a proof-of-concept baseline, and competition results on a real-world dataset from Bishkek, Kyrgyzstan.
5. Theoretical Analysis: Discussion of RowNet as a learned comparable-property model, contrasting it with GBDTs, tabular transformers, and k-nearest-neighbor methods.

4. Experimental Results

• Dataset: The study uses a Kaggle competition dataset from Bishkek containing 7,128 training listings and 1,784 test listings.
• Performance:
  • Proof-of-Concept: A non-learned row-attention baseline achieved 14.76% MAPE, demonstrating that raw similarity contains signal but is insufficient without learning.
  • XGBoost Baseline: Reported at 8.33% MAPE.
  • RowNet Training: The model reduced self-masked training MAPE from 8.36% to 6.37% over 10 epochs.
  • Competition Result: The final submission achieved a 7.44% MAPE (note: the abstract mentions 7.44%, while Section 6.3 mentions 8.11% as the reported score after 246 submissions; the paper presents 7.44% as the demonstrated effectiveness in the abstract and 8.11% in the results table, likely reflecting different evaluation points or submission versions).
• Ablation Insights: The paper argues that removing retrieval converts the model to a global parametric regressor (losing local adaptation), while removing the target-conditioned layer or residual correction leads to bias or reduced refinement.

5. Significance and Claims

The paper argues that learned row retrieval is a natural inductive bias for real estate valuation. It posits that:

• Hybrid Approach: RowNet successfully combines the statistical strengths of nearest-neighbor estimation (local adaptability) with the differentiability and specialization of neural attention mechanisms.
• Interpretability: Unlike "black box" transformers, RowNet's predictions can be inspected through attention weights over specific historical rows, providing a more direct explanation path: a prediction is high because specific comparable properties were retrieved and weighted highly.
• Domain Fit: The architecture explicitly models the domain practice of estimating a property's value by identifying and weighting comparable transactions, rather than forcing a global function to approximate local regimes.

6. Limitations

The authors acknowledge several limitations:

• Scalability: The full-memory attention mechanism results in $O(N^2)$ training complexity, making it computationally expensive for very large datasets (e.g., >150k rows).
• Evaluation Scope: The study does not provide held-out validation metrics for all baselines (XGBoost, CatBoost, etc.) or full ablation metrics across multiple seeds.
• Data Dependence: The model inherits biases and temporal drift from the memory bank; stale or duplicate data in the memory can lead to misleading retrievals.
• Interpretability: While better than standard transformers, the learned compatibility weights and residual adjustments still require analysis and are not fully human-readable.

The paper concludes that RowNet serves as a practical complement to gradient boosting for domains where local comparability is central, provided that scalability challenges are addressed via techniques like top-k preselection or approximate nearest-neighbor indexing.