Technical Summary: Categorize Early, Integrate Late

Problem Statement

While Transformer and Conformer architectures currently dominate the frontier of Automatic Speech Recognition (ASR), achieving comparable Word Error Rates (WER) on standard benchmarks, it remains unknown whether this performance parity stems from convergent internal processing strategies or distinct architectural inductive biases. Unlike text modeling, where architectural uniformity (the Transformer) obscures representational convergence, the coexistence of these two speech encoder families offers a unique opportunity to determine if different architectures solving the same task learn fundamentally different internal hierarchies. Understanding these differences is critical for interpretability, demographic fairness auditing, and deployment decisions regarding latency and streaming.

Methodology

The authors introduce Architectural Fingerprinting, a probing framework designed to isolate the effect of architecture on representation by controlling for model scale, dataset, and training regime.

• Model Suite: The study analyzes 24 pre-trained spoken language models (17 Transformers and 7 Conformers) ranging from 39M to 3.3B parameters. All models were evaluated in their pre-trained state without fine-tuning.
• Datasets: Probing was conducted across over 50,000 utterances from seven diverse speech corpora (including L2-ARCTIC, CMU ARCTIC, Common Voice, and Speech Accent Archive) to ensure generalizability across native/non-native speakers and recording conditions.
• Feature Taxonomy: Following the signal-to-symbol hierarchy, the authors probe for four levels of abstraction:
  1. Acoustic: F0, formants, intensity.
  2. Demographic: Gender and L1 accent.
  3. Phonetic: Phoneme categories (39 categories from the CMU dictionary).
  4. Temporal: Utterance-level duration.
• Probing Mechanism: Linear probes (linear regression for continuous targets, logistic regression for categorical) are trained at every layer to predict these features from hidden states. The study measures linear accessibility—when a feature becomes extractable by a simple readout—rather than making causal claims about the underlying mechanism.
• Metrics:
  • Peak Position ($\pi_f$): The normalized depth (0 to 1) where probe performance is maximized.
  • Peak Strength ($s_f$): The maximum performance achieved.
  • Positional Deltas ($\Delta$): The difference in peak positions between feature pairs to quantify hierarchical separation.
• Statistical Controls: Two-sample t-tests and linear regression models were used to isolate architectural effects while controlling for parameter count. A logistic regression classifier was trained to predict architecture type solely from peak position profiles.

Key Results

The analysis reveals two divergent, architecture-dependent processing strategies that persist across scales and training regimes:

1. Divergent Hierarchies

• Conformers ("Categorize Early"): Conformers exhibit a segregated hierarchy where categorical features are resolved in shallow layers.
  • Gender peaks at 16% of network depth.
  • Phoneme peaks at 21% of network depth.
  • This front-loading occurs before the resolution of fine-grained acoustic details.
  • Duration (requiring long-range integration) is deferred to the final layers (~70% depth).
• Transformers ("Integrate Late"): Transformers implement an integrated strategy where high-level linguistic and sociolinguistic information co-locate in deeper layers.
  • Phoneme, Accent, and Duration all peak within a narrow band of 49–57% of total depth.
  • Gender is distributed more broadly (mean peak at 28%) compared to Conformers.
  • The architecture resolves multiple feature types concurrently in the latter half of the network rather than through a staged pipeline.

2. Statistical Significance

• Phoneme Divergence: The largest observed effect is in phoneme categorization, where Conformers peak 29% earlier than Transformers ($\Delta\pi = -0.29, p < 0.01$).
• Accent and Gender: Conformers also peak significantly earlier for accent ($\Delta\pi = -0.225$) and gender ($\Delta\pi = -0.126$).
• Architectural Fingerprinting: A logistic regression classifier can distinguish between Transformer and Conformer architectures based solely on the 5-dimensional vector of peak positions with an AUC of 0.88. This confirms that representational profiles are architecturally distinctive.
• Training Regime Modulation: While architecture sets the default profile, training objectives modulate specific features. For example, Whisper models (multilingual, supervised) delay gender and accent processing compared to self-supervised Transformers, yet the underlying "Integrate Late" strategy relative to Conformers persists.

Significance and Claims

The paper claims that high-performance speech recognition does not imply representational convergence. Instead, benchmark parity can arise from fundamentally different computational solutions shaped by architectural inductive bias.

• Design Heuristics: The findings suggest practical selection criteria:
  • Streaming/Low-Latency: Conformers may be preferable for streaming applications because useful phoneme and gender representations are available much earlier (21% depth vs. 49%).
  • Speaker Normalization: Transformers may be better suited for tasks requiring rich context and cross-utterance normalization, as they tightly integrate speaker cues (gender, accent) with phoneme categories in deep layers.
• Interpretability: The framework provides a diagnostic tool for model developers to look beyond standard metrics (like WER) to assess whether a model's internal processing strategy aligns with its intended use case.
• Limitations: The authors explicitly note that their claims are restricted to linear accessibility, not the first emergence of information in any form. The analysis is correlational; while architecture predicts hierarchy, the study does not definitively establish causation without controlled experiments (e.g., modifying architectures from scratch). Additionally, the feature taxonomy focuses on the signal-to-phoneme transformation and does not cover lexical or discourse-level organization.

In conclusion, the paper establishes that the "path to expert performance is not singular," with Conformers utilizing a local-then-global inductive bias and Transformers relying on global-only attention to achieve distinct but equally effective processing strategies.