Technical Summary: Test-Time Compute Scaling for ASR with Depth-Conditioned Looped Transformers

Problem Statement

End-to-end Automatic Speech Recognition (ASR) systems typically rely on fixed-depth acoustic encoders. At inference, the computational depth is rigidly determined by the architecture, preventing models from trading additional test-time computation for improved recognition accuracy without retraining a larger model. While test-time compute scaling has gained traction in autoregressive language modeling (where iterative generation allows for reasoning and self-correction), applying this paradigm to continuous, non-autoregressive ASR tasks is non-trivial. Standard fixed-depth encoders lack a mechanism to progressively refine acoustic representations during inference, and naive attempts to reuse shared Transformer blocks recurrently often fail to fully exploit the additional compute, resulting in suboptimal performance compared to deeper, unshared-parameter baselines.

Methodology: The LARM Architecture

To address these limitations, the authors propose the Loop Audio Recurrent Model (LARM), a depth-conditioned looped Transformer that decouples parameter count from inference depth. LARM applies a shared Transformer block recurrently to a latent acoustic sequence, treating computational depth as a dynamic, controllable axis. The architecture integrates four key mechanisms to structure the loop and enable stage-specific specialization:

1. Sparse Supervision and Checkpoint Structure:
   Instead of applying loss at every loop or only at the final step, LARM employs a sparse supervision schedule. The recurrent process is structured into a sequence of supervised recognition checkpoints separated by intermediate, unsupervised refinement phases. A supervision clock embedding ($e^{(k)}_{clock}$) encodes the loop's position within the checkpoint interval ($c$), allowing the model to distinguish between supervised recognition stages and latent refinement steps.

2. FiLM Depth Conditioning:
   To prevent the shared weights from performing redundant computations across iterations, LARM utilizes Feature-wise Linear Modulation (FiLM). This mechanism explicitly modulates the hidden state based on the normalized loop iteration depth ($\bar{d}^{(k)}$). Small MLPs generate scale and bias terms that condition the shared encoder, enabling it to implement specialized transformations at different stages of the recurrence without untying parameters.

3. Delayed Prediction Feedback:
   LARM introduces an ASR-specific recurrent mechanism where soft CTC posteriors from the previous iteration are projected back into the hidden space, shifted by one frame, and reinjected into the network. This "delayed prediction feedback" propagates left-to-right token-level continuity through the depth of the model, guiding the trajectory of acoustic refinement and allowing later loops to build upon evolving hypotheses.

4. State Aggregation:
   The recurrent state $h^{(k)}$ is formed by aggregating the current encoder output, the initial acoustic representation (via a skip connection), and the delayed prediction feedback. Learnable scalars ($\alpha, \beta$) control the contribution of the feedback signal, ensuring that each loop conditions on both the original acoustic features and the refined soft-token history.

Key Contributions

• Encoder-Level Test-Time Scaling: LARM introduces a framework where ASR models can dynamically scale inference computation by increasing the number of recurrent loops without increasing the number of learned parameters.
• Stage-Specialized Recurrent Training: The paper proposes a novel loop-structuring mechanism combining sparse intermediate CTC supervision, periodic clock embeddings, and FiLM depth conditioning. This allows a single set of shared weights to specialize for distinct roles (recognition vs. refinement) across recurrent iterations.
• Delayed Prediction Feedback: The authors introduce a feedback mechanism that reinjects shifted soft-token posteriors, explicitly propagating token-level continuity through the recurrent depth to improve iterative acoustic recognition.
• Empirical Validation: The work demonstrates that LARM improves Word Error Rate (WER) as the number of inference loops increases, achieving performance competitive with significantly deeper, unshared-parameter baselines while using a fraction of the parameter count.

Experimental Results

Experiments were conducted on the LibriSpeech benchmark (100h and 960h training sets).

• Performance vs. Parameter Count: On the 100h dataset, a LARM model with 4 shared blocks and 12 loops (7.7M parameters) achieved a WER of 11.34% (greedy CTC), outperforming a parameter-matched standard 4-block encoder (26.78%) and surpassing a 16-block unshared encoder (14.43%) which has nearly four times the parameters (28.9M).
• Test-Time Compute Scaling: For a fixed trained model, increasing the number of executed loops ($K$) consistently improved WER. The 960h model showed a smooth improvement trajectory, while the 100h model exhibited non-monotonic intermediate refinement but still showed gains at supervised checkpoints.
• Scaling with Width and Data: Increasing the model dimension ($d$) from 384 to 1024 further improved performance. The largest LARM configuration ($d=1024$, 85.7M parameters) achieved competitive results (3.39% WER on test-clean for 960h) against a standard 48-block encoder of equivalent parameter count.
• Ablation Studies:
  • Naive Looping: Simple recurrence improved over non-looped encoders but was inferior to full LARM, confirming the necessity of the proposed conditioning and feedback mechanisms.
  • Supervision Schedule: Moderately sparse supervision (checkpoint intervals $c > 1$) outperformed both dense supervision ($c=1$) and final-only supervision, validating the "refinement between checkpoints" hypothesis.
  • Conditioning: FiLM depth conditioning proved superior to simple additive embeddings or no conditioning.
  • Feedback: Removing prediction feedback significantly degraded performance, and using previous-frame (delayed) feedback with learned mixing scalars yielded the best results.

Significance and Claims

The paper claims that LARM successfully extends the concept of test-time compute scaling—previously dominant in autoregressive language modeling—to continuous, non-autoregressive speech recognition. By turning recurrent encoder depth into a controllable compute axis, LARM allows ASR systems to trade additional inference time for improved accuracy without the cost of training larger models. The results suggest that shared-parameter architectures, when properly conditioned and structured with sparse supervision and feedback, can achieve performance comparable to much deeper, unshared models. The authors note that while the current configuration shows signs of saturation at very large widths, the approach offers a parameter-efficient pathway for dynamic inference scaling in ASR.