Technical Summary: SIREM – Speech-Informed MRI Reconstruction with Learned Sampling

Problem Statement

Real-time magnetic resonance imaging (rtMRI) of speech production is essential for visualizing dynamic vocal-tract motion without ionizing radiation, supporting both speech science and clinical assessment. However, rtMRI is fundamentally constrained by the trade-off between spatial resolution, temporal resolution, and acquisition speed. To achieve high frame rates, systems must undersample k-space, rendering reconstruction a strongly ill-posed inverse problem.

While existing methods exploit temporal structure (e.g., low-rank models) or coil redundancies (parallel imaging), they typically ignore the synchronized audio waveform available during acquisition. The authors posit that the acoustic signal is not merely an annotation but a physical consequence of the vocal-tract configuration. Since articulators (tongue, lips, velum) directly shape the spectral envelope and formant trajectories, the audio channel contains structured cues about the underlying image content that are currently discarded by standard reconstruction pipelines.

Methodology: SIREM Framework

The authors propose SIREM, a hybrid framework that formulates speech rtMRI reconstruction as a multimodal inverse problem. The core hypothesis is that part of the articulatory configuration is predictable from synchronized speech, while the remaining anatomy must be recovered from measured MRI data.

1. Architecture

SIREM models each frame as a fusion of two components via a spatial weighting map:

• Audio-Driven Branch: A frozen HuBERT encoder extracts contextual speech features from the synchronized audio segment. These features are temporally pooled and passed through a trainable image decoder to produce an audio-driven estimate ($x^a_t$). This branch aims to capture articulator geometries strongly coupled to acoustics (e.g., tongue, lips, velum).
• MRI-Driven Branch: This branch processes undersampled multi-coil k-space data ($k_t$). It applies a learnable soft spiral-arm profile ($p$) to the raw measurements before reconstruction. The profile is parameterized by trainable logits and passed through a sigmoid function, allowing for differentiable modulation of arm usage. The weighted k-space is reconstructed using an adjoint Non-Uniform Fast Fourier Transform (NUFFT) and SENSE coil combination to produce an MRI-driven estimate ($x^m_t$).
• Spatial Fusion: The final frame ($\hat{x}_t$) is generated by a pixelwise weighted sum:
  $$\hat{x}_t = w_{EbA} \odot x^a_t + (1 - w_{EbA}) \odot x^m_t$$
  Here, $w_{EbA}$ is the "explained-by-audio" spatial weighting map. In the reported experiments, this map is derived from manual segmentation masks (highlighting articulator regions) rather than being learned, to provide an interpretable anatomical prior and stabilize training on the limited USC-16 dataset.

2. Learning and Optimization

The framework is trained to minimize a composite loss function:

• Reconstruction Loss ($L_{recon}$): $\ell_2$ fidelity between the fused output and the reference frame.
• Point Spread Function Regularization ($L_{psf}$): Penalizes the energy of the inverse Fourier transform of the arm profile $p$ to discourage irregular weighting that causes artifacts.
• Budget Regularization ($L_{budget}$): Encourages the sum of arm weights to match a target effective budget ($K=2$).
• Mask Regularization ($L_{mask}$): Penalizes the Fourier energy of the complement of $w_{EbA}$ to ensure smooth transitions between audio-dominant and MRI-dominant regions.

3. Sampling Policy

A key innovation is the learnable soft spiral-arm profile. Unlike static undersampling patterns, this profile is optimized jointly with the reconstruction. While interpreted retrospectively in this study (as a differentiable surrogate for arm importance), it enables the investigation of how speech-informed fusion interacts with specific k-space arm usage.

Key Contributions

1. Multimodal Formulation: The authors formulate speech rtMRI reconstruction as a multimodal estimation problem where synchronized audio serves as a cross-modal prior on vocal-tract structure.
2. Hybrid Framework: SIREM introduces a method that fuses audio-driven predictions with MRI-driven reconstructions via a spatial weighting map, coupled with a learnable soft spiral-arm reweighting profile.
3. Benchmarking: The work establishes an initial benchmark for multimodal speech-informed rtMRI reconstruction on the USC corpus, comparing against standard baselines (gridding, wavelet-based compressed sensing, and total variation).

Results

The evaluation was conducted on the USC speech rtMRI corpus (USC-16 subset) using a subject-independent split.

• Fidelity vs. Throughput: SIREM does not uniformly outperform classical iterative methods on distortion-based metrics (e.g., PSNR, SSIM). Wavelet-based compressed sensing and Total Variation (TV) retain the strongest scores in most evaluation protocols.
• Value of Audio: Despite not matching the absolute fidelity of classical baselines, SIREM consistently outperforms its own ablated variant (SIREM without audio) across all 21 metric-target comparisons. This confirms that synchronized speech provides useful complementary information.
• Efficiency: The most significant advantage of SIREM is computational throughput.
  • SIREM: ~14.6 ms per frame (feed-forward inference).
  • Wavelet/TV: ~600–658 ms per frame (100 optimization iterations).
  • SIREM is approximately 41–45× faster than iterative compressed sensing baselines, operating within the real-time regime (<33.3 ms/frame), whereas iterative methods do not.
• Qualitative Performance: SIREM produces reconstructions with sharper articulatory boundaries compared to gridding and maintains a direct inference pipeline, though it may preserve less fine anatomical detail than TV in some contexts.

Significance and Claims

The paper claims that SIREM establishes a new paradigm for rtMRI reconstruction that prioritizes a distinct fidelity-efficiency trade-off. While classical iterative methods offer higher fidelity, they are computationally prohibitive for real-time applications. SIREM demonstrates that:

1. Feasibility: Direct, multimodal reconstruction using synchronized audio as an auxiliary prior is feasible.
2. Throughput: The method operates in a substantially higher-throughput regime than iterative approaches, making it suitable for real-time applications where speed is a critical constraint.
3. Auxiliary Prior: Synchronized speech is a valid and useful prior for reconstructing undersampled speech rtMRI directly from raw k-space.

The authors conclude modestly, noting that while SIREM does not yet surpass classical baselines in pure image fidelity, it successfully demonstrates the potential of multimodal fusion for fast reconstruction. They acknowledge limitations, including the use of fixed segmentation masks for weighting and the retrospective nature of the sampling profile, suggesting future work should explore learned fusion predictors and prospective sampling strategies.