Technical Summary: Latent-IM for Speech LLMs

Problem Statement

Classical spoken dialogue systems traditionally separated dialogue management (state estimation and policy selection) from response realization (generation). In this paradigm, a dialogue manager maintained a belief state and selected an action, which was then realized by a separate component. However, as systems shift toward Large Language Models (LLMs), this decomposition has largely disappeared into the model's hidden representations. Modern LLMs generate responses end-to-end without explicit state modules or action policies.

The central question addressed by this work is whether an internal analogue of state estimation and action control can be recovered from the hidden representations of a frozen speech LLM to govern conversational behavior. Specifically, the authors investigate if the model can be guided to perform specific conversational "moves" (e.g., acknowledging, checking, querying, explaining, replying) and control turn-taking (when to yield the floor) without fine-tuning the backbone or using symbolic policies.

Methodology: Latent-IM

The authors introduce Latent-IM, a framework that recovers interaction management (IM) from the residual-stream activations of a frozen speech LLM. The framework decomposes move control into two coupled problems: Selection and Realization.

1. Selection: Predicting the Next Move

A lightweight, recurrent controller acts as a "sidecar" to the frozen backbone.

• Input: The controller reads mean-pooled residual-stream activations from the backbone at a specific layer ($\ell^*_{ctrl}$) for each dialogue turn.
• Architecture: It utilizes a selective State-Space Model (SSM) to maintain a persistent streaming state across turns, updating its internal state $z_t$ based on the current context representation $x_t$.
• Output: A linear head predicts the next conversational move ($\hat{y}_t$) from a taxonomy of five domain-general moves: acknowledge, check, explain, query, reply. An auxiliary head simultaneously regresses the log-response length.
• Training: The controller is trained using next-move cross-entropy and smooth-L1 loss on the training split, while the backbone remains frozen.

2. Realization: Causal Move Expression

Once a move is selected, the framework uses activation steering to causally realize that move during generation.

• Direction Estimation: For each move type, the authors estimate a "move direction" vector ($v_m$) in the residual stream. This is derived from the difference in activations between the onset of a response and the preceding context token. The direction is computed using a Fisher discriminant approach (PCA followed by Ledoit-Wolf shrinkage) on the training data (MapTask) to maximize class separability.
• Steering Mechanism: During inference, the selected move direction is injected into the residual stream at a specific layer ($\ell^*_m$) for every generated token:
  $$h^{\ell^*_m}(s) \leftarrow h^{\ell^*_m}(s) + \alpha c^{\ell^*_m} v^{\ell^*_m}_m$$
  where $\alpha$ is a dimensionless steering coefficient and $c$ is a normalization factor based on the median residual norm.
• Turn-Boundary Control: A separate "turn-boundary direction" ($v_{EOT}$) is derived to control response length and when the model yields the floor. Positive steering coefficients encourage termination, while negative coefficients prolong the response.
• Realization-Aware Gating: The system employs a gating rule that withholds steering for moves where the backbone naturally performs well (e.g., reply) or where steering might degrade performance (e.g., shortening content-heavy moves like explain).

Key Contributions

1. Latent-IM Framework: An end-to-end dialogue management system that operates without gold move labels or backbone updates. It combines activation-based selection and realization, outperforming other control baselines and matching supervised fine-tuning (SFT) in end-to-end accuracy.
2. Lightweight Streaming Controller: Demonstrates that a lightweight controller can recover conversational state from frozen LLM activations, outperforming transcript-based selectors across three task-oriented datasets (MapTask, FindTask, CReST).
3. Reusable Move-Specific Directions: Derives activation directions that causally realize conversational actions. These directions, learned from MapTask, transfer without re-estimation to FindTask and CReST, outperforming prompting, candidate-selection, and decoding-time control baselines.
4. Turn-Boundary Subspace: Identifies a distinct direction that causally controls response length, allowing for monotonic variation of mean response length (from 71.3 to 10.4 words) and integration with realization-aware gating to regulate turn-taking.

Results

The framework was evaluated on three multimodal language models (Qwen2.5-Omni, Qwen3-Omni, Phi-4-Multimodal) across three datasets.

• Selection Accuracy: The streaming activation controller achieved an average next-move selection accuracy of 0.60, outperforming transcript-based selectors (e.g., TF-IDF + logistic at 0.48) and few-shot LLM classifiers. It approached the performance of a Markov baseline using gold history (0.65) on the MapTask dataset.
• Oracle Realization: When provided with the gold target move, Latent-IM improved move realization accuracy by 10.4 percentage points over the strongest baseline (DeAL), achieving a mean accuracy of 60.0%.
• End-to-End Control: In the fully autonomous setting (no gold labels at inference), Latent-IM achieved an average end-to-end move accuracy of 46.9%, slightly exceeding Supervised Fine-Tuning (SFT) at 46.6% and significantly outperforming the unsteered backbone (34.4%) and prompting baselines.
• Length Control: Steering along the turn-boundary direction allowed for graded control over response length, reducing mean length from 71.3 words to 10.4 words while maintaining fluency.

Significance and Claims

The paper claims that frozen speech LLM activations serve as an observable and causally controllable interface for local interaction management. By separating the selection of a conversational move from its realization, Latent-IM recovers a classical state-action decomposition within modern LLMs without requiring parameter updates.

The authors emphasize that their approach provides a modular, label-free inference pipeline that can match the performance of direct supervised fine-tuning. They note that while the controller currently imitates human move choices, the framework is designed to support future goal-conditioned policies that select moves based on predicted task utility rather than imitation likelihood. The work suggests that the "state" of a dialogue is linearly available in the residual stream and can be read out more reliably than it can be prompted for, offering a new pathway for controlling LLM behavior in spoken dialogue systems.