Controlling Long-Horizon Behavior in Language Model Agents with Explicit State Dynamics

Technical Summary: Controlling Long-Horizon Behavior in Language Model Agents with Explicit State Dynamics

Problem Statement
Large Language Model (LLM) agents frequently exhibit abrupt, uncoordinated shifts in tone and persona during extended multi-turn interactions. This phenomenon, termed "instantaneous affective reversal," undermines social consistency and user confidence. While prior research has addressed this through memory mechanisms (e.g., retrieval-augmented memory) or persona-based prompting, these methods focus on information storage rather than governing the temporal evolution of internal states. Consequently, agents may possess extensive context yet still display discontinuous emotional transitions. The core issue identified is a lack of inertia: human emotional states change gradually due to momentum and resistance to sudden shifts, a property absent in standard stateless LLM inference.

Methodology
The authors propose a framework that imposes explicit dynamical structure on an external, low-dimensional affective state, operating as an inference-time overlay without modifying the underlying LLM parameters.

• Latent Representation: The agent's state is represented by a continuous Valence-Arousal-Dominance (VAD) vector, $\mathbf{a}_t \in \mathbb{R}^3$, maintained externally. The analysis focuses primarily on the valence component ($v_t$).
• Signal Extraction: An instantaneous, memoryless affective signal ($\hat{\mathbf{a}}_t$) is extracted from the dialogue context at each turn using a fixed, lexicon-based sentiment analyzer (VADER).
• Dynamical Regimes: The study compares four conditions to evaluate the impact of state persistence and inertia:
  1. Stateless Control: No state is maintained; $\mathbf{a}_t = \hat{\mathbf{a}}_t$.
  2. First-Order Dynamics: State persistence without momentum ($\mu=0$), modeled as exponential smoothing: $\mathbf{a}_{t+1} = \mathbf{a}_t + \alpha(\hat{\mathbf{a}}_t - \mathbf{a}_t)$.
  3. Second-Order Dynamics (Moderate Inertia): Introduces affective velocity ($\mathbf{u}_t$) and momentum ($\mu=0.8$) to simulate damped physical systems.
  4. Second-Order Dynamics (High Inertia): Strong momentum ($\mu=0.95$) to test excessive persistence.
• Feedback Loop: The updated affective state is injected back into the generation context via a structured natural language prompt, modulating the LLM's tone and discourse style.
• Experimental Protocol: Experiments utilize a fixed 25-turn dialogue protocol involving an adversarial phase (turns 6–13) followed by a reconciliation phase (turns 14–25). Five independent runs per condition (using different sampling seeds) were conducted on a 7B-parameter quantized LLM.

Key Contributions

1. Explicit State Dynamics: The paper introduces a method to regulate long-horizon agent behavior by modeling affective state as a continuous variable governed by first- and second-order update rules, rather than relying solely on memory retrieval.
2. Inertia as a Control Mechanism: It demonstrates that affective inertia (modeled via momentum) can be tuned to induce temporal coherence, delayed responses, and hysteresis, properties not achievable with stateless or purely reactive models.
3. Inference-Time Overlay: The framework requires no model retraining or fine-tuning, functioning as a plug-in subsystem compatible with any pre-trained LLM.

Results

• Temporal Structure: Stateless agents exhibited locally reactive but temporally unstructured behavior with no coherent descent-recovery patterns. In contrast, stateful agents (first-order and second-order) displayed delayed affective responses to adversarial inputs.
• Recovery Dynamics:
  • Stateless and High-Inertia ($\mu=0.95$) agents failed to recover (return to positive valence) within the 25-turn horizon.
  • First-Order and Moderate-Inertia ($\mu=0.8$) agents consistently recovered at the onset of the reconciliation phase (turn 14).
  • Recovery timing was governed by external forcing (the dialogue protocol) rather than internal momentum; inertia affected the depth and curvature of the trajectory but not the timing of recovery.
• Hysteresis and Path Dependence: Affective hysteresis (the area between descent and recovery trajectories) increased monotonically with inertia. The stateless condition had zero hysteresis, while the high-inertia condition produced the largest hysteresis (AUC = 21.19), confirming that second-order dynamics introduce distinct path dependence.
• Stability-Responsiveness Trade-off: Moderate inertia balanced stability and responsiveness, allowing for smooth transitions and recovery. Excessive inertia ($\mu=0.95$) led to "locking" in negative affective states, preventing timely recovery despite external reconciliation.

Significance and Claims
The paper claims that incorporating affective inertia provides a systematic mechanism for creating agents that are not only competent but temporally coherent. By anchoring dynamics in explicit physical principles, the authors argue that agent behavior can be stabilized against spurious mood changes while maintaining sensitivity to legitimate contextual shifts.

The work suggests that agent stability does not solely depend on memory capacity or architectural modifications but can be achieved through external, low-dimensional dynamical subsystems. This approach offers interpretable control over temporal properties like persistence, lag, and hysteresis, which are difficult to enforce via prompt engineering alone. The authors position this as a complementary approach to existing memory-based methods, particularly valuable for domains requiring sustained social engagement and emotional dependability, such as human-robot interaction and digital mental health interventions.

Limitations
The authors acknowledge that affect extraction relies on a coarse, lexicon-based estimator and that experiments were limited to a single dialogue protocol. They note that the evaluation horizon was finite, and longer interactions might reveal different dynamical regimes, such as eventual recovery in high-inertia scenarios. Future work is suggested to explore adaptive inertia and multidimensional analysis beyond valence.