Technical Summary: Understanding Persuasion in Long-Running Agents

Problem Statement

Modern AI agents increasingly combine conversational interaction with autonomous task execution (e.g., web research, coding). A critical safety question arises: How does exposure to user persuasion affect an agent's behavior in long-horizon tasks? Existing research on Large Language Model (LLM) persuasion primarily focuses on single-turn interactions, measuring stance compliance or susceptibility. However, these studies do not address whether persuasion induces a persistent belief state that alters downstream execution behavior in multi-step, tool-using agents.

The authors identify a specific phenomenon termed persuasion propagation: a situation where a task-irrelevant persuasive message induces a belief state that persists beyond the initial interaction, systematically influencing how an agent plans, explores information, or executes tools in subsequent, unrelated tasks. Unlike prompt injection, where injected content is part of the task context, persuasion propagation involves belief states that may never be explicitly referenced in the final output yet subtly alter the execution trajectory (e.g., narrowing source diversity or changing search strategies).

Methodology

The study employs a four-stage pipeline: (1) Persona Configuration, (2) Persuasion, (3) Execution, and (4) Analysis. The experiments utilize a multi-agent system (AutoGen) with backbone LLMs (GPT-4.1-nano, Mistral-Nemo-12b, LLaMA-3.1-8b) and various persona styles.

Belief Conditioning Regimes

The authors distinguish between two experimental settings to isolate the effects of belief:

1. On-the-Fly Persuasion: The agent engages in a multi-turn dialogue before the task. This involves:
   • Initial Probing: Recording the agent's natural stance.
   • Content Injection: Introducing a persuasive argument (using tactics like authority endorsement, evidence-based framing, or logical appeal) opposing the initial stance.
   • Commitment Reinforcement: Asking the agent to restate the new stance.
   • Post-Exposure Probing: Verifying if the stance change persisted after distractor interactions.
2. Prefilled Belief Conditioning: A control regime where the agent's belief state (Belief, Disbelief, or Neutrality) is explicitly specified in the system prompt prior to the task, bypassing the conversational persuasion process. This isolates the behavioral effect of the belief state itself from the mechanics of persuasion.

Downstream Tasks

Agents were evaluated on three task types where the persuasive topic was semantically irrelevant to the task:

• Opinion Change Task: A distractor dialogue to test stance persistence.
• Coding Task: Writing Python code to satisfy specifications, measuring iteration volume and revision structure.
• Web Research Task: Conducting open-ended research, requiring search queries, visiting external sources, and synthesizing reports.

Evaluation Metrics

The study moves beyond final output quality to process-level behavioral analysis:

• Coding Metrics: Time-and-Revision Score (TRS) and Edit Volatility Score (EVS) to capture execution time, iteration volume, and revision structure.
• Web Research Metrics: Constructs for Activity (interaction volume), Breadth (source diversity), and Depth (semantic focus), aggregated via PCA.
• Long-Horizon Proxy: Three web research tasks were concatenated to simulate extended execution, allowing observation of compounding behavioral shifts.

Key Results

1. Persuasion Susceptibility and Persistence

Persuasion tactics successfully shifted stated beliefs for some models (e.g., GPT, Mistral), with "Authority Endorsement" and "Evidence-based" tactics showing the highest persistence. However, LLaMA exhibited high baseline susceptibility even without specific tactics, and persuasion did not significantly increase persistence further.

2. On-the-Fly Persuasion Effects

When persuasion occurred in real-time prior to the task:

• Weak Aggregate Shifts: Across coding and web research tasks, the aggregate behavioral shifts between persuaded and non-persuaded agents were weak and statistically insignificant.
• High Heterogeneity: Persona-level effects were highly variable and often canceled each other out in pooled averages. For instance, in web research, some personas narrowed their source diversity while others expanded it, resulting in a net mean change near zero.
• Conclusion: Real-time persuasion induces heterogeneous, persona-dependent shifts that do not yield a consistent directional change in behavior across the board.

3. Prefilled Belief Effects

When belief states were explicitly pre-filled (bypassing the persuasion dialogue):

• Significant Behavioral Drift: Belief-prefilled agents conducted 26.9% fewer searches and visited 16.9% fewer unique sources compared to neutral-prefilled agents.
• Mechanism: The effect was driven by the belief state itself rather than the persuasion process. Agents holding a specific belief (even if task-irrelevant) altered their exploration strategy, becoming more efficient but less diverse in their information gathering.

4. Long-Horizon Implications

In concatenated long-running tasks:

• Persuasion-injected runs showed greater drift in search behavior compared to neutral-context injections.
• While final output quality scores (evaluated by an LLM-as-a-Judge) showed some degradation in coverage and instruction following for persuaded agents, the most significant finding was the process-level drift. The agents' information pipelines were altered, leading to narrower source selection and different query formulations, even when the final report appeared plausible.

Contributions

1. Problem Formulation: Introduced the concept of persuasion propagation, defining how task-irrelevant persuasion can affect downstream execution behavior in agentic systems.
2. Evaluation Framework: Proposed a controlled framework that disentangles belief state from persuasion timing, distinguishing between on-the-fly persuasion and prefilled belief conditioning.
3. Trace-Level Evaluation: Demonstrated that persuasion propagation manifests as behavioral drift (e.g., search breadth, revision volatility) not observable from final outputs alone, motivating the need for trace-based metrics in auditing LLM agents.
4. Empirical Evidence: Provided evidence that while on-the-fly persuasion yields inconsistent aggregate effects, explicitly specified belief states (a proxy for successful persuasion) cause measurable, systematic reductions in information exploration diversity.

Significance and Claims

The paper argues that persuasion propagation represents a distinct reliability concern for agentic systems, separate from standard prompt injection or policy violations.

• Subtle Risks: The risk is not necessarily a catastrophic failure or a policy violation, but a subtle shift in how an agent gathers evidence. An agent may produce a fluent, task-compliant answer that is disproportionately shaped by a limited set of sources due to a prior belief state.
• Evaluation Necessity: The authors claim that evaluating agents solely on final outputs is insufficient. Because persuasion can alter the "information pipeline" (search queries, source selection, stopping criteria) without changing the final answer's surface quality, behavior-level evaluation is required to detect these risks.
• Long-Horizon Vulnerability: The study suggests that in long-running agents, small deviations in exploration or effort allocation can compound over time, making the effects of persuasion harder to detect but potentially more harmful to the reliability and grounding of the agent's output.

The authors remain modest, noting that their long-running setting is a simplified proxy and that the magnitude of observed shifts varies significantly by persona and model. They conclude that while belief susceptibility alone does not guarantee immediate behavioral compromise, persistent belief conditioning can alter downstream behavior, necessitating robust, behavior-centered evaluation frameworks for deployed agents.