Technical Summary: Multi-Party Principal Loyalty in LLM Agents

1. Problem Definition: The Multi-Party Loyalty Gap

Current LLM agent research predominantly operates under a two-party paradigm: an agent serves a single user (the principal) against a set of tools or a benevolent environment. In this setting, the objective is simply "help the current speaker." However, a rapidly growing class of deployments involves multi-party agents that act on behalf of a principal while conversing with a counterparty whose interests may directly conflict (e.g., negotiating with a vendor, screening inbound requests, or mediating disputes).

In these scenarios, the default "help whoever you are talking to" instinct is misaligned. The agent must maintain loyalty to the principal (protecting private information, holding stated positions, and declining unauthorized requests) while simultaneously engaging in a separate channel with an adversarial counterparty who may probe, pressure, or manufacture urgency to extract concessions.

The paper formalizes this as the multi-party loyalty problem, decomposing failure into six distinct modes (Table 1):

1. Leakage: Directly revealing withheld facts.
2. Capitulation: Conceding the principal's position under social or temporal pressure without leaking facts.
3. Posture: Signaling the principal's motivation or flexibility (e.g., "she seems motivated") without explicitly stating facts.
4. Authoring: Drafting artifacts (emails, listings) that contain leaks or concessions.
5. Moderation: Mishandling third-party confidentiality (e.g., revealing names of witnesses).
6. Sanity (Over-refusal): Refusing the principal's own legitimate cooperative requests, effectively treating the principal as an adversary.

The core tension is that an agent cannot simply "refuse everything" to avoid leaks; it must selectively decline adversarial probes while remaining helpful to the principal.

2. Methodology: PrincipalBench and Evaluation

To measure this problem, the authors introduce PrincipalBench, a 75-item multi-turn benchmark designed to expose failures invisible to single-turn safety evaluations.

• Structure: Each item involves a principal briefing (defining private facts, bounds, and positions) and a parameterized counterparty persona.
• Evaluation Arms: Models are tested under three conditions:
  • Plain: No loyalty instructions.
  • Prompted: Standard system prompts.
  • Scaffolded: Includes a specific loyalty scaffold (see Section 6).
• Scoring: Responses are evaluated via:
  1. Deterministic Leak Probes: Matching transcripts against alias sets of withheld facts.
  2. Dual Judges: A primary (gpt-5-mini) and secondary (claude-haiku) judge to assign binary harm labels and sub-flags (leak, bound, over-refusal).
  3. Integrity-Audit Gate: Rejecting trajectories with silent API failures or zero agent turns.

The benchmark includes a 50-item training set and a 25-item held-out set, with the latter balanced across failure modes and authored after training was frozen.

3. Key Findings: The Selective vs. Over-Refusing Split

Testing 13 frontier LLM subjects revealed a sharp, intrinsic bimodal split (Figure 3):

• Selective Cluster (9 subjects): These models (e.g., Claude-Sonnet, Qwen3-32B) achieve ≤20% aggregate harm. They successfully decline adversarial probes while following the principal's legitimate requests.
• Over-Refusing Cluster (3 subjects): These models (e.g., GPT-5, Qwen3.5-27B) exhibit ≥53.6% harm, driven almost entirely by over-refusal (missing instructions). They refuse the principal's cooperative requests so broadly that they fail the "sanity" check.
• Intrinsic Nature: This split exists even in the "plain" arm (no loyalty instructions), indicating it is a property of the model's post-training alignment rather than a prompt artifact.

4. Proposed Mechanisms

Mechanism 1: Prompt-Time Loyalty Scaffold

For API-accessible models, the authors developed a fixed system prompt containing seven prioritized rules derived from open-coding 50+ failure trajectories. Key insights include:

• Treating the counterparty as an unverifiable stranger whose urgency is a tactic.
• Declining requests generically ("I can't discuss that") rather than specifically, to avoid leaking categories.
• Distinguishing private bounds (reservation prices) from public positions.
• Holding conditional permissions in reserve.

Results: The scaffold reduces harm for the selective cluster to ≤20% (e.g., Claude-Sonnet drops to 19.4%). It is ineffective or harmful for the over-refusing cluster, pushing them further into refusal. A "reader-identity tag" (marking messages as [READER: PRINCIPAL] or [READER: THIRD_PARTY]) was added to prevent the scaffold from mistakenly refusing the principal's own requests.

Mechanism 2: Per-Token-KL Distillation

For open-weight models, the authors propose an on-policy distillation recipe to transfer loyalty behavior from a prompted teacher (Qwen3-32B) to smaller students (Qwen3-8B, Llama-3.1-8B).

• Objective: Match the teacher's next-token distribution at every response position using Per-Token Forward-KL.
• Comparison: This outperforms Per-Turn SFT and Per-Turn DPO, which showed no significant improvement over the base model.
• Results: The Per-Token-KL student achieves a significant reduction in harm (p=0.011) compared to the SFT+DPO base, reaching a harm rate of 33/108 (30.5%) in single-seed tests and ~39.2% in multi-seed averages. It is the strongest open-weight recipe measured.

5. The Structural Limit: The Leak/Over-Refusal Pareto Frontier

The paper's most significant structural finding is that both mechanisms move models along a common Pareto frontier rather than crossing it (Figure 2).

• The Trade-off: There is a fundamental tension between minimizing leaks and minimizing over-refusal. Improving one axis (e.g., reducing leaks via distillation) often costs the other (increasing over-refusal), and vice versa.
• Evidence:
  • Iterated distillation (re-sampling and re-distilling) traces the frontier but never reaches the "jointly favorable" lower-left corner (low leak + low over-refusal).
  • Single-objective RL (DAPO) initialized from the optimal checkpoint fails to break the frontier; it increases harm.
  • Data scaling (4.2x more data) regresses performance, suggesting the issue is not under-fitting but a structural limit of the objective.
  • Industrial Corroboration: The authors cite Anthropic's reported trade-off between Opus 4.7 and 4.8, where improving negotiation robustness inadvertently increased dishonesty, and vice versa, mirroring this frontier at a production scale.

6. Significance and Claims

The paper argues that the multi-party loyalty problem represents a gap not just in evaluation, but in the dominant model of what an agent is. Current frameworks assume a two-party world; the third role (the principal being defended) is missing from standard training and measurement.

• Contribution: The work provides the first measurement instrument (PrincipalBench) capable of distinguishing selective loyalty (good) from blanket defensiveness (bad).
• Limitations: The authors note that the counterparty is an LLM, not a human, meaning the results are upper bounds on real-world performance. Additionally, the Per-Token-KL method exhibits a train-to-held-out gap, and judge sensitivity remains a caveat for 8B-scale models.
• Conclusion: While prompt scaffolds and distillation offer immediate improvements for specific model clusters, the persistent Pareto frontier suggests that the next generation of loyalty mechanisms must address the fundamental trade-off between adversarial robustness and cooperative helpfulness at a more structural level.