Act or Escalate? Evaluating Escalation Behavior in Automation with Language Models

This paper evaluates how language models decide between acting and escalating in uncertain scenarios across five domains, revealing that their implicit escalation thresholds are model-specific and miscalibrated, but can be robustly aligned through supervised fine-tuning on chain-of-thought targets that explicitly reason about uncertainty and decision costs.

The Gist

Imagine you hire a very smart, but sometimes overconfident, robot assistant to make decisions for your business. You tell it: "If you're sure, go ahead and do it. If you're not sure, ask me for help."

This paper is about figuring out when that robot actually decides to ask for help, and why some robots are too reckless while others are too timid.

Here is the breakdown of the research using simple analogies:

1. The Core Problem: The "Go or Ask?" Dilemma

Think of the robot as a Junior Chef and you as the Head Chef.

• Acting: The Junior Chef plates a dish and serves it to the customer. If it tastes bad, the customer is angry (this is the Error Cost).
• Escalating: The Junior Chef asks the Head Chef, "Should I serve this?" The Head Chef tastes it and decides. This takes time and interrupts the Head Chef's work (this is the Labor Cost).

The goal is to find the perfect balance: Don't ask the Head Chef for every single salt shaker (too slow), but don't serve a burnt steak just because you're too shy to ask (too risky).

2. The Big Discovery: Robots Have Hidden "Personality Types"

The researchers tested eight different AI models (like Qwen, GPT, Llama, and Mixtral) across five different jobs (like predicting loan approvals, spotting toxic comments, or guessing movie ratings).

They found that every robot has a hidden "bravery threshold" that is totally different from the others, and you can't guess it just by looking at how big or fancy the robot is.

• The Reckless Robot: Some models (like GPT-5-mini) are like a gambling driver. They will speed down the highway even when the road is foggy. They think, "I'm probably right, so I'll just go for it," even when they are only 54% sure. They rarely ask for help.
• The Timid Robot: Other models (like Llama 4 Maverick) are like a paranoid passenger. They will stop the car and ask for directions even when the GPS says they are 95% sure of the route. They escalate (ask for help) way too often.
• The Size Myth: You might think a bigger, more expensive robot would be smarter and more balanced. Not true. Sometimes the bigger version of the same robot family is more reckless, and sometimes it's more timid. There is no pattern.

3. The Confidence Trap: "I Know What I'm Doing!"

The researchers also asked the robots: "How sure are you that you are right?"

• The Overconfident: Some robots consistently say, "I'm 90% sure!" when they are actually only 70% right. They are like a student who studies for 10 minutes but claims they know the whole textbook.
• The Underconfident: Others say, "I'm only 50% sure," when they are actually 80% right. They are like a genius student who thinks they failed the test because they forgot one tiny detail.

The scary part: Being overconfident or underconfident doesn't always match how they act. A robot can be overconfident but still too scared to act, or underconfident but still too eager to act. It's a chaotic mix.

4. The Solutions: How to Fix the Robots

The researchers tried three ways to fix this behavior:

• Just Telling Them (Prompting): You tell the robot, "Hey, if you get it wrong, it costs $100. If you ask me, it only costs $10."
  • Result: This barely worked on its own. The robot heard the numbers but didn't really "think" about them.
• Forcing Them to Think (Chain-of-Thought): You tell the robot, "Stop and think step-by-step about the costs before you decide."
  • Result: This helped, but only if you also told them the costs. It's like giving a calculator to a person who doesn't know how to use it; they need both the tool and the instructions.
• Training Them (Fine-Tuning): This was the magic bullet. The researchers took a robot and taught it specifically how to calculate the risk: "If accuracy is X and cost is Y, then I should escalate."
  • Result: The robot became perfect. It learned the logic of the decision, not just the answer. It could handle new situations it had never seen before, like a student who learns the formula instead of memorizing the answers.

The Bottom Line

If you are building an AI system to make important decisions (like approving loans or driving cars), you cannot assume the AI knows when to ask for help.

1. Test first: You have to run experiments to see if your specific robot is a "reckless gambler" or a "timid worrier."
2. Don't guess: Bigger models aren't automatically better at this.
3. Train for the job: If you want the robot to make the right choice, you have to train it to explicitly think about the costs of being wrong versus the cost of asking for help.

In short: Don't just buy a robot and hope it behaves. Teach it the rules of the game first.

Technical Summary

1. Problem Statement

As Large Language Model (LLM) agents are increasingly deployed for consequential automated decisions (e.g., loan approvals, content moderation, autonomous driving), a critical gap exists in how these agents decide when to act versus when to escalate to a human.

• The Trade-off: An agent must balance the cost of implementing an incorrect decision ($c_w$) against the cost of escalating to a human for review ($c_\ell$).
• The Gap: Most evaluations focus on raw accuracy or speed, ignoring the "escalation decision." An agent that is overconfident will implement errors at scale, while an underconfident agent will escalate valid decisions, negating the efficiency gains of automation.
• Core Question: Do LLMs possess calibrated internal beliefs about their own accuracy, and do they implicitly adopt decision thresholds that align with the economic costs of errors vs. escalation?

2. Methodology

Theoretical Framework

The authors model the escalation decision as a binary choice under uncertainty:

1. Input: An agent receives a scenario $x$ and produces a prediction $\hat{y}$.
2. Internal State: The agent estimates its probability of being correct, $\hat{p}$.
3. Decision Rule: The agent implements if $\hat{p} \geq \tau$ and escalates if $\hat{p} < \tau$.
4. Optimal Threshold: Theoretically, the optimal threshold $\tau^*$ that minimizes expected cost is derived as:
   $$\tau^* = 1 - \frac{c_\ell}{c_w}$$
   Where $c_\ell$ is the labor cost of escalation and $c_w$ is the error cost of a wrong implementation.
5. Miscalibration Impact: If an agent has a systematic bias $\mu$ (where $\hat{p} = p + \mu$), the effective threshold shifts, leading to suboptimal costs.

Experimental Design

• Models: Eight models across four families (Qwen3.5, GPT-5, Llama 4/3.3, Mixtral/Mistral), comparing smaller and larger variants.
• Datasets: Five domains derived from human decision data:
  1. Demand Forecasting (Hotel Bookings)
  2. Loan Approval (LendingClub)
  3. Content Moderation (Wikipedia Toxicity)
  4. Content Recommendation (MovieLens)
  5. Moral Dilemmas (Moral Machine - used as a robustness check)
• Protocol: A two-turn prompting structure:
  • Turn 1: Agent predicts the outcome based on a scenario and an external signal (a decision tree summary providing ground-truth accuracy for that specific feature set).
  • Turn 2: Agent decides whether to Implement (DECISION: 0) or Escalate (DECISION: 1).
• Interventions Tested:
  • Baseline: Signal provided, no cost framing.
  • Cost Framing: Explicitly stating the cost ratio (e.g., "Wrong answer costs 4x more than escalating").
  • Thinking: Enabling extended reasoning modes (Chain-of-Thought).
  • Supervised Fine-Tuning (SFT): Training models on chain-of-thought targets that explicitly calculate expected costs.

3. Key Contributions & Results

A. Latent, Model-Specific Escalation Thresholds

The study reveals that LLMs have distinct, latent "implicit thresholds" ($p^*$) for escalation that are not predictable by model architecture or scale.

• Variation: Thresholds vary wildly even within the same model family.
  • Example: Qwen3.5-9B has a low threshold ($p^* \approx 56\%$, acts aggressively), while Qwen3.5-397B has a high threshold ($p^* \approx 81\%$, acts cautiously).
  • Example: GPT-5-nano ($p^* \approx 91\%$) vs. GPT-5-mini ($p^* \approx 53\%$).
• Implication: Scaling up does not consistently shift behavior toward optimality; practitioners must empirically characterize the threshold for every specific model deployment.

B. Systematic Miscalibration

LLMs are generally miscalibrated regarding their self-estimated accuracy ($\hat{a}$):

• Directional Bias: Some models are systematically overconfident (e.g., Qwen3.5-9B, Mixtral 8x7B), while others are underconfident (e.g., Llama 3.3 70B).
• Decoupling: Self-estimated accuracy does not predict the escalation threshold. A model can be overconfident yet cautious, or underconfident yet aggressive.
• Consequence: Overconfident models implement errors they should have escalated; underconfident models escalate decisions they could have handled.

C. Effectiveness of Interventions

The paper tests methods to align agent behavior with the optimal threshold ($\tau^*$):

1. Prompting (Cost Framing + Thinking):
   • Cost framing alone had minimal effect.
   • Extended thinking alone improved prediction accuracy but did not fix escalation behavior (agents still acted too aggressively).
   • Synergy: Combining Cost Framing with Extended Thinking yielded significant improvements (e.g., GPT-5-mini accuracy rose from 64.7% to 87.1% in sample-level decision accuracy). This suggests reasoning models need explicit cognitive space to process cost structures.
2. Supervised Fine-Tuning (SFT):
   • Most Robust Solution: SFT on chain-of-thought targets (where the model explicitly calculates: "Signal accuracy is X%, error rate is Y%, expected cost is Z") achieved near-perfect accuracy (100%).
   • Generalization: The SFT model generalized perfectly to held-out domains (MovieLens) and varying cost ratios, proving it learned a general decision procedure rather than memorizing data.

4. Significance and Implications

• Deployment Safety: The findings argue that "black box" deployment of LLM agents for high-stakes decisions is risky. Organizations must empirically measure a model's implicit escalation threshold and calibration before deployment.
• Alignment Strategy: Robust alignment for automation requires training models to reason explicitly about uncertainty and decision costs, not just task accuracy.
• Methodological Advance: The paper introduces a framework using external signals to decouple an agent's prediction from its escalation decision, allowing for precise measurement of latent behavioral dynamics.
• Future Work: The authors suggest future research should explore more complex action spaces, uncertain cost structures, and broader model surveys.

Conclusion

The paper establishes that escalation behavior is a model-specific property that cannot be assumed to improve with scale. While prompt engineering (specifically combining cost framing with reasoning) offers improvements, Supervised Fine-Tuning with Chain-of-Thought is the most effective method for creating agents that reliably align with human-defined cost structures, ensuring automation is both efficient and safe.