💻 computer science

Email in the Era of LLMs

This paper introduces the HR Simulator, a communication game demonstrating that while large language models (LLMs) often outperform humans in formal email scenarios and exhibit more homogenous, tactful preferences, human-LLM collaboration significantly enhances success rates, revealing both the efficacy of LLMs in structured communication and current limitations in mimicking human-like low-empathy, informal styles.

Original authors: Dang Nguyen, Harvey Yiyun Fu, Peter West, Chenhao Tan, Ari Holtzman

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dang Nguyen, Harvey Yiyun Fu, Peter West, Chenhao Tan, Ari Holtzman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're playing a high-stakes video game called "HR Simulator." In this game, you aren't fighting dragons or racing cars; you are the new Human Resources officer at a fictional tech company called "NeuroGrid." Your only weapon? Email.

Every level of the game presents a tricky social situation: a new employee wants a private office they aren't eligible for, two coworkers are fighting over a project, or a senior engineer is secretly unhappy but won't say why. Your goal is to write an email that solves the problem without starting a war, making someone cry, or getting yourself fired.

The twist? In this future world, your emails aren't just read by humans. They are read, judged, and sometimes even written by AI (Large Language Models).

Here is what the researchers discovered by playing this game with 600+ emails:

1. The AI Judges are Getting "Groupthinky"

Imagine a panel of judges tasting a soup.

Small AI models are like a group of friends who all have different opinions. One says it needs salt, another says it needs pepper. They can't agree on what "good" tastes like.
Big, powerful AI models are like a group of food critics who have read the same cookbook. They all agree: "Good soup must be perfectly seasoned and warm."

As AI gets smarter, they all start agreeing on what a "good email" looks like. They are converging on a single, very specific style. The problem? That style isn't always what humans prefer.

2. Humans vs. AI: The "Polite Robot" Problem

When humans wrote emails alone, they were often too blunt, too casual, or just "off."
When AI wrote emails alone, they were too perfect. They were like a robot trying to be a human: overly formal, excessively empathetic, and sometimes a bit robotic in their kindness.

The Result: When an AI judge graded the emails, the AI-written ones usually won. Humans lost. It's like a human trying to sing a song against a perfectly tuned synthesizer; the machine hits every note, but the human has "soul" that the machine misses.

3. The Magic Combo: Human + AI (The "Cyborg" Strategy)

Here is the best news: The best emails were written by a Human and an AI working together.

Think of it like this:

The Human is the Architect. They know the goal: "I need to tell Sam he can't have a private office, but I need to keep him excited about the job." They know the nuance of the situation.
The AI is the Interior Designer. They take the human's rough draft and polish it. They add the right amount of "warmth," fix the grammar, and make sure the tone sounds professional.

When you combine them, the success rate skyrocketed. In some difficult scenarios, a human alone had a 40% chance of winning. An AI alone had a 50% chance. But Human + AI together had nearly a 100% chance.

4. The "Tact" Discovery: Bigger AI = More Subtle

The researchers found something fascinating called "Emergent Tact."

Smaller AI models were like blunt instruments. They preferred emails that were direct, maybe a bit aggressive, or tried to "buy" the other person's agreement with perks (like "You can't have an office, but here's a noise-canceling headset!").
Larger, smarter AI models were like master diplomats. They preferred subtlety. They knew that saying "You can't have an office" is bad, but framing it as "We value collaboration in our open space" is good.

As AI gets smarter, it learns that being tactful is the key to winning.

5. The One Thing AI Can't Do: The "Grumpy Text"

There is one corner of the email world where AI fails miserably: The "Low Empathy, Low Formality" zone.

Imagine you are so frustrated with a company's bureaucracy that you want to write a short, angry, rude email just to get their attention.

Humans can do this easily. We can be curt, dismissive, and angry when we need to be.
AI cannot. Even when asked to be rude, the AI keeps trying to be polite and helpful. It's like asking a golden retriever to bite someone; it just wags its tail and offers a treat.

This is important because sometimes, in real life, you need to be blunt and unemotional to get a point across. AI can't do that alone. This is another reason why we need humans in the loop—to provide that "grumpy" edge when necessary.

The Big Picture: What Does This Mean for Us?

The paper suggests a future where:

If you try to compete with AI alone, you will likely lose. AI is getting too good at the "standard" rules of polite communication.
If you team up with AI, you become unstoppable. You bring the human intuition and the "grumpiness" when needed; the AI brings the polish and the diplomatic finesse.

The Verdict: The future of email isn't "Human vs. Machine." It's "Human + Machine." We shouldn't try to out-write the robots; we should let them be our editors while we remain the architects of our messages.

1. Problem Statement

As Large Language Models (LLMs) increasingly integrate into workplace communication tools (e.g., Gmail's Gemini, personal assistants), there is a critical lack of intuition regarding how these models will read, write, and optimize for nuanced social goals. Unlike objective tasks like math, email communication involves subtle social dynamics, tone, and context-dependent strategies (e.g., tact, empathy, formality). The authors investigate three core questions:

How do LLMs perform in complex, socially charged email scenarios compared to humans?
Do LLMs converge on a specific "quality" standard as they scale, potentially diverging from human judgment?
Can a hybrid approach (Human + LLM) outperform both human-only and LLM-only writing?

2. Methodology: HR Simulator™

To address these questions, the authors introduced HR Simulator™, a game-based evaluation framework where players act as Human Resources officers at a fictional company ("NeuroGrid").

Task Structure: Players navigate five distinct, socially challenging scenarios (e.g., declining a private office request, resolving team conflicts, enforcing return-to-office policies, eliciting introspection from a laid-off employee).
Simulation Pipeline:
- Player Input: The player writes an email response to a scenario prompt.
- Recipient Simulation: An LLM (acting as the recipient, e.g., "Sam" or "Adam") responds in-character based on the player's email.
- Outcome Simulation: For complex scenarios, a "Game Master" LLM simulates the future consequences of the interaction.
- Judgment: A Judge LLM (primarily GPT-4o, but also others) evaluates whether the email achieved the communication goal based on the context, the recipient's response, and the simulated outcome.
Data Collection: The study analyzed over 600 emails generated by:
- Humans: Human players.
- LLMs: Various models ranging from smaller (e.g., Gemini 2.5 Flash) to state-of-the-art (e.g., Kimi K2, Qwen 3).
- Hybrid: Human drafts rewritten by LLMs.
Evaluation Metrics:
- Pass Rate: The percentage of emails that successfully solve the scenario.
- ELO Ranking: Pairwise comparisons of emails by different judges to derive relative quality scores.
- Tone Analysis: Annotation of emails on Empathy (1–7 scale) and Formality (1–7 scale).
- Tactfulness: A context-specific metric measuring the ability to navigate social situations without causing future pitfalls (e.g., avoiding passive-aggressive comparisons of salary).

3. Key Contributions

HR Simulator™ Framework: A novel, scalable instrument for measuring human-LLM communication in high-stakes, socially nuanced contexts.
Emergent Tact: The discovery that as LLMs scale, their preference for "tactful" communication increases. Smaller models prefer direct, less tactful strategies, while larger models prefer subtle, highly tactful approaches.
Homogenization of Judgment: Evidence that larger LLM judges become increasingly homogeneous in their quality assessments (reaching ~0.5 Krippendorff's $\alpha$ agreement), converging on a standard that favors LLM-written emails over human ones.
Hybrid Advantage: Demonstration that a "Human + LLM" approach often outperforms both human-only and LLM-only writing, particularly when LLMs rewrite human drafts to align with preferred social norms.
Tone Limitations: Identification of a specific "low empathy, low formality" quadrant where current LLMs struggle to generate or imitate human emails, highlighting a gap in current post-training paradigms.

4. Key Results

A. Performance and the Hybrid Advantage

Human vs. LLM: Under LLM judges, humans significantly underperform LLMs. The average human success rate was 23.5%, whereas top-tier LLMs (Qwen 3, Kimi K2) achieved 48–54%.
Human + LLM: Rewriting human emails with LLMs creates a "hybrid advantage." In several scenarios (e.g., Scenario 1), rewriting increased the pass rate of mid-tier models (GPT-4o, Claude 3.5 Haiku) from ~40% to nearly 100%. This suggests that while humans struggle to write optimally for LLM judges, they possess the intent that, when refined by an LLM, yields superior results.

B. Emergent Tact and Judge Agreement

Scaling Effect: As model size increases, pairwise agreement on email quality increases.
Tact Preference: There is a clear divergence in "tact" preferences based on model capability:
- Smaller Models: Prefer direct, less tactful emails (average tact score ~3.5).
- Larger Models: Prefer subtle, highly tactful emails (average tact score ~4.5+).
- Example: In a scenario declining a private office, smaller models preferred mentioning the employee's higher salary as a "perk" to soften the blow. Larger models rejected this as "passive-aggressive" and preferred focusing on policy and collaboration.

C. Tone Analysis (Empathy vs. Formality)

Distribution: LLM emails cluster heavily in the High Empathy / High Formality quadrant. Human emails are more diverse, occupying low-empathy/low-formality quadrants (often due to frustration or directness).
Rewriting Effects: LLM rewrites consistently shift human emails toward the High Empathy/High Formality quadrant, which correlates with higher pass rates.
The "Low-Low" Gap: LLMs struggle to generate or imitate emails in the Low Empathy / Low Formality quadrant (e.g., curt, frustrated, or aggressively direct messages). Even with few-shot examples, models exhibit high perplexity on these human styles. This suggests a limitation in current post-training: models are optimized for "polite" corporate communication and struggle with "thrust" or necessary bluntness.

5. Significance and Implications

Future Workplace Dynamics: The results suggest that in a future where emails are read and optimized by LLMs, unaided human writers will likely be at a disadvantage due to misalignment with LLM-defined "quality" (homogenized, highly tactful, formal).
The Rise of Co-Writing: The most effective communication strategy will likely be Human-LLM co-writing, where humans provide the core intent and social nuance, and LLMs refine the tone to meet the expectations of the recipient (whether human or AI).
Alignment Risks: The convergence of large models on a specific, highly tactful, and formal value system may create a "fixed landscape" of communication that differs from human intuition, potentially suppressing diverse communication styles (e.g., necessary bluntness).
Training Directions: The difficulty models face in the "low empathy, low formality" quadrant indicates a need for new post-training objectives that allow autonomous agents to communicate with "thrust" when the situation demands it, rather than defaulting to excessive politeness.

In conclusion, the paper posits that while LLMs are becoming the dominant arbiters of communication quality, the optimal future workflow is not full automation, but a symbiotic partnership where humans and LLMs collaborate to navigate the complex, evolving landscape of social interaction.