💬 NLP

Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Framework

This paper presents a unified, evaluation-driven framework developed at Nubank that integrates structured context engineering, human-in-the-loop iteration, and rigorous LLM judge evaluation to successfully deploy production-ready customer support AI agents across five domains for over 100 million users, achieving significant improvements in satisfaction and self-service rates while demonstrating a strong correlation between offline metrics and online impact.

Original authors: Aman Gupta, Kevin Rossell, Edesio Alcobaça, Jose Chrystian Lima Pacheco, Carolina Baptista de Lima, Shao Tang, Luiz Paulo Rabachini, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Gupta, Kevin Rossell, Edesio Alcobaça, Jose Chrystian Lima Pacheco, Carolina Baptista de Lima, Shao Tang, Luiz Paulo Rabachini, Luis Moneda, Herbert Fei, Daniel Silva, Rohan Ramanath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive bank with 100 million customers. Every day, millions of people call or chat asking questions like, "Where is my credit card?" or "Why do I owe this much?"

In the past, you'd need an army of human agents to answer these. But now, you want to build a super-smart AI robot to handle these conversations. The problem? If the robot gives a wrong answer, a customer gets angry, loses trust, and leaves. You can't just "guess" if the robot is good; you need to be 100% sure before you let it talk to real people.

This paper is a blueprint from Nubank (a giant bank) on how they built these AI robots safely and successfully. Here is how they did it, explained simply:

1. The Problem: Building a Robot Without a Crash Test

Usually, when people build AI, they write instructions, test it a little, and hope for the best. The authors say this is like building a car and driving it off a cliff to see if the airbags work. In customer support, a crash is too expensive.

They needed a way to test the robot thousands of times in a "simulator" before letting it talk to a single real human.

2. The Solution: The "Evaluation-Driven" Factory

Instead of just writing instructions, they built a factory line with four main stations. Think of it like tuning a race car:

Station 1: The Modular Kit (Context Engineering)
Instead of writing one giant, messy instruction manual for the robot, they broke it into Lego blocks.
- The Rules: How the robot should speak (polite, concise).
- The Playbook: Step-by-step guides for specific problems (e.g., "If the card is lost, do step A, then step B").
- The Tools: A list of things the robot can actually do (like checking a database or reissuing a card).
- The Memory: A scratchpad where the robot writes down what it learned during the chat.
- Why it matters: If the robot gets the greeting wrong, you only swap the "Greeting Lego block." You don't have to rebuild the whole car.
Station 2: The Robot Judges (LLM-as-a-Judge)
How do you know if the robot is doing a good job? You can't ask the robot to grade itself. So, they used another AI (a "Judge") to grade the first robot's answers.
- The Catch: Sometimes the Judge AI is biased or lazy. To fix this, they used a special technique called GEPA. Imagine a coach who watches the Judge AI grade a test, realizes the Judge was too harsh, and rewrites the grading rubric to be fairer. They did this automatically until the grading became super consistent.
Station 3: The Human Safety Net (Inter-Rater Reliability)
Before trusting the AI Judge, they had real humans grade the same answers. They checked if the humans agreed with each other. If the humans agreed 90% of the time, they knew the test was fair. Then, they made sure the AI Judge agreed with the humans just as much.
Station 4: The Real-World Test (A/B Testing)
Once the robot passed the simulator tests, they let it talk to a tiny group of real customers (like 1% of users). They compared this robot to the old system. If the new robot made customers happier, they let it talk to more people.

3. The Results: The Robot Wins

They tested this system on five different types of problems:

Card Delivery: "Where is my card?"
Debt Management: "How do I pay off my loan?"
Credit Limits: "Can I get a higher limit?"
Card Management: "Change my PIN."
Product Explainer: "What does this feature do?"

The Magic Numbers:

Happiness: For the "Card Delivery" robot, customer happiness (measured by a score called tNPS) jumped by 37 points. That's a massive improvement.
Self-Service: More people solved their problems without needing a human. The "self-service rate" went up by 29 points.
Human vs. Robot: In four out of five cases, the robot was almost as good as a top-tier human expert (within a few percentage points). The only place it struggled a bit was the very complex "Debt Management" area, which makes sense because that's the hardest math and empathy problem.

4. The Big Lesson: "If you can measure it, you can fix it"

The most important takeaway from the paper is this: The quality of your testing determines how fast you can improve.

Because they built such a rigorous testing system (the "simulator"), they could make changes to the robot's instructions and know immediately if it got better or worse. They didn't have to wait weeks to see if customers were happy. They could iterate (improve) the robot dozens of times in the simulator, and when they finally launched it, it was already a champion.

In short: They didn't just build a smart AI; they built a smart testing lab for the AI. And because the lab was so good, the robot they sent out to the world was incredibly reliable, happy-making, and ready for 100 million users.

Technical Summary: Building Customer Support AI Agents at 100M-User Scale

Problem Statement

The rapid advancement of Large Language Models (LLMs) has made AI agents viable for complex tasks, yet deploying production-ready customer-facing agents at scale remains a significant challenge. In the customer support (CS) domain, specifically within high-stakes environments like banking, agents face unique constraints:

High Reliability Thresholds: A single failure can erode trust and drive churn; financial mishaps can lead to significant losses.
Data Sensitivity: Agents must handle real customer data (balances, transactions) under strict privacy and regulatory constraints.
Specialization vs. Diversity: Agents utilize narrow, domain-specific toolsets (5–15 tools) but must handle substantial edge-case diversity.
Graceful Handoff: Agents must operate within a human-in-the-loop workflow, transferring control with full context preserved when automation fails.

A critical gap identified is that evaluation methodology, context engineering, training, and online measurement are typically developed in isolation. This creates "blind spots" that only surface after deployment, slowing iteration velocity and risking production failures.

Methodology: An Evaluation-Driven Framework

The authors present a unified framework that bridges offline development with online impact, validated on a company serving 100M+ users. The core philosophy is that evaluation-pipeline quality directly determines iteration velocity.

1. Modular Context Engineering

Instead of monolithic prompts, the system employs a semantically versioned, modular architecture where components are independently managed:

Instructions: Role definitions, behavioral constraints, and compliance guardrails.
Routines: Ordered, step-by-step procedures derived from human Standard Operating Procedures (SOPs) (e.g., card delivery investigation flows).
Macros: Pre-written response templates adapted by the agent.
Tool Specifications: Independent descriptions of tools (schema, input/output, invocation conditions) allowing recombination.
Working Memory: A persistent scratchpad tracking conversation state to prevent redundant calls.

2. Rigorous Evaluation Pipeline

The framework utilizes an LLM-as-a-Judge approach, enhanced by specific reliability measures:

Human-in-the-Loop (HITL) Iteration: Domain experts annotate data with binary labels and textual rationales.
GEPA Optimization: The system uses GEPA (within the DSPy framework) to automatically refine judge prompts. A strong model critiques errors and synthesizes revisions based on human feedback, optimizing for consistency.
Inter-Rater Reliability (IRR): The framework enforces minimum agreement thresholds (measured via Cohen's $\kappa$ ) between multiple judge models. Prompts are only trusted if they yield stable scores across different model architectures (e.g., GPT-4.1, GPT-5, o3).
Cross-Model Validation: Ensures that well-specified rubrics yield stable scores regardless of the specific judge model used.

3. Development Lifecycle

The process operates on two nested loops:

Fast Loop: Offline evaluation results (simulation scores) drive immediate prompt fixes. If an evaluator flags a failure, the specific prompt component is surgically modified and re-evaluated.
Slow Loop: Online metrics (from A/B tests) inform architectural changes, such as adding new routines or redesigning tool interfaces.

4. Infrastructure and Deployment

Tool Design: Tools are designed for idempotency and deterministic orchestration (wrapping multi-step logic into single tools) to reduce latency and error.
Progressive Rollout: New agents are deployed at 1–5% traffic with confidence-based escalation to human agents.
Model Portability: The modular architecture allows for rapid adaptation when migrating between foundation models (e.g., GPT-4.1 to GPT-5) by adjusting specific prompt components rather than rewriting the entire system.

Key Contributions

Unified Framework: A closed-loop system connecting agent architecture, prompt versioning, offline evaluation, and online measurement.
Modular Context Engineering: A design pattern allowing independent versioning of instructions, routines, macros, and tools.
Calibrated LLM-as-Judge Pipeline: A rigorous evaluation system featuring inter-rater reliability requirements, GEPA-optimized judge prompts, and cross-model validation.
Prod-Validated Results: Empirical evidence from five distinct production deployments (Card Delivery, Debt Management, Credit Limit, Card Management, Product Explanation).

Results

The framework was validated across five production use cases, with the Card Delivery agent serving as the primary case study.

Card Delivery Deployment (Headline Results)

Customer Satisfaction: Achieved a 37 percentage-point (p.p.) improvement in AI Transactional Net Promoter Score (tNPS) compared to prior variants.
Automation: Achieved a 29 p.p. gain in Self-Service Rate (SSR).
Human Parity: The final AI tNPS landed within 10 p.p. of expert human agents.
Offline-Online Correlation: A strong positive correlation was observed between offline evaluation failure rates and online tNPS/SSR gains. Versions with the largest offline improvements (e.g., V10) delivered the highest online gains.

Cross-Domain Performance

Across all five domains, the framework delivered consistent gains:

Debt Management: +40 p.p. tNPS gain (though a larger gap of -23.6 p.p. remained compared to humans, reflecting the domain's complexity).
Credit Limit: +4.5 p.p. tNPS gain; significant reductions in evaluator failure rates for context and empathy.
Card Management: +38.3 p.p. tNPS gain.
Product Explainer: +12.3 p.p. tNPS gain.

Evaluation Pipeline Efficacy

Prompt Optimization: Automated optimization (GEPA) consistently outperformed manual prompt engineering and majority-class baselines. For example, in Eval E3, accuracy rose from 58.07% (manual) to 73.01% (optimized).
Inter-Rater Agreement: Unoptimized "starter" prompts resulted in negligible agreement between models ( $\kappa \approx 0.00$ to $0.10$). GEPA-optimized prompts increased agreement significantly (e.g., $\kappa = 0.745$ between GPT-4.1 and GPT-4.1-mini; $\kappa = 0.950$ between reasoning-intensive models).

Significance and Claims

The paper claims that evaluation quality is the primary determinant of iteration velocity. By establishing a reliable "fast loop" through calibrated judges and rigorous inter-rater agreement, teams can iterate confidently offline without waiting for sparse online signals.

The authors assert that their framework successfully bridges the gap between offline simulation and online reality, demonstrating that eval-driven development reliably predicts production impact. The methodology proves generalizable across diverse customer support domains, enabling organizations to scale AI agents to 100M+ users while maintaining high quality, compliance, and customer satisfaction. The work emphasizes that while AI agents can approach human-level performance in many domains, the hardest tasks (like debt management) still require careful human oversight and iterative refinement.