Technical Summary: Measuring Agents in Production (MAP)

Problem Statement

Despite the rapid proliferation of Large Language Model (LLM)-based agents in research and the growing excitement around their potential in industries like finance, healthcare, and education, there is a significant lack of empirical understanding regarding what technical methods actually enable successful production deployments. While research literature often focuses on maximizing autonomy and benchmark performance, industry reports suggest that many agent projects fail or underdeliver. The field lacks a systematic, first-hand account of how organizations build, evaluate, and maintain agents in real-world environments, creating a gap between research narratives and engineering realities.

Methodology

To address this knowledge gap, the authors present MAP (Measuring Agents in Production), the first large-scale systematic study of AI agents in production. The study employs a mixed-methods approach combining qualitative depth with quantitative breadth:

• In-depth Case Studies: The authors conducted 20 semi-structured interviews (30–90 minutes) with deployment teams ranging from major technology companies to agent startups. These cases span five sectors, various organizational maturities (startups to global enterprises), and deployment scales from hundreds to millions of users. The study utilized snowball sampling to ensure diversity in domains and geographic reach.
• Large-Scale Survey: A 47-question survey was distributed to practitioners building agent systems across 26 domains. The study received 306 valid responses. To focus on production realities, the authors filtered the data to 86 systems explicitly in production or pilot phases, serving hundreds to millions of daily users.
• Data Analysis: Qualitative data from interviews was analyzed using grounded theory (open and focused coding) by multiple researchers to identify recurring themes. Quantitative survey data was processed with statistical rigor, including 95% confidence intervals computed via 1,000 bootstrap samples. Domain classification for free-text responses utilized semantic aggregation (LOTUS) followed by human annotation.

Key Contributions

The paper makes three primary contributions to the field:

1. First Large-Scale Empirical Study: It provides the first systematic dataset on production agents, combining 20 deep-dive case studies with survey data from 86 deployed systems, moving beyond anecdotal evidence or purely theoretical surveys.
2. Characterization Across 17 Design Dimensions: The study offers quantitative and qualitative data on critical technical decisions, including model selection, architecture, prompting strategies, evaluation methods, application domains, and operational constraints.
3. Data-Driven Insights on Reliability: It challenges common research assumptions by demonstrating that practitioners achieve reliability through system-level design choices (e.g., bounded autonomy, human-in-the-loop) rather than solely through algorithmic or model-level advances.

Key Results

The study addresses four research questions (RQs), yielding the following findings:

RQ1: Applications of Agents

• Motivation: The primary driver for adoption is productivity gains (80% of respondents), specifically reducing human task hours. Harder-to-quantify benefits like risk mitigation are less common.
• Domains: While Finance, Technology, and Corporate Services lead adoption, agents are being deployed across 26 diverse domains, extending well beyond traditional software engineering tasks.
• Users: 92.5% of deployed agents serve human users (internal employees or external customers) rather than other software systems. This human-centric design allows for direct oversight.
• Latency: Production agents tolerate surprisingly relaxed latency; 66% allow response times of minutes or longer. Asynchronous workflows are common, as agents often outperform human baselines even with significant delays.

RQ2: Models, Architectures, and Techniques

• Model Selection: Deployed agents rely heavily on proprietary frontier models (17/20 case studies). Open-source models are used primarily for cost or regulatory constraints.
• Tuning: Teams overwhelmingly favor prompting over weight tuning. 70% of cases use off-the-shelf models without supervised fine-tuning (SFT) or reinforcement learning (RL). Post-training is viewed as brittle to model upgrades and costly to maintain.
• Prompting: Humans remain central to prompt construction (79% manual or manual+LLM). Automated prompt optimization is rare (9%). Prompt lengths vary widely, with some exceeding 10,000 tokens for complex guardrails.
• Architecture: Structured workflows dominate (80% of cases). Agents typically execute fewer than 10 steps before requiring human intervention (68%). Full autonomy is deliberately constrained to ensure reliability.
• Frameworks: Production teams prefer custom in-house implementations (85% of case studies) over third-party frameworks (e.g., LangChain) to ensure flexibility, simplicity, and control over proprietary infrastructure.

RQ3: Evaluation Methods

• Lack of Standardization: 75% of teams evaluate without formal benchmarks, relying instead on A/B testing, user feedback, and production monitoring.
• Human-in-the-Loop: 74% of deployed agents rely primarily on human-in-the-loop evaluation. Automated methods like "LLM-as-a-judge" (52%) serve as complementary verification but rarely replace human review.
• Baselines: Only 39% of teams explicitly compare agents against non-agentic baselines, often because the alternative is a complex hybrid of human and software processes.

RQ4: Top Challenges

• Reliability: Reliability (consistent correct behavior) is the top development challenge. Teams mitigate this through strict environmental constraints (e.g., read-only modes, sandboxing) and bounded autonomy.
• Evaluation: The scarcity of formal benchmarks and the difficulty of verifying real-world tasks (where feedback is delayed) hinder automated evaluation.
• Security: Security is often addressed implicitly through system-level design (e.g., read-only access, contractual agreements with model providers) rather than as a primary development bottleneck, though it remains a concern as deployments expand.

Significance and Claims

The paper claims that MAP provides the research community with visibility into deployment realities, connecting theoretical advances with practical constraints. The authors argue that the "underlying principle" of successful production agents is reliability through system-level design, not model-level optimization.

Key takeaways for the research community include:

• Divergence from Research: Deployment techniques (simple, controllable, human-supervised) diverge from popular research methods (high autonomy, complex RL, automated optimization).
• Research Opportunities: The study identifies underexplored avenues, such as sample-efficient post-training that survives model updates, inference-time scaling for latency-tolerant agents, and multi-model orchestration.
• Human-in-the-Loop as a Design Principle: Rather than viewing human oversight as a temporary constraint to be overcome, the data suggests it is a deliberate, enduring architectural component for ensuring correctness in complex domains.

The authors position MAP not as a comprehensive census of all global agent development, but as an initiative to document the "leading edge" of production practices, offering empirical evidence to guide future research toward delivering real-world value.