🤖 AI

How can we assess human-agent interactions? Case studies in software agent design

This paper introduces PULSE, a human-centric evaluation framework that combines user feedback with machine learning predictions to assess LLM agent designs in real-world software engineering, demonstrating that such collaborative assessments yield more robust insights and reveal significant discrepancies compared to traditional benchmark-driven evaluations.

Original authors: Valerie Chen, Rohit Malhotra, Xingyao Wang, Juan Michelini, Xuhui Zhou, Aditya Bharat Soni, Hoang H. Tran, Calvin Smith, Ameet Talwalkar, Graham Neubig

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Valerie Chen, Rohit Malhotra, Xingyao Wang, Juan Michelini, Xuhui Zhou, Aditya Bharat Soni, Hoang H. Tran, Calvin Smith, Ameet Talwalkar, Graham Neubig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot assistant to help people write computer code. For a long time, the only way to see if your robot was "good" was to give it a standardized test, like a multiple-choice exam. If the robot got a high score, developers assumed it was a star employee.

But the authors of this paper argue that this is like judging a chef only by how well they can follow a recipe book, without ever seeing them cook for a real, hungry customer. In the real world, humans and AI agents work together as a team. Sometimes the human has to step in, correct the robot, or change the plan. A robot that gets a perfect test score might still be frustrating to work with, while a robot that gets a slightly lower score might be a joy to collaborate with.

This paper introduces a new way to measure how well these human-robot teams actually get along. They call their new system PULSE.

The Problem: The "Exam" vs. The "Real Job"

Think of current AI benchmarks as a driving test on an empty track. The car (the AI) has to stay in its lane and stop at red lights perfectly. If it does, it passes.

But real life is more like driving in rush hour traffic with a passenger. The passenger (the human) might say, "Hey, take a different route," or "Wait, I need to stop at the grocery store first." The car has to listen, adapt, and keep the passenger happy. The old "empty track" tests don't measure if the car is annoying to the passenger or if the passenger feels safe.

The Solution: PULSE (The "Feedback Loop")

The researchers built a framework called PULSE to measure this real-world teamwork. They tested it using a software agent (a robot coder) called OpenHands that 15,000 real people used to write code.

Here is how PULSE works, using a simple three-step recipe:

1. The "Ride-Share" Rating (Data Collection)
Instead of waiting until the very end of a project to ask, "How was the robot?", they asked for feedback after every small task, just like you rate a Uber driver after a single ride.

How it worked: When the robot finished a small chunk of work (like fixing a bug), a pop-up appeared asking the user to give it 1 to 5 stars.
The Catch: Most people are lazy and don't want to stop and rate the robot every time. So, they only got ratings for about 5% of the tasks. The other 95% were "silent" interactions where the user just kept working without saying a word.

2. The "Mind Reader" (Prediction Model)
Since they only had ratings for 5% of the tasks, they needed a way to guess how the user felt about the other 95%.

They trained a "Mind Reader" machine learning model. This model looked at clues from the conversation, such as:
- User Mood: Was the user angry or happy in their messages?
- Robot Mistakes: Did the robot misunderstand the instructions?
- Progress: Did the user actually save the code they wrote?
Using these clues, the model predicted what the user would have rated the silent tasks.

3. The "Super-Statistician" (Efficient Results)
This is the magic trick. By combining the few real ratings with the many predicted ratings, they could calculate the results much faster and with more certainty.

The Analogy: Imagine you want to know if a new flavor of ice cream is better than the old one.
- Old Way (Standard A/B Test): You ask 100 people to taste both and rate them. It takes a long time and the results might be fuzzy.
- PULSE Way: You ask 100 people to taste it, but you also have a "taste predictor" that looks at how they licked the spoon, how fast they ate, and their facial expressions to guess the rating of 1,000 other people who didn't even taste it.
- Result: PULSE gave them the same level of certainty as a standard test but with 40% less noise (uncertainty). It's like getting a sharper photo with a shorter exposure time.

What They Found (The Surprises)

When they used PULSE to test different robot designs, they found some things that the old "exam scores" missed:

The Brain Matters Most: The biggest factor in whether a human liked the robot was which "brain" (the underlying AI model) the robot used. Switching to a smarter brain made a huge difference in user happiness.
The "Body" Matters Less: Changing how the robot organized its thoughts (like showing a to-do list or remembering things better) had a much smaller effect on user happiness.
The Exam Lies: This was the biggest shock. They compared their real-world human ratings against standard AI benchmarks.
- The Result: The benchmarks said Robot A was better than Robot B. But the humans actually preferred Robot B!
- The Lesson: A robot can ace the written test but be a terrible partner in a real conversation.

The Takeaway

The paper concludes that if we want to build AI that actually helps humans, we can't just look at test scores. We have to watch how humans and robots interact in the wild. PULSE is a tool that helps developers do this efficiently, using a mix of real feedback and smart predictions to build robots that people actually enjoy working with.

They also noted that while they focused on coding, this same "PULSE" method could be used for any robot that works with humans, like a travel planner or a shopping assistant, by simply changing the "clues" the Mind Reader looks for.

Technical Summary: Assessing Human-Agent Interactions via PULSE

Problem Statement

Current evaluation paradigms for Large Language Model (LLM) powered agents rely heavily on static benchmarks that measure accuracy under the assumption of full automation. These benchmarks typically evaluate an agent's ability to complete a well-specified task without user intervention. However, real-world agent deployment often involves collaborative, human-in-the-loop workflows where agents work alongside human supervisors. Existing literature lacks a rigorous evaluation protocol for these interactive settings, and prior attempts to simulate human interaction often remain within controlled, benchmark-driven conditions rather than capturing genuine "in-the-wild" user feedback. Consequently, there is a gap in understanding how specific agent design decisions impact actual user satisfaction and how benchmark performance correlates with real-world human experience.

Methodology: The PULSE Framework

To address these gaps, the authors propose PULSE (Prediction-powered User Label Synthesis and Evaluation), a three-step framework designed for efficient, human-centric evaluation of agent designs.

1. Feedback Data Collection

The framework deploys a minimally invasive feedback mechanism within a web-based interface (built around the open-source agent OpenHands). Instead of asking for feedback at arbitrary intervals, the system prompts users to rate the agent's performance on a 5-star scale immediately after each "work segment." A work segment is defined as the sequence of events from when a user sends a command, the agent enters a "running" state, and returns to "stopped." This design grounds ratings in specific, just-finished tasks, reducing noise and unfocused judgments.

2. Predicting Human Satisfaction

Given that explicit ratings are sparse (only ~5% of interactions in the study received ratings), the authors train a Machine Learning (ML) model to predict user satisfaction for unlabeled trajectories.

Feature Engineering: Through an iterative human-in-the-loop process, 15 features were identified and categorized into:
- User-based: User messages, sentiment, and message count.
- Agent-based: Task type (e.g., bug fixing vs. new features) and failure modes (e.g., insufficient testing, not following instructions).
- Task Progression: Git actions (push, commit, reset) indicating task completion.
Model Training: Various ML models (Logistic Regression, Random Forest, HistGradientBoosting) were trained on labeled trajectories. The study found that these feature-based models significantly outperformed "LLM-as-a-judge" baselines (using models like o3 and gemini-2.5-pro) in predicting satisfaction, achieving higher correlation with human ratings and lower error rates.

3. Computing Effect Size with Prediction-Powered Inference (PPI)

To compare different agent designs (e.g., varying LLM backbones or planning strategies), the framework employs Prediction-Powered Inference (PPI).

Mechanism: PPI leverages the trained prediction model $f$ to generate pseudo-labels for unlabeled data. It constructs an effect size estimator that combines the mean of labeled data with a correction term derived from the difference between predictions on labeled and unlabeled data.
Statistical Advantage: This approach reduces the variance of the estimator, allowing for the computation of valid confidence intervals even with sparse labeled data. The framework compares the "naive" effect size (using only labeled data) against the "augmented" effect size (using PPI).

Experimental Deployment and Case Studies

The framework was deployed in a large-scale software engineering context involving 15,000 users and 36,000 sessions. The study conducted three specific case studies to evaluate design decisions:

LLM Backbone: Comparing claude-3.7-sonnet, claude-4-sonnet, and gpt-5.
Planning Strategy: Comparing agents that display a structured task plan (TASKS.md) versus those that proceed with direct implementation.
Memory Management: Comparing context window management strategies (reducing max_step from 120 to 80 to save costs).

Key Results

Impact of Agent Design on User Satisfaction

LLM Backbone Dominance: The choice of the underlying LLM had the most significant impact on user satisfaction. Users significantly preferred claude-4-sonnet over claude-3.7-sonnet (+5.86%) and over gpt-5 (-7.83%).
Scaffolding Effects: Changes to scaffolding (planning and memory) had smaller but statistically significant effects. Showing a plan improved satisfaction by ~3.1%, while reducing memory steps saved costs without degrading user experience.
Behavioral Insights: Analysis of interaction features revealed that lower satisfaction with gpt-5 correlated with fewer user messages (indicating early disengagement) and fewer code pushes. Conversely, showing plans reduced "misunderstood intention" and increased user engagement.

Efficiency of Evaluation

Confidence Interval Reduction: By utilizing PPI, the framework reduced the width of confidence intervals by an average of 39.5% compared to standard A/B testing using only labeled data. In several cases, this reduction allowed researchers to draw statistically significant conclusions where the naive approach could not.
Model Performance: The feature-based ML models outperformed LLM-as-a-judge baselines, improving correlation with outcomes by at least 26%.

Benchmark vs. Real-World Discrepancies

The study highlighted a critical misalignment between benchmark performance and human satisfaction:

While gpt-5 outperformed claude-4-sonnet on 6 out of 7 code-related benchmarks, humans preferred claude-4-sonnet in 4 out of 7 task subsets in the wild.
The correlation between benchmark deltas ( $\Delta_B$ ) and human rating deltas ( $\Delta_H$ ) was weak or negative ( $\rho = -0.18$ ) for the claude-4 vs. gpt-5 comparison, suggesting that benchmark improvements do not always translate to better user experiences.

Significance and Claims

The authors claim that PULSE provides a rigorous, scalable framework for evaluating human-agent interactions, moving beyond the limitations of full-automation benchmarks. The study demonstrates that:

Human-in-the-loop evaluation is essential: Benchmark performance is an imperfect proxy for real-world utility, particularly in collaborative settings.
Efficient evaluation is possible: PPI allows for robust statistical conclusions with significantly fewer labeled samples, making large-scale human-agent studies more feasible.
Design priorities: For software agents, investing in stronger base models yields larger returns in user satisfaction than optimizing scaffolding mechanisms, though scaffolding changes (like planning visibility) still offer measurable benefits.

The paper concludes that while software engineering served as the primary testbed, the PULSE framework is generalizable to other domains where agents and humans collaborate, offering a pathway to better align AI systems with user needs.