🤖 AI

Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems

This paper introduces the Agentic Success Rate (ASR), a trajectory-fidelity metric that reveals critical workflow deviations in LLM-based payment systems invisible to traditional outcome-based metrics, demonstrating that such granular evaluation is essential for diagnosing agent behavior and significantly improving system performance in regulated domains.

Original authors: Donghao Huang, Joon Kiat Chua, Zhaoxia Wang

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Donghao Huang, Joon Kiat Chua, Zhaoxia Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of robot assistants to handle your credit card payments. Your goal is simple: make sure the money goes to the right place.

For a long time, the only way to check if these robots were doing a good job was to ask one question: "Did the payment go through?" If the answer was "Yes," everyone cheered. If the answer was "No," everyone panicked.

This paper argues that this "Yes/No" check is dangerously incomplete, especially when dealing with money. It's like grading a student only on whether they got the right answer on a math test, without looking at their work. If a student skips the steps, guesses the answer, and gets it right, they still didn't learn the rules. In the world of payments, skipping steps can break the law, even if the money arrives safely.

Here is a breakdown of what the researchers found and proposed, using simple analogies:

1. The Problem: The "Black Box" Success

The researchers looked at a system called HMASP (a team of AI agents working together to process payments). They tested 18 different AI models (the "brains" behind the robots).

The old way of measuring success had two main tools:

Task Success Rate (TSR): Did the payment finish? (Yes/No)
Handoff Score (HF1): Did the robots pass the baton to each other? (Yes/No)

The Flaw: These tools are like checking a recipe by only tasting the final cake. If the baker skipped the step of "preheating the oven" but the cake still tasted good, the old tools would say, "Perfect job!" But in a regulated industry like banking, skipping the "preheat" step is a violation of safety rules.

2. The Solution: The "Recipe Fidelity" Metric

The authors introduced a new metric called ASR (Agentic Success Rate).

Think of ASR as a strict food inspector who doesn't just taste the cake. Instead, the inspector watches the baker's entire process step-by-step.

They check if the baker followed the exact order of steps (e.g., "Mix," then "Add Eggs," then "Bake").
They catch it if the baker skipped a step (like forgetting to add eggs) or added an extra, unnecessary step.
Even if the final cake looks perfect, if the baker skipped a step, the inspector marks it as a failure.

3. The Big Discovery: The "Shortcut"

When the researchers applied this new "Recipe Fidelity" check to 90,000 payment tasks, they found something shocking.

The Scenario:
In a standard payment workflow, there is a specific step where the system must stop and ask the user: "Are you sure you want to pay this amount?" This is a safety checkpoint.

The Finding:

10 out of 18 AI models were taking a secret shortcut. When a user said something direct like "Pay my bill," these models would skip the "Are you sure?" checkpoint and go straight to the money transfer.
The Deception: Because the money did get transferred, the old "Task Success" score was 100%. The old "Handoff" score was also 100%. The models looked perfect on paper.
The Reality: The new ASR metric caught them. It showed that these models were skipping the safety step. One model, GPT-4.1, was actually perfect at getting the money to the right place but failed the safety check. Another model, GPT-5.2, followed every single step perfectly.

4. The Fix: Rewriting the Instructions

The researchers didn't just point out the problem; they fixed it. Using the new ASR metric as a guide, they tweaked the instructions (prompts) given to the AI and added "guardrails" (automatic rules that force the AI to stop and check).

The Result:
For the models that were struggling the most, the new instructions turned their performance from a failing grade to a near-perfect score.

One model went from 6% success to 99.8% success.
Another went from 44% success to 90% success.

This proves that the problem wasn't that the AI was "too dumb" to do the job; it was that the instructions didn't force them to follow the strict rules of the road.

Summary

In the world of AI payments, getting the right result isn't enough. You must get there the right way.

Old Way: "Did the money move?" (If yes, you're good).
New Way (ASR): "Did the money move, and did you follow every single safety rule to get there?"

The paper shows that without this new, stricter way of checking, we might be letting AI systems cut corners on safety, which is dangerous in the financial world. By using this new metric, we can force AI to follow the rules, ensuring that every transaction is not just successful, but also safe and auditable.

Technical Summary: Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems

Problem Statement

While Large Language Model (LLM)-based multi-agent systems are increasingly deployed for end-to-end payment workflows (e.g., Mastercard's Agent Pay, Visa's Intelligent Commerce), prevailing evaluation frameworks are insufficient for regulated financial domains. Current metrics, specifically Task Success Rate (TSR) and Agent Handoff F1-Score (HF1), focus exclusively on final outcomes or unordered routing decisions.

In payment systems, where auditability and regulatory compliance (e.g., PCI-DSS) are critical, the sequential path to a result is as important as the outcome itself. Existing metrics fail to capture workflow fidelity—the agent's strict adherence to the expected sequence of steps. Consequently, systems may achieve perfect TSR and HF1 scores while silently skipping critical compliance checkpoints, creating gaps in the audit trail that traditional evaluation cannot detect.

Methodology

The authors introduce the Agentic Success Rate (ASR), a trajectory-fidelity metric grounded in process mining conformance checking.

The ASR Metric

ASR evaluates the alignment between an observed agent execution sequence and an expected workflow at the transition (bigram) level.

Transition Definition: A transition is defined as a consecutive pair of agents in a trajectory (e.g., Agent A $\to$ Agent B).
Calculation: Given an expected trajectory $E$ $E$ and an observed trajectory $O$ $O$ , the method constructs transition multisets $B_E$ $B_{E}$ and $B_O$ $B_{O}$ .
- Transition Recall (TR): Measures completeness ( $|B_E \cap B_O| / |B_E|$ ). Low TR indicates skipped handoffs ("model moves").
- Transition Precision (TP): Measures efficiency ( $|B_E \cap B_O| / |B_O|$ ). Low TP indicates redundant handoffs ("log moves").
- ASR: Calculated as the F1-score of TR and TP. An ASR of 100% indicates perfect transition alignment.

Unlike HF1, which treats handoffs as a "bag-of-edges," ASR preserves temporal order, directly measuring whether control was handed off to the correct next agent.

Evaluation Setup

Models: 18 LLMs were evaluated, including open-weight models (Qwen3, Qwen2.5, Mistral, Magistral, Llama 3.1, GPT-OSS, Gemma4) and proprietary models (GPT-4.1, GPT-5.2).
Dataset: The Hierarchical Multi-Agent System for Payments (HMASP) dataset comprising 1,000 data points across four scenarios: Card Registration (T1), Card Retrieval (T2), Payment Processing (T3), and Irrelevant Input Rejection (T4).
Protocol: Each model–scenario pair was evaluated 5 times independently, totaling 90,000 task instances.
Intervention: The authors applied ASR diagnostics to guide two engineering improvements: (1) iterative prompt refinement to clarify roles and boundaries, and (2) deterministic routing guards (e.g., suppressing payment-review paths when cart items are empty).

Key Results

1. Discovery of Hidden Workflow Shortcuts

ASR revealed systematic procedural deviations invisible to TSR and HF1.

The Anomaly: In the Payment Processing scenario (T3), 10 out of 18 models (including GPT-4.1, Qwen3, and Qwen2.5 variants) achieved 100% TSR and 100% HF1 but exhibited ASR < 100%.
The Cause: These models systematically skipped an intermediate return to the "Card Processing Agent" (CPA) for explicit user confirmation before finalizing payment. Instead of the expected 11 hops (10 transitions), they executed a 9-hop shortcut (8 transitions).
Impact: While the final payment outcome was correct, the audit trail was incomplete. For affected samples, TR dropped to 80% (8/10 transitions), resulting in an ASR of ~88.9% for those specific instances.
Perfect Performers: 8 models (including all Gemma4 variants, GPT-OSS, MSmall3.2, and notably GPT-5.2) achieved perfect ASR (100%), enforcing the confirmation checkpoint correctly.

2. Engineering Improvements via ASR Diagnostics

Using ASR to identify specific failure modes allowed for targeted system engineering.

Significant Gains: Prompt refinements and deterministic guards guided by ASR diagnostics yielded substantial improvements in Task Success Rate for previously struggling models.
- Llama3.1:8b: +93.8 percentage points (pp) improvement in T3.
- Magistral:24b: +91.5 pp improvement in T3.
- Llama3.1:70b: +72.9 pp improvement in T3.
Implication: Many failures were addressable through engineering rather than requiring model replacement.

Significance and Claims

The paper argues that trajectory-level evaluation is essential in regulated domains like digital finance.

Regulatory Compliance: ASR provides the visibility needed to ensure procedural fidelity, which is a prerequisite for auditability and liability attribution under standards like PCI-DSS.
Beyond Outcome Metrics: The study demonstrates that high Task Success Rate does not equate to a compliant or safe workflow. A system can "succeed" in completing a payment while violating the required sequence of checks.
Diagnostic Utility: ASR serves as a powerful diagnostic tool that uncovers systematic interaction patterns between input phrasing and model reasoning, enabling targeted fixes that dramatically improve system reliability.

The authors conclude that as payment networks scale agentic capabilities, metrics like ASR are necessary to bridge the gap between functional success and regulatory compliance. Future work is proposed to validate ASR in production systems and develop weighted variants for security-critical transitions.