💻 computer science

Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking

This paper introduces GenAI Evaluation, a scalable and governed pipeline that leverages LLM-as-a-judge with selective re-evaluation and strict schema controls to reliably assess retail conversational agents across multiple dimensions, achieving high alignment with human judgments on over two million interactions.

Original authors: Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Niranjan Kumar M, Balaji Nagarajan, Karthik Nair, Faysal Satter, Nithin Surendran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive, 24/7 digital help desk for a giant retail store. Every single day, about 50,000 customers chat with your robot assistant, asking everything from "Where are the hammers?" to "Can I return this shirt?" and even translating those questions into different languages. That's over 2 million conversations a year!

Now, imagine trying to grade every single one of those chats to see if the robot was helpful, truthful, and polite. If you tried to do this with human teachers, you'd need an army of them, and it would cost a fortune. If you tried to use old-school computer math (the kind that just counts how many words match), you'd miss the point entirely—like giving an A+ to a robot that said, "The sky is green," because it used the same number of words as a correct sentence, even though it was totally wrong.

The Paper's Big Idea: The "Smart, Selective" Grading Machine

The authors built a new system called GenAI Evaluation to solve this. Think of it as a super-organized, rule-following robot teacher that uses other, smarter robots (Large Language Models) to grade the first robot's work.

Here's how it works, using a few fun metaphors:

The Assembly Line: Instead of looking at the whole mountain of 2 million chats at once, the system chops them up into smaller, manageable piles (called "shards"). It's like a factory where workers only handle one box at a time, so if one box gets stuck, the whole factory doesn't stop.
The "Selective Re-do" Trick: This is the coolest part. Usually, if a robot teacher makes a mistake or gets confused, you'd have to throw away the whole pile and start over. That's a waste of time and energy. This new system is like a super-attentive editor who only highlights the specific sentences that are wrong. It says, "Hey, you got 99% right, but this one line is messy. Let's just fix that line." This is called selective re-evaluation. It saves a ton of computing power and makes sure the final grade book is complete without doing unnecessary work.
The Rulebook (Governance): The system is obsessed with rules. It locks down exactly how the grades should look (the "schema") so that no one accidentally writes a grade in the wrong column. It keeps a detailed diary of exactly which version of the robot teacher did the grading, what rules they used, and when it happened. This means if you ever need to check the work, you can trace it back perfectly, like following a breadcrumb trail.

What They Found (The Scorecard)

The team tested this system on real retail chat logs. They didn't just guess; they compared the robot's grades against a carefully chosen group of 12,980 conversations that were graded by four real human experts. The humans didn't know which robot was doing the grading (to keep it fair).

Here are the numbers, exactly as they measured them:

Understanding the Chat: When it came to figuring out what the customer wanted (like "I need a drill" vs. "I need a hammer"), the system got a 0.93 score on a scale where higher is better (called a "macro F1 score"). That's a very strong match with the human experts.
Translation: For the chats that needed to be translated into other languages, the system was 89% accurate according to human reviewers.
The Robot Teachers: They tested different sizes of "judge" robots. The tiny ones were fast but sometimes missed the nuance. The giant ones (like the 70B parameter model) were the most accurate, hitting that 0.93 score, but they needed more powerful computers (specifically NVIDIA H100 chips) to run.

What They Explicitly Say It's NOT

It's important to know what this system doesn't do, or what the authors warn against:

It's not a magic "Truth" machine: The authors are very clear that these robot grades are "quality signals," not absolute, unchangeable facts. They are suggestions based on patterns, not a divine verdict.
It's not perfect for every job yet: The system was built for retail chats. The authors explicitly say we don't know if it works for totally different worlds like giving medical advice, legal help, or writing code. Those might need different rules.
It's not a replacement for humans in big moments: If a chat is about something dangerous, sensitive, or really confusing, the system suggests sending it to a human. The robot is a helper, not the final boss.
It's not a "solved" problem: The authors admit that because they only had one human grade each chat (instead of multiple humans agreeing), they can't be 100% sure about how much the humans would agree with each other. They suggest future tests should have humans double-check the same chats to be more certain.

The Bottom Line

The paper suggests that by using a smart, rule-bound pipeline that only re-does the work that actually needs fixing, we can grade millions of robot conversations quickly and fairly. It's not a perfect, magic solution that works everywhere forever, but it's a huge step forward for making sure our retail robot friends are actually helpful, truthful, and polite, without needing a human to read every single word.

Technical Summary: GenAI Evaluation Pipeline for Retail Conversational Agents

1. Problem Formulation

The paper addresses the critical gap in evaluating retail conversational agents (chatbots) at an enterprise scale. Traditional metrics like BLEU, ROUGE, and METEOR rely on lexical overlap and fail to capture pragmatic dimensions essential in retail, such as intent alignment, factuality, tone, helpfulness, and cultural appropriateness in translation scenarios. While human evaluation is reliable, it is cost-prohibitive and unscalable for high-volume production environments generating tens of thousands of daily interactions. Conversely, existing automated "LLM-as-a-judge" approaches often lack the governance, reproducibility, and operational reliability required for production deployment, specifically regarding schema consistency, auditability, and cost management.

The core challenge is to operationalize a multi-dimensional evaluation framework that is:

Scalable: Capable of processing ~50,000 records daily (over 2 million total).
Governed: Enforcing strict schema, versioning, and traceability.
Efficient: Minimizing redundant computation through selective re-evaluation.
Multi-dimensional: Supporting both general retail tasks (intent, helpfulness) and translation-specific tasks (cultural fit, selling relevance).

2. Methodology: The GenAI Evaluation Pipeline

The authors propose GenAI Evaluation, a configuration-driven, governed pipeline built on Kubeflow Pipelines. The system ingests production chatbot logs from BigQuery and processes them through the following stages:

Data Preprocessing and Sharding

Normalization: Raw logs are standardized (e.g., converting context_name to context_id, casting timestamps and booleans).
Filtering: Invalid records (empty prompts/responses, missing IDs, malformed timestamps) are removed prior to evaluation to ensure failures reflect model quality rather than data defects.
Sharding: The dataset is divided into approximately balanced shards (e.g., $N=4$ ) using round-robin slicing and serialized to Parquet. This enables parallel processing and isolates failures to individual shards.
Metadata Augmentation: Records are enriched with stable global row identifiers, shard IDs, worker metadata, and configuration hashes to support traceability and deduplication.

Asynchronous LLM-as-a-Judge Execution

Architecture: The pipeline uses asynchronous mini-batch execution with bounded concurrency (managed via semaphores) to prevent network overload.
Task Specificity: It supports two distinct workflows:
1. General Retail: Evaluates helpfulness, clarity, coherence, conciseness, instruction adherence, tone alignment, and truthfulness.
2. Translation: Evaluates semantic accuracy, cultural appropriateness, readability, selling relevance, and template fit.
Configuration: Task-specific rubrics, prompts (Jinja2 templates), and output schemas are loaded from cloud storage as YAML configurations, ensuring versioned reproducibility.

Selective Re-evaluation and Governance

Selective Re-evaluation: Instead of rerunning the entire dataset upon failure, the system identifies incomplete, malformed, or schema-invalid records. It regenerates only these specific rows using stable row identifiers and merges them back, significantly reducing cost and latency.
Schema Governance: All outputs are type-checked and range-validated against a locked schema. Violations trigger audit logs.
Traceability: Every output includes provenance metadata (model version, config hash, worker ID, shard index). Results are streamed incrementally to BigQuery using schema-locked writers to prevent output drift.

3. Key Contributions

The paper outlines five primary contributions:

Governed Evaluation Stack: A scalable, configuration-driven pipeline for large-scale retail chatbot assessment that enforces schema consistency and auditability.
Selective Re-evaluation Mechanism: A targeted regeneration process for invalid or incomplete records that improves completeness while avoiding redundant computation.
Dual-Workflow Support: Separate schemas and workflows for non-translation (general intent/quality) and translation-specific (cultural/selling relevance) evaluation tasks.
End-to-End Auditability: Implementation of deterministic Parquet streaming, validation logs, versioned configurations, and per-record provenance to ensure reproducibility.
Model Benchmarking: A comparative analysis of open-source evaluator models to analyze quality-throughput trade-offs in high-volume retail settings.

4. Experimental Results

The framework was evaluated on over 2 million retail chatbot records (approx. 50,000 daily), with 95% non-translation and 5% translation examples. Validation was performed against a stratified-random human-labeled subset of 12,980 records using four blinded annotators.

Performance Metrics

Classification: The pipeline achieved a macro F1 score of 0.93 across 14 intents, 156 sub-intents, 18 major domain values, and 129 sub-domain values.
Translation: Achieved 89% human-acceptability accuracy for translation outputs.
Quality Scores: Dimensions like helpfulness, truthfulness, and clarity were recorded as numeric scores (1–5 scale).

Model Benchmarking

Five open-source models were benchmarked (Llama 4B/8B/70B, Qwen3-14B, OSS-120B) under identical conditions (4-bit quantization, NVIDIA H100, batch size 512).

Trade-offs: Smaller models (e.g., Llama 4B) offered higher throughput but lower scores on nuanced tasks (F1 0.82). Larger models (Llama 70B) achieved the highest alignment with human labels (F1 0.93) but required more resources.
Strategy: The authors suggest a tiered approach where smaller models handle daily monitoring and larger models are reserved for high-risk or audit-selected samples.

5. Significance and Claims

The paper claims that schema-governed LLM-based evaluation can provide scalable, reliable quality signals for enterprise retail workflows. Its significance lies in bridging the gap between the theoretical capabilities of "LLM-as-a-judge" and the practical requirements of production environments (governance, cost, and auditability).

The authors emphasize that:

Automated evaluation is a diagnostic signal, not a final judgment, and must be paired with human oversight for sensitive cases.
Selective re-evaluation is a critical engineering pattern for maintaining data completeness without incurring the cost of full re-runs.
Governance features (schema locking, versioned configs, deterministic streaming) are essential for reproducibility and trust in enterprise settings.

6. Limitations

The authors modestly acknowledge several constraints:

Validation Design: Each human-labeled record was reviewed by only one annotator, meaning the reported metrics reflect agreement with available labels rather than a fully adjudicated human consensus (no inter-annotator agreement statistics like Krippendorff's alpha were computed).
Domain Generalization: Results are specific to retail chatbot data and may not fully generalize to domains with different risk profiles (e.g., healthcare, legal).
Scope: The framework currently focuses on text-based turn-level evaluation and does not yet cover multimodal inputs or long-horizon interactive conversations.
Reproducibility: Long-term reproducibility remains challenging in dynamic environments where model weights or infrastructure may change.

Future work is proposed to include multimodal evaluation, online real-time scoring, and stronger reliability measurements (e.g., calibration curves, bootstrap confidence intervals).