🤖 AI

Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applications

This paper introduces Kaleidoscope, a practical and iterative workflow that integrates persona-based test generation, contextualized rubrics, and human-in-the-loop reliability gating to enable scalable, policy-aligned evaluation of real-world AI applications.

Original authors: Leanne Tan, Rohan Jaggi, Shaun Khoo, Roy Ka-Wei Lee

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leanne Tan, Rohan Jaggi, Shaun Khoo, Roy Ka-Wei Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot chef. You can teach it to chop vegetables and flip burgers with incredible speed, but how do you know it's actually making a good meal for your specific family? Maybe your family hates spicy food, or maybe you have a strict rule about not using plastic containers. A general cookbook might say, "This recipe is a 9/10," but that doesn't help you if your family is allergic to cilantro. This is the big problem in the world of Artificial Intelligence right now. We have super-smart AI models that can write stories, solve math problems, and chat like humans, but testing them is a nightmare. The standard tests used by scientists are like generic driving tests; they check if a car can stop at a red light, but they don't tell you if that car is safe to drive on the icy, winding roads of your specific neighborhood.

To fix this, we need a way to test AI based on our specific rules, our specific users, and our specific risks. This is where the idea of "functional evaluation" comes in. It's not about asking, "Is this AI smart?" but rather, "Does this AI do the job we need it to do, without breaking our rules?" The challenge is that doing this by hand is slow and boring, while letting a computer grade the computer is risky because the computer might be biased or lazy. So, the big question is: How can we build a testing system that is fast, fair, and actually understands the context of the real world?

Enter Project KALEIDOSCOPE, a new workflow designed by a team from GovTech Singapore and universities to solve this exact puzzle. Think of KALEIDOSCOPE not as a single test, but as a high-tech, multi-lens camera system for checking AI. Instead of taking one blurry photo of an AI's performance, it snaps hundreds of pictures from different angles, checks them against a custom rulebook, and only lets the final score count if the camera is sure it's focused correctly.

Here is how the magic happens. First, the system builds a cast of characters, or "personas." Imagine you are testing a customer service bot. Instead of just asking it generic questions, KALEIDOSCOPE creates a "grumpy customer," a "confused tourist," and a "tech-savvy teenager." It generates questions that these specific characters would actually ask, covering everything from normal situations to weird, edge-case scenarios. This ensures the test feels like real life, not a robot's dream.

Next, the system needs a rulebook. In the old days, teams would just guess what a "good" answer looked like. KALEIDOSCOPE lets users write their own rubrics—specific scoring criteria for their unique situation. For a finance bot, the rule might be "Must be 100% accurate with numbers." For a chatbot, it might be "Must sound friendly." The system then takes the AI's answers and starts grading them.

But here is the clever part: the system doesn't just trust a computer to grade the computer. It uses a "reliability gate." Imagine a panel of three different AI judges. Before they are allowed to give a final score, they have to prove they agree with a human expert on a small set of practice questions. If the AI judges can't match the human's logic at least 50% of the time (a specific threshold the team set), the system hits the brakes. It says, "Wait, these judges aren't reliable yet," and flags the results for more human review. This prevents the AI from confidently giving a wrong score. Only when the judges pass this test do they vote together to give a final, automated score.

The team tried this out in a three-week pilot with four different real-world teams: a finance bot, an HR assistant, a procurement bot, and a staff helper. They found that the system worked well. In fact, 83% of the people who used it said it helped them evaluate their apps more efficiently. They also learned some valuable lessons along the way. For instance, they realized that if you don't give the AI enough background information about the app it's testing, the test cases it generates can be weird and unrealistic. So, they added a feature where the system can search the web to learn more about the app's context if the user's description is too short.

The results suggest that this "human-in-the-loop" approach is a promising way to make AI testing practical. The team found that using a single AI judge to grade everything often led to inconsistent results, but using a "jury" of judges who had to agree with humans first made the scores much more trustworthy. They also discovered that breaking down answers into small claims (like checking each sentence for facts) was better than just giving a vague overall grade.

However, the paper is careful not to call this a perfect solution. The pilot was small, involving only a few teams and a limited number of test cases (108 annotated pairs). The authors suggest that while this workflow is a great step forward, it's not a magic wand that solves all AI safety problems. It still requires human effort to set up the rules and review the results, and it works best for checking if an AI gives the right answer to a question, rather than checking complex, long-term behaviors like an AI agent planning a multi-day trip.

In short, KALEIDOSCOPE is a toolkit that helps teams stop guessing and start knowing. It builds a bridge between the messy reality of human needs and the rigid logic of AI testing, ensuring that when an AI says "I'm ready to go," it actually is. By combining smart test generation, custom rulebooks, and a strict "trust but verify" system for automated scoring, it offers a practical path forward for anyone trying to deploy AI in the real world without losing their mind.

Technical Summary: Project KALEIDOSCOPE

Problem Statement

Evaluating Large Language Model (LLM) applications for real-world deployment faces a critical bottleneck: public benchmarks rarely align with specific organizational users, contexts, policies, or risk tolerances. While general capability benchmarks exist, they fail to capture the nuances of "good" performance in functional contexts (e.g., an HR bot requiring policy-grounded answers vs. a math bot requiring step-by-step solutions). Current organizational evaluation processes are often manual, fragmented, and tedious to scale. Existing tools address parts of the workflow (test generation, scoring, logging) but lack an integrated system that combines contextual test construction, configurable rubrics, human calibration, and reliability-gated automated scoring into a single, inspectable workflow.

Methodology: The KALEIDOSCOPE Workflow

KALEIDOSCOPE is an end-to-end workflow designed for contextual functional evaluation. It operates on four core design considerations: (1) representative test cases, (2) configurable rubrics, (3) optimized Human-in-the-Loop (HITL) experience, and (4) transparent, reliability-gated automated scoring. The workflow proceeds through the following stages:

Target Application Definition: Users specify the application's purpose, target users, and knowledge base. The system optionally retrieves public web information to ground the setup if user-provided context is sparse.
Persona-Based Test Generation: Instead of static datasets, KALEIDOSCOPE generates test cases by varying five dimensions: Persona (user background/style), Type (typical vs. edge cases), Scope (in-knowledge base vs. out-of-knowledge base), Input Style (brief vs. detailed), and Language. This ensures coverage of realistic user scenarios.
Human Review and Annotation: The system collects application responses and guides human reviewers to label a calibration set. To reduce automation bias, the interface uses an LLM-assisted workflow where the LLM suggests labels and highlights evidence, but the human reviewer must actively select a label from the rubric options rather than simply approving/rejecting. This establishes a trusted reference set.
Evaluation Rubrics: Users define custom rubrics using natural language or select presets. The system employs an LLM-based augmenter to rewrite user-defined criteria into structured judge prompts. Crucially, the design favors a "one-metric-per-judge-prompt" approach to avoid the performance degradation seen when a single prompt attempts to score multiple dimensions.
Automated Scoring and LLM-as-a-Jury:
- For each rubric, three candidate LLM judges are instantiated.
- Judges are evaluated against the human-labeled calibration set.
- Reliability Gate: Only judges achieving a Macro F1 score > 0.5 on the calibration set are eligible for aggregation.
- Aggregation: Eligible judges' scores are aggregated via majority vote. If no judge passes the gate, the system withholds automated scoring and flags the rubric for human review.
- Error Analysis: Disagreements between judges are surfaced to identify ambiguous examples or areas requiring closer human inspection.

Key Contributions

Workflow Artifact: The presentation of KALEIDOSCOPE as a practical, inspectable, and iterative workflow for contextual functional evaluation, covering the full lifecycle from target setup to error analysis.
Rubric and Judging Design: A novel approach to configurable evaluation criteria that utilizes claim-level or response-level scoring and employs a reliability-gated "jury" mechanism. This mechanism aggregates scores only when LLM judges demonstrate alignment with human-reviewed labels, mitigating the risks of uncalibrated automated scoring.
Empirical Evidence: Early results from a three-week pilot across four organizational use cases (Finance, HR, Procurement, Staff Assistant) and experiments on 108 annotated Q&A pairs across 14 evaluation dimensions.

Results and Findings

Pilot Usability: In a pilot involving eight users across four teams, 83% of respondents reported that KALEIDOSCOPE helped evaluate applications more efficiently. All respondents considered the judge reliability score when interpreting results, indicating the calibration signal is salient for decision-making.
Rubric Experiments:
- Multi-Metric vs. Single-Metric: Experiments comparing multi-metric judges (scoring multiple criteria in one prompt) against single-metric judges revealed that narrowing the scope improved alignment. For example, single-metric judges for "empathy" and "verbosity" showed Cohen's $\kappa$ improvements of +15.2% and +11.6% respectively over the multi-metric baseline.
- Prompt Iteration: Iterating on prompts within multi-metric brackets did not consistently improve performance and sometimes degraded it, suggesting scope overload.
- Prompt Augmentation: Using an LLM to augment user-defined rubrics into metric-specific prompts yielded mixed results. It improved performance for "structure" but not for "specificity," suggesting that augmentation effectiveness depends on the specific criterion and the ambiguity of the human annotation boundaries.
Design Revisions: User feedback led to revisions in test-case allocation (shifting from even distribution to realistic usage ratios), the addition of backend web-search grounding for sparse contexts, and the inclusion of import/export points to integrate with existing evaluation processes.

Significance and Claims

The paper positions KALEIDOSCOPE not as a universal benchmark or a solution to the general reliability of LLM judges, but as a practical workflow artifact for product teams. Its significance lies in:

Contextual Alignment: Shifting focus from general capability benchmarking to application-specific functional evaluation that respects local policies and governance.
Reliability Gating: Providing a mechanism to gate automated scoring behind human-verified alignment, ensuring that automated scores are only used when they meet a configured threshold of reliability.
Inspectability: Creating a transparent process where judge disagreements and calibration failures are visible, allowing teams to understand why a score was generated or withheld.

The authors modestly claim that the pilot provides "early evidence" and "early usability evidence" rather than definitive validation. They acknowledge that the current workflow is strongest for input-output evaluation and that future work is needed to extend the scope to complex agentic systems, multi-turn workflows, and retrieval-augmented generation (RAG) traces. The ultimate goal is to productize these lessons into a broader testing platform that closes the loop from diagnosis to targeted improvement.