EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair

The paper introduces EigenData, a self-evolving multi-agent platform that automates the synthesis, auditing, and repair of high-quality function-calling training data, demonstrating its effectiveness by systematically correcting the Berkeley Function-Calling Leaderboard (BFCL-V3) to achieve model rankings that better correlate with human judgments of functional correctness.

The Gist

Imagine you are trying to teach a brilliant but inexperienced apprentice (an AI) how to run a complex business, like a travel agency or a bank. To do this, you need to give them a playground where they can practice. This playground needs three things:

1. The World: A realistic database (like a ledger of flights, passengers, and tickets).
2. The Tools: Working buttons and levers (software code) that let the apprentice interact with that world.
3. The Scenarios: A script of conversations showing how a human might ask for help and how the apprentice should respond.

The problem is that for a long time, people built these playgrounds by hand. It was slow, expensive, and full of mistakes. Sometimes the "buttons" were broken, sometimes the "ledgers" had math errors, and sometimes the "scripts" told the apprentice to do things that were impossible.

Enter EigenData. Think of EigenData not as a single person, but as a self-correcting construction crew made entirely of AI specialists. It doesn't just build the playground; it audits it, fixes the cracks, and even rewrites the scripts while the apprentice is practicing.

Here is how this "AI Construction Crew" works, broken down into simple parts:

1. The Foreman: EigenCore

Imagine a project manager named EigenCore. You don't need to tell it exactly how to code or how to design a database. You just say, "Build me a training ground for a hotel booking agent," or "Fix the broken test questions in our current exam."

EigenCore listens, breaks the big job into smaller tasks, and assigns them to three specialized teams. It acts as the conductor of an orchestra, making sure the database team, the coding team, and the scriptwriting team are all talking to each other.

2. The Three Specialized Teams

The Database Architect (DatabaseAgent)

• The Job: Builds the "World."
• The Analogy: Imagine a master carpenter building a realistic model city. They don't just put random houses down; they ensure the plumbing works, the streets connect logically, and the population numbers make sense.
• What they do: They create the digital databases (the ledgers) that the AI will use. They make sure that if a user asks for a "flight to Paris," the database actually has flights to Paris and that the prices make sense. They also sneak in tricky scenarios, like "sold-out flights," to test if the AI can handle problems.

The Code Mechanic (CodingAgent)

• The Job: Builds the "Tools."
• The Analogy: Imagine a mechanic who builds the actual buttons and levers for the model city. But here's the twist: this mechanic has a super-strict inspector (a Judge) who checks every single button they build.
• The Process:
  1. The mechanic builds a button (code).
  2. The inspector tries to break it (runs tests).
  3. If the button breaks, the mechanic fixes it.
  4. Crucially: If the button works but the inspector thinks it's broken because the inspector's instructions were wrong, the mechanic can say, "Hey, your test is wrong, not me!" and they fix the test instead.
  5. They keep looping until the button is perfect. This ensures the tools the AI uses are actually functional.

The Scriptwriter (DataAgent)

• The Job: Writes the "Scenarios" (Conversations).
• The Analogy: Imagine a director running a play. They have actors (AI agents) playing the "Customer" and the "Agent."
• The Process:
  • They generate thousands of conversations.
  • They have a "Drama Coach" (a Judge Agent) who watches the play. If the "Agent" character says something weird or uses a tool incorrectly, the coach stops the scene, tells the actors what to fix, and they re-do the scene.
  • They use a two-phase approach: First, they rehearse with a small group to get the script perfect. Once the script is polished, they cast the whole theater company to perform the full play (generate the massive dataset).

3. The "Self-Healing" Superpower

The coolest part of EigenData is that it doesn't just build things once and leave. It can audit and repair existing broken things.

The authors tested this on a famous exam called BFCL (Berkeley Function-Calling Leaderboard). Think of this exam as a standardized test for AI agents.

• The Problem: They found that 71.5% of the exam questions were broken!
  • Some questions asked for a tool that didn't exist.
  • Some "correct answers" were actually wrong.
  • Some tools were programmed to fail even when the AI did the right thing.
• The Fix: EigenData went through the exam like a team of expert editors.
  • The Database Architect fixed the fake data.
  • The Code Mechanic fixed the broken tools.
  • The Scriptwriter rewrote the confusing questions.
• The Result: They created a "Repaired Exam." When they re-ran the test, the rankings of the AI models changed completely. Some models that looked bad on the broken exam were actually geniuses; others that looked good were just cheating the broken system.

4. Why This Matters: The "Outcome" vs. The "Script"

In the old days, if an AI took a different path to solve a problem than the one written in the textbook, it was marked wrong.

• Old Way: "You were supposed to turn left, then right. You turned right, then left. Fail." (Even if you ended up at the same destination!)
• EigenData Way: "Did you get to the destination? Is the database updated correctly? Pass."

EigenData introduces an "Outcome-Aware" system. It doesn't care if you followed the script word-for-word; it cares if you actually fixed the problem. It checks the result (did the flight get booked?), not just the steps.

Summary

EigenData is a self-evolving platform that uses a team of AI specialists to:

1. Build realistic training worlds.
2. Create working tools for those worlds.
3. Generate and fix training conversations.
4. Audit and repair existing tests to make sure they actually measure intelligence, not just the ability to guess the right answer to a broken question.

It's like having a construction crew that builds a school, teaches the students, grades the exams, and then realizes the exams were flawed, so they immediately tear down the bad questions and rewrite them to be fair—all without a human needing to lift a finger.

Technical Summary

Here is a detailed technical summary of the paper "EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair."

1. Problem Statement

Function-calling agents (LLMs that invoke tools and APIs) require high-quality, domain-specific training data to perform effectively in real-world scenarios (e.g., travel booking, logistics). However, the current landscape faces three critical bottlenecks:

• Data Scarcity & Cost: High-quality data currently relies on expensive, slow human annotation, creating a trade-off between scale and quality.
• Synthetic Data Brittleness: Existing automated pipelines (e.g., BFCL, TOUCAN) often produce data with schema bugs (inconsistent definitions), ambiguous intents (multiple valid outcomes labeled as one), and weak evaluation metrics (focusing on turn-level matching rather than actual task success).
• Benchmark Contamination: Widely used benchmarks often contain systematic errors in function schemas, code implementations, and reference trajectories, leading to misleading model rankings that do not correlate with human judgments of functional correctness.

2. Methodology: The EigenData Platform

The authors introduce EigenData, an integrated, self-evolving multi-agent platform designed to automate the full lifecycle of function-calling data: from environment construction to trajectory synthesis, auditing, and repair.

Core Architecture

The system is orchestrated by EigenCore, a top-level controller that decomposes user requests into a dependency-aware task graph. It coordinates three specialized sub-systems:

1. DatabaseAgent (Environment State):

   • Constructs realistic, internally consistent domain databases (relational, JSON, or key-value) based on natural language or schema specifications.
   • Employs constraint-aware sampling, distribution modeling, and edge-case injection to ensure the database supports diverse tool-use scenarios.
   • Performs consistency verification to ensure referential integrity before passing data to the next stage.

2. CodingAgent (Executable Layer):

   • Generates the executable code layer (Python simulations, REST APIs, or MCP servers) that exposes tool interfaces backed by the database.
   • Utilizes a two-stage iterative test-debug workflow:
     • Stage 1 (Unit Testing): Generates unit tests for individual functions. If tests fail, a JudgeAgent performs fault attribution to determine if the error lies in the code or the test itself, triggering targeted regeneration.
     • Stage 2 (Workflow Testing): Generates integration tests for dependency groups to ensure state consistency across function calls.
   • Includes a ReviewAgent for final quality gates.

3. DataAgent (Trajectory Synthesis):

   • Synthesizes multi-turn function-calling trajectories using a hierarchy of worker agents (e.g., UserSimulator, TrajectoryAgent, ToolGraphComposer).
   • Operates in a two-phase self-evolving process:
     • Phase 1: Optimizes prompts and agent behaviors on a small pilot batch via iterative judge feedback.
     • Phase 2: Scales to full production with continuous quality monitoring.
   • Supports surgical repair: Instead of regenerating entire conversations, it segments dialogues into "safe," "affected," and "candidate" zones to fix specific errors caused by schema changes or logic bugs.

Key Innovation: Outcome-Aware Evaluation

EigenData introduces an evaluation protocol that moves beyond simple turn-level function matching. It utilizes VerificationFunctionAgent to generate programmatic reward models that assess:

• Whether the predicted database state matches the ground truth.
• Whether key functions were correctly invoked.
• Whether the user's intent was successfully fulfilled.

3. Key Contributions

• End-to-End Self-Evolving Infrastructure: Unlike linear pipelines, EigenData provides a closed-loop system where cross-component feedback allows for targeted repairs (e.g., fixing a schema bug automatically triggers updates to the code and reference trajectories).
• Comprehensive Benchmark Auditing: The system successfully audited the Berkeley Function-Calling Leaderboard (BFCL-V3), identifying systematic errors in 71.5% of multi-turn entries.
• Novel Repair Workflows:
  • Schema Polish: Iteratively refines API specifications based on verification feedback.
  • Data Repair: Performs "surgical" reconstruction of dialogues, preserving valid context while fixing only the erroneous turns.
  • Schema-Triggered Patch: Automatically updates existing trajectories when function signatures evolve, avoiding full dataset regeneration.
• Outcome-Aware Metrics: Shifts evaluation from syntactic matching to semantic success (database state correctness), providing a more reliable signal for model capability.

4. Results: The BFCL-V3 Case Study

The authors applied EigenData to audit and repair the BFCL-V3 multi-turn benchmark (200 representative cases).

Error Discovery:

• Function Schema Errors (19.5%): Type mismatches (e.g., integer vs. string) and ambiguous descriptions.
• Implementation Errors (45.5%): Logic bugs (e.g., else branches swallowing valid enum values) and state management flaws.
• Trajectory/Evaluation Errors (41.0%): Incorrect ground-truth labels and overly rigid evaluation criteria that penalized valid alternative solutions.

Evaluation Outcomes:

• Ranking Correction: The original BFCL leaderboard showed significant rank inversions compared to human judgment. For example, GPT-5.2 scored 28.12% on the original benchmark but achieved a 65.0% human pass rate on the repaired benchmark. Conversely, GLM-4.6 had the highest original score (68.00%) but ranked fourth in human evaluation (54.8%).
• Correlation with Human Judgment: The repaired benchmark's automated metric ("All Three" metric: Config Match + Key Function + LLM Judge) showed a substantially better correlation with human evaluations of functional correctness than the original benchmark.
• Failure Mode Analysis: The study revealed that models often fail on specific axes (e.g., correct function selection but malformed arguments), highlighting the need for multi-dimensional evaluation.

5. Significance

• Reliability: EigenData demonstrates that automated, multi-agent auditing can uncover and fix systemic errors in major benchmarks that human curators missed, leading to more trustworthy model rankings.
• Scalability: The platform offers a scalable alternative to human annotation, capable of generating, auditing, and repairing complex, domain-specific datasets without manual intervention.
• Paradigm Shift: It advocates for a shift from "trajectory matching" to "outcome-based evaluation," ensuring that agents are judged on their ability to achieve real-world goals rather than just mimicking reference text.
• Accessibility: The authors released a CLI tool allowing practitioners to apply these auditing and repair capabilities to their own domains and benchmarks.

In conclusion, EigenData represents a significant step forward in the infrastructure of AI agent development, providing a robust framework to ensure that the data used to train and evaluate agents is accurate, consistent, and reflective of true functional capability.