🤖 AI

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

The paper introduces CRAFT, a method that converts rubric-based evaluations into a hierarchical capability tree to diagnose specific model weaknesses and generate targeted fine-tuning data, resulting in measurably improved performance across finance and legal domains compared to existing clustering and random generation baselines.

Original authors: Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

Published 2026-07-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly clumsy, robot how to do a new job. You give it a test, and it fails. A standard report might just say, "The robot failed the math section," or "The robot is bad at writing legal contracts." That's helpful, but it's also a bit vague. It's like a doctor telling you, "You have a stomach ache," without telling you if you ate something bad, if you're stressed, or if you need to take a pill. To fix the robot for next time, you need to know exactly what went wrong so you can practice that specific thing.

This is the world of Large Language Models (LLMs), the super-smart AI brains that write stories, solve problems, and answer questions. Scientists have spent years building better ways to test these AIs, but most tests just give a final score. They tell us where the AI is weak, but not why. The big question researchers are asking is: How do we turn a simple "you failed" into a specific "here is exactly what you need to practice"? If we can answer that, we can build better training data to help the AI learn faster and smarter, rather than just guessing what to teach it next.

The "Rubric" Detective: CRAFT

Meet CRAFT (Clustering Rubrics for Actionable Fine-Tuning). Think of CRAFT as a super-powered detective that doesn't just look at the final grade of a student's test; it looks at every single question on the rubric (the checklist of rules for a perfect answer) to figure out exactly which tiny skill the student missed.

Here is how the story unfolds:

1. The Problem with "Just a Score"
Imagine you are grading a math homework assignment. A standard test might just say, "You got 60%." That's not very helpful for fixing the problem. Did the student forget the formula? Did they make a simple addition error? Did they forget to write the units (like "meters" or "dollars")?
In the world of AI, researchers use rubrics. A rubric is like a detailed checklist. For a math problem, the rubric might say: "1. Identify the right numbers. 2. Set up the equation. 3. Solve the math. 4. Add the correct units."
Most AI tests stop at the final score. But CRAFT says, "Wait! Let's look at the checklist." It treats every single item on that checklist as a tiny "probe" or a mini-test for a specific skill.

2. The Magic Tree
CRAFT takes thousands of these checklist items from different questions and asks a smart AI to describe what skill each one is testing. Then, it does something magical: it builds a family tree of skills.

At the top of the tree, you have broad categories like "Finance" or "Legal."
As you go down the branches, the skills get more specific. "Finance" splits into "Corporate Finance," which splits into "Financing Costs," which splits into "Calculating Interest Rates."
At the very bottom leaves of the tree are the specific checklist items, like "Did the answer mention the interest rate?"

This tree is special because it's not just a list; it's a map. It shows how skills are related.

3. Finding the Weak Spots
Now, CRAFT tests the AI model on all these questions. It doesn't just count the total score; it checks the AI's performance at every single branch of the tree.

Maybe the AI is great at "Corporate Finance" (84% pass rate) but terrible at "Calculating Interest Rates" (48% pass rate).
Maybe another AI is okay at "Interest Rates" but fails at "Understanding Policy Changes."

CRAFT is smart enough to know that you shouldn't just pick the very bottom leaves (too specific) or the very top branches (too vague). It dynamically searches the tree to find the "sweet spot" where the weakness is clearest. It's like a coach who realizes, "You don't need to practice all of soccer; you specifically need to practice left-footed corner kicks."

4. The Targeted Practice
Once CRAFT finds these weak spots, it doesn't just stop there. It uses them to generate targeted practice data.
Imagine you are a teacher. Instead of giving the student 1,000 random math problems, CRAFT says, "Here are 40 problems specifically about 'Calculating Interest Rates' because that's where you keep failing."
The researchers used this method to create synthetic (computer-generated) practice examples for four different AI models (Qwen3-4B, Qwen3-8B, Gemma-3-4B, and Llama-3.1-8B-Instruct) in two tough fields: Finance and Legal.

5. The Results: Did It Work?
The team compared CRAFT against two other methods:

Random: Picking practice problems at random from the same topic.
EvalTree: A similar method that groups whole questions together instead of breaking them down into checklist items.

The results were clear:

In Finance: CRAFT was the winner for all four AI models. It gave them the biggest boost in performance, especially for the smaller models.
In Legal: CRAFT was the best for three out of four models. For the fourth model, it was just as good as the best competitor, but not worse.

The study showed that breaking things down into tiny, specific skills (the rubric criteria) is much better than just looking at whole questions. It's the difference between telling a student "You're bad at math" versus "You need to practice long division."

What CRAFT is NOT
It's important to note what this paper doesn't say. It doesn't claim that CRAFT fixes every problem or that it works perfectly on every single benchmark. In fact, the results varied a bit depending on the specific AI model and the specific test. Sometimes, one method was slightly better on one specific question, but CRAFT consistently won when looking at the average performance across the whole domain. The paper also emphasizes that this is a method for generating better training data, not a magic wand that instantly makes an AI perfect.

The Big Takeaway
CRAFT suggests that if we want to make AI smarter, we need to stop looking at the big picture and start looking at the tiny details. By using detailed checklists (rubrics) to find exactly why an AI fails, we can build targeted practice sessions that actually fix the problem. It turns a vague "you failed" into a clear roadmap for improvement, proving that the best way to learn is to know exactly what you need to practice.

Technical Summary: CRAFT – Clustering Rubrics to Diagnose Weak LLM Capabilities

1. Problem Statement

Current evaluation pipelines for Large Language Models (LLMs) primarily serve to measure performance but often fail to diagnose why a model fails. Standard benchmarks typically summarize results at the task or category level (e.g., "weak on math"), which conflates distinct underlying failures. For instance, a low score on a math problem could stem from selecting the wrong equation, making an arithmetic error, or omitting units—failures that require different interventions.

While rubric-based evaluations decompose responses into explicit criteria, existing work largely treats these criteria as independent scoring devices or reward signals. There is a gap in leveraging rubric datasets to generate a structured, model-specific diagnosis of weak capabilities that can directly guide the collection of targeted post-training data.

2. Methodology: CRAFT

The authors propose CRAFT (Clustering Rubrics for Actionable Fine-Tuning), a method that converts rubric-annotated evaluation data into a hierarchical, model-specific diagnosis of weak capabilities. The pipeline consists of four main stages:

2.1 Capability Extraction

CRAFT treats each prompt–rubric pair as a "capability probe." For every criterion in the dataset, a language model extracts a decontextualized, short natural-language description of the specific capability required to satisfy that criterion. This transforms the dataset from a collection of grading rules into a pool of capability descriptions.

2.2 Hierarchical Clustering

The extracted capability descriptions are organized into a hierarchical tree using a bottom-up, level-by-level procedure adapted from Clio:

Fixed Top Levels: If the dataset provides metadata (e.g., domain or subcategory), these define the uppermost levels of the tree to preserve intended taxonomies.
Base-Level Clustering: Within each group, capability descriptions are embedded and partitioned using $k$ -means into base clusters. An LLM names each cluster to generate a semantic description.
Higher-Level Construction: The tree is built dynamically. Internal nodes are created by summarizing child clusters, consolidating candidates, routing children to parents, and refining parent descriptions. The depth of the tree is not fixed; it is determined by the data size, allowing larger datasets to produce deeper hierarchies.

2.3 Scoring and Dynamic Selection

The target model is evaluated against the rubric criteria using an LLM-as-a-judge to generate pass/fail labels. These leaf scores are aggregated upward to calculate a pass rate for every node in the tree.

Dynamic Selection: Instead of committing to a fixed tree depth, CRAFT performs a top-down, cross-level search to identify low-performing nodes. A node is selected only if it has sufficient support (e.g., >30 criteria) and meets a pass-rate threshold.
Granularity: Once a node is selected, its ancestors and descendants are excluded to ensure disjoint weak-capability regions. This allows the method to select weaknesses at the specific granularity where they are most clearly expressed (e.g., a broad parent node for a general weakness or a narrow child node for a specific failure).

2.4 Targeted Data Generation

The selected weak capability nodes serve as targets for generating supervised fine-tuning (SFT) data. For each weak node, 40 synthetic prompts are generated, conditioned on the node's description, examples, and tree context. These prompts are designed to target the specific diagnosed weakness while avoiding overlap with adjacent capabilities.

3. Experimental Setup

The authors evaluated CRAFT against two baselines across four open-source models (Qwen3-4B, Qwen3-8B, Gemma-3-4B, Llama-3.1-8B-Instruct) in two professional domains: Finance and Legal.

Baselines:
- EvalTree: A prompt-level hierarchical clustering method that organizes whole prompts rather than rubric criteria.
- Random: A non-targeted approach that samples prompts uniformly from the domain pool without building a hierarchy or diagnosing weaknesses.
Controls: All methods used identical downstream settings: the same teacher model for generation, a fixed 1,000-example training budget, identical SFT hyperparameters, and the same evaluation protocol.
Benchmarks: Performance was measured on 13 held-out benchmarks (7 legal, 6 finance) disjoint from the diagnostic data. Results were averaged over repeated temperature-decoding runs to account for variance.

4. Key Results

Finance Domain: CRAFT achieved the strongest average domain performance for all four models. The improvements were particularly significant for Qwen3-4B and Llama-3.1-8B, where CRAFT outperformed baselines by over 5 points on the domain average.
Legal Domain: CRAFT achieved the strongest average performance for three of the four models. For the fourth model (Qwen3-8B), CRAFT's score fell within the decoding-variance band of the best baseline (EvalTree), indicating comparable performance.
Ablation Study: The authors compared CRAFT's dynamic selection against fixed-level selection (L4 and L5). Dynamic selection consistently outperformed fixed levels, demonstrating that weaknesses manifest at different depths depending on the model and domain. Fixed levels either fragmented broad weaknesses or merged narrow ones, whereas dynamic selection adapted to the evidence.

5. Significance and Claims

The paper claims that CRAFT demonstrates the value of moving beyond task-level or prompt-level diagnostics to rubric-criterion-level diagnosis.

Diagnostic Granularity: By treating rubric criteria as capability probes, CRAFT identifies why a model fails (e.g., specific reasoning steps or formatting requirements) rather than just where it fails.
Actionable Output: The method translates measured failures directly into targeted fine-tuning data. The results suggest that evaluations are most useful for post-training when they explain failure modes and convert them into specific training targets.
Superiority over Prompt-Level Clustering: The comparison with EvalTree indicates that organizing data by rubric criteria provides more informative diagnostic units than organizing by whole prompts.
Value of Structure: The comparison with the Random baseline confirms that simply adding more in-domain data is less effective than structuring that data around a scored capability hierarchy and targeting specific weak nodes.

In summary, the authors posit that CRAFT provides a sharper picture of model limitations and yields measurably better models after fine-tuning by addressing the root causes of failure identified through rubric clustering.