🤖 machine learning

Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?

This paper proposes and evaluates a method for automatically inferring reusable natural-language rubrics from accumulated inline comments, demonstrating that these distilled criteria can effectively support comment prediction, rubric understanding, and automatic artifact revision in real-world and controlled settings.

Original authors: Kotaro Yoshida, So Kuroki, Yuki Imajuku, Taishi Nakamura, Ryunosuke Iwai, Haruki Goda, Takuya Akiba

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kotaro Yoshida, So Kuroki, Yuki Imajuku, Taishi Nakamura, Ryunosuke Iwai, Haruki Goda, Takuya Akiba

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a junior editor at a publishing house. You have a stack of manuscripts, but you don't know exactly what your senior editors are looking for. You've seen their red pen marks (inline comments) on old drafts, but you've never been told the rules behind those marks. You know they circle a sentence and write "Too vague," but you don't know if that means "give more numbers," "explain the logic," or "cite a source."

This paper, titled "Feedback-to-Rubrics," is about teaching a computer to figure out those hidden rules just by looking at the pile of red pen marks.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The "Tacit" Secret Sauce

Usually, when experts (like senior editors or scientists) review work, they have a mental checklist of what makes something good or bad. But they rarely write this checklist down. It's "tacit" knowledge—like a chef who knows a soup needs more salt but can't explain exactly why or how much without tasting it first.

Large Language Models (LLMs) are great at writing, but they are terrible at guessing these hidden, unwritten rules. If you ask an AI to "make this better," it might change the wrong things because it doesn't know the specific "flavor" of the expert's preferences.

2. The Solution: Turning "Scratches" into a "Recipe"

The authors propose a new way to learn these rules. Instead of asking humans to write a rulebook (which is hard because they don't know how to explain their gut feelings), they look at the inline comments experts have already left on thousands of drafts.

Think of the inline comments as scratches on a map.

The Old Way: Try to guess the treasure location by looking at the whole map.
This Paper's Way: Look at every single scratch, figure out what kind of treasure the map-maker was hunting for, and then draw a new, clear "Treasure Map" (a Rubric) that explains exactly where to look.

3. How the Machine Learns: The "Guess, Check, and Fix" Loop

The method works like a student studying for a test by doing practice questions and checking the answer key.

The First Guess: The computer looks at a bunch of old drafts and the comments on them. It tries to write a rough "Rulebook" (Rubric) based on what it sees.
The Simulation: The computer then pretends to be an expert. It takes a new draft, reads its own Rulebook, and tries to write comments on it.
The Reality Check: It compares its own comments to the real comments written by the human expert.
- Did the computer miss a point the human made?
- Did the computer complain about something the human ignored?
The Fix (The Magic Step): This is the most important part. The computer doesn't just say "I got it wrong." It looks at exactly which rule in its Rulebook caused the mistake.
- Analogy: Imagine you are trying to learn to bake a cake. You taste it, and it's too salty. Instead of just saying "The cake is bad," you realize, "Ah, the rule I wrote said 'add a pinch of salt,' but I actually needed 'a pinch of salt per cup of flour.'"
- The computer updates its Rulebook to be more specific, adding boundaries and examples so it doesn't make the same mistake next time.

They do this over and over (iteratively), refining the Rulebook until it perfectly mimics the expert's style.

4. What They Found (The Results)

The authors tested this on nine different types of writing, from research proposals to medical chat logs. They found three main things:

Better Comments: When they used the computer's learned Rulebook to generate feedback, the feedback was much more accurate and helpful than when the computer had no rulebook or just looked up similar past comments.
A Clearer Rulebook: The final Rulebook wasn't just a list of vague ideas like "be clear." It became a detailed, specific guide (e.g., "If a research question is too broad, point out that it needs a specific scope"). It actually captured the "secret sauce" of the experts.
Better Revisions: When they gave this learned Rulebook to an AI and asked it to fix a bad draft, the AI did a much better job of improving the text to match what the human experts wanted.

5. The Bottom Line

This paper shows that you don't need to ask experts to write a manual on how to review things. You can just feed a computer their past notes (comments), let it play "guess the rule" and "fix the mistake" repeatedly, and it will eventually learn a reusable, written set of criteria that captures their expertise.

What they did NOT do:

They did not test this on medical diagnoses or clinical treatments.
They did not claim this will replace human editors entirely.
They did not say this works for any type of data, only for text drafts with inline comments.

In short: They taught a computer to read between the lines of human feedback and turn those messy notes into a clean, usable instruction manual.

Technical Summary: Feedback-to-Rubrics

Problem Setting

Large language models (LLMs) are increasingly deployed for writing and review support, yet their effectiveness is often limited by a lack of context-dependent criteria, such as tacit expert preferences or organization-specific conventions. While explicit rubrics can guide LLM outputs, these criteria are frequently undocumented and difficult to elicit directly from humans. Existing approaches to learning criteria from human behavior typically rely on global judgment signals (e.g., satisfaction scores or pairwise preferences) rather than the local, span-specific feedback found in inline comments.

This paper introduces Feedback-to-Rubrics, a new problem setting that formalizes the learning of reusable, natural-language rubrics from accumulated inline comments on text artifacts (e.g., human-written or LLM-generated drafts). The core challenge is to infer explicit criteria that explain what concerns experts raise, when they raise them, and at what level of granularity, based solely on observations of tacit criteria embedded in comments.

Methodology

The proposed method learns rubrics through an iterative loop consisting of comment-prediction-based learning and comment-wise refinement.

1. Rubric Definition

A rubric at round $t$ is defined as a set of natural-language criteria $R_t = \{c_k^{(t)}\}$ . Each criterion includes:

Applicability conditions: When to comment in a local context.
Criterion text: The specific concern and its granularity.
Boundaries: Constraints to prevent confusion with nearby criteria.
Examples: Representative target quotes and reference comments.

2. Iterative Refinement Loop

The method operates in rounds, starting with an initial rubric $R_0$ inferred from training data:

Comment Prediction & Evaluation: A comment LLM ( $G$ ) generates comments for target quotes conditioned on the current rubric $R_t$ . Crucially, the model outputs both the comment and the specific subset of criteria $C_{ij}^{(t)}$ used to generate it. An evaluation LLM ( $J$ ) then compares the generated comment against the reference comment, returning a content score and reasoning.
Comment-Wise Refinement Signal: Unlike prior work that uses aggregate signals, this method constructs a refinement signal $h_{ij}^{(t)}$ for each instance, pairing the prediction result with the specific criteria used. This signal includes the target quote, reference comment, generated comment, score, judge reasoning, and the selected criteria.
Rubric Update: A refinement LLM uses these comment-wise signals to update the rubric to $R_{t+1}$ . It distinguishes between overly broad criteria, missing concerns, unclear boundaries, and redundant criteria. The refinement process allows the system to split broad criteria, add new ones for missing concerns, or merge redundant ones, preserving criteria that align well with reference behavior.

Key Contributions

Problem Formulation: The introduction of Feedback-to-Rubrics, a setting for learning reusable natural-language rubrics from accumulated inline comments, addressing the gap between global preference learning and local span-level feedback.
Methodological Innovation: A comment-prediction-based method that infers initial rubrics and refines them using comment-wise mismatch signals. This approach links specific prediction failures to the criteria used, enabling granular updates to the rubric's applicability conditions and boundaries.
Empirical Evaluation: A comprehensive evaluation across three downstream applications:
- Comment Prediction: Testing if refined rubrics improve the accuracy of generating comments for held-out artifacts.
- Rubric Understanding: Qualitatively inspecting the learned rubrics and quantitatively evaluating their coverage and precision against reference rubrics.
- Rubric-Guided Revision: Assessing whether learned rubrics help LLMs iteratively revise artifacts to better satisfy expert criteria.

Experimental Results

The method was evaluated on nine diverse tasks, including real-world review datasets (research proposals, essays) and benchmark settings derived from HealthBench and ExpertLongBench (Bio, Chemical, Cyber, Edu, Health, Material).

Comment Prediction: The full method, utilizing comment-wise refinement, achieved a mean content score of 4.93, outperforming baselines including retrieval-based methods (Top-3 RAG: 3.97) and generation without rubrics (2.90). The use of comment-wise signals provided a consistent improvement over initial rubrics and aggregate refinement signals.
Rubric Understanding: Quantitative evaluation showed that the refined "Best-val" rubrics improved recall (coverage of reference criteria) by an average of +1.17 points across domains compared to initial rubrics, with a slight trade-off in precision. Qualitative analysis confirmed that refinement expanded coverage to include finer-grained patterns (e.g., structural organization, positive feedback) absent in initial hypotheses.
Artifact Revision: In revision tasks, artifacts revised using the Best-val rubric satisfied significantly more reference criteria (average improvement of +4.30) compared to the no-rubric condition (+3.41) and the initial rubric condition (+4.03).

Significance and Claims

The paper claims that inline feedback can be distilled into reusable, context-dependent criteria that serve as effective conditioning signals for LLMs. The significance of this work lies in demonstrating that:

Tacit criteria can be externalized: Expert preferences, often undocumented, can be inferred from accumulated inline comments and organized into explicit rubrics.
Local signals drive global improvement: By using comment-wise mismatch signals rather than global judgments, the method learns rubrics that are grounded in local text spans while generalizing across artifacts.
Versatility: The learned rubrics are not limited to comment generation; they effectively support rubric understanding and guide the automatic revision of artifacts to align with expert standards.

The authors note limitations, including reliance on LLM-based evaluation for comment scoring, the assumption that target quotes are pre-provided (rather than jointly localized), and the use of synthetic benchmark data for rubric-grounded evaluation due to the scarcity of public datasets with both inline comments and reference rubrics.