💬 NLP

SERPO: Self-Evolving Rubric Policy Optimization for Open-Ended Test-Time Reinforcement Learning

SERPO introduces a self-evolving framework for open-ended test-time reinforcement learning that replaces answer voting with a closed-loop system co-optimizing response archives, query-specific rubrics, and policy parameters to generate reliable reward signals and achieve significant performance gains across in-domain and out-of-distribution benchmarks.

Original authors: Jianze Wang, Kunwang Zheng, Ying Liu, Yu Cao, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Jianze Wang, Kunwang Zheng, Ying Liu, Yu Cao, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to write a perfect story, but you have a strange rule: you can never show it the "correct" ending, and you can't hire a human teacher to grade its work. This is the challenge of Test-Time Reinforcement Learning (TTRL). Usually, when computers learn, they are given a clear "right" answer to aim for, like a math problem with a single solution. But in the messy world of open-ended writing—like giving medical advice or explaining complex science—there is no single "right" answer. Two perfect stories can look completely different.

To solve this, scientists have tried a trick called voting. They ask the robot to write ten different versions of a story, then pick the one that most of the other versions agree with. It's like asking a crowd of people to guess the answer to a riddle; if most people say "apple," the robot assumes "apple" is right. But this method breaks down when the answers are creative or nuanced. If ten people give ten different but equally good medical advice stories, the robot might get confused, or worse, it might pick a "popular" answer that is actually missing a crucial safety warning because everyone forgot to mention it. The paper we are about to explore suggests that instead of just counting votes, we need a smarter way to judge quality.

The Problem: When "Most Popular" Isn't "Best"

Imagine you are a judge at a talent show where the contestants are all trying to give the best advice for a sick person. In the old way of doing things (the "voting" method), the judge would look at all the advice, see which one the most contestants gave, and say, "Okay, that's the winner!"

But here's the catch: what if the most popular advice was, "Drink water and rest"? That's good advice, but what if the patient actually has a serious condition that needs a doctor immediately? If every contestant forgot to mention the doctor, the "voting" system would still pick "Drink water" as the winner because it was the most common answer. The robot learns to be popular, not to be safe or accurate. It's like a student who memorizes the most common answers on a test but fails to understand the underlying logic, leading to dangerous mistakes in real life.

The Solution: SERPO, the Self-Evolving Rubric

Enter SERPO (Self-Evolving Rubric Policy Optimization). Think of SERPO not as a judge who counts votes, but as a dynamic rulebook that writes itself while the game is being played.

Instead of asking, "Which answer do most people agree on?", SERPO asks, "What specific rules make one answer better than another?" It does this by creating a closed loop of three things that constantly upgrade each other:

The Evidence Archive (The Good, The Normal, and The Bad):
Every time the robot tries to answer a question, SERPO saves three specific examples: one that turned out to be Good, one that was Normal, and one that was Bad. It doesn't just save the "best" one; it saves the whole spectrum. It's like a coach keeping a video reel of a player's best goal, a mediocre pass, and a terrible mistake, all side-by-side.
The Rubric (The Rulebook):
SERPO looks at the differences between the Good, Normal, and Bad examples and asks, "What exactly made the Good one better?" Maybe the Good one mentioned a specific warning sign that the Bad one missed. SERPO writes a new rule (a "criterion") to catch that difference. It's like a teacher realizing that students keep forgetting to cite their sources, so they add a new rule to the grading sheet: "Must include at least one source."
Crucially, this rulebook isn't static. As the robot gets better, the old rules might become too easy. SERPO throws away the weak rules and invents new, harder ones to keep the robot challenged.
The Policy (The Robot's Brain):
The robot uses these new rules to grade its own future attempts. If it follows the new "source citation" rule, it gets a high score. If it forgets, it gets a low score. It then updates its brain to try harder next time.

How It Works: The "Good-Normal-Bad" Dance

The magic happens in a loop.

Step 1: The robot generates a bunch of answers.
Step 2: SERPO sorts them into a "Good," "Normal," and "Bad" pile.
Step 3: The system looks at the "Good" pile and the "Bad" pile and asks, "What is the exact difference?"
Step 4: It creates a specific rule to spot that difference. For example, if the "Good" medical advice mentioned checking blood pressure and the "Bad" one didn't, the new rule becomes: "Must mention checking blood pressure for pregnancy headaches."
Step 5: The robot is rewarded for following this new rule.
Step 6: The robot tries again, and the cycle repeats.

This is different from voting because it doesn't care if everyone agrees. It cares if the answer is safe and complete. Even if only one robot out of ten figured out the blood pressure check, SERPO sees that difference, writes a rule for it, and teaches the whole group to do it.

The Results: Smarter, Safer, and More Adaptable

The researchers tested this idea on two different types of questions: medical advice (HealthBench) and scientific research questions (ResearchQA). They used two different sizes of robot brains (4 billion and 9 billion parameters).

The results were quite impressive. When the robot learned using SERPO:

On medical questions, it improved its score by up to 20.63 points compared to the starting robot.
On scientific questions, it improved by up to 20.31 points.
Across all six tests they ran, the average score went up by 8.06 points.

But the coolest part wasn't just the score; it was the transfer. When they trained the robot on medical questions and then asked it questions about science (which it had never seen before), it still performed better than the robots trained with the old "voting" method. This suggests that SERPO taught the robot how to think about quality, rather than just memorizing popular answers.

The paper also showed that this method works without needing any human teachers or external "super-judges" during the learning process. The robot teaches itself by comparing its own "Good" and "Bad" attempts.

Why This Matters

This approach changes the game for how AI learns to handle complex, open-ended tasks. Instead of hoping that the "most popular" answer is the right one, SERPO builds a system that constantly refines its own definition of "good." It's like a student who doesn't just memorize the answer key but learns to write their own grading rubric, ensuring they understand the why behind every point.

The authors suggest that this method could help AI systems become safer and more reliable in fields like healthcare, where missing a single detail can have serious consequences. By evolving its own rules, the AI can adapt to new situations and catch subtle errors that a simple voting system would miss. It's a step toward AI that doesn't just follow the crowd, but actually understands what makes a good answer.

Technical Summary: SERPO – Self-Evolving Rubric Policy Optimization

1. Problem Statement

The paper addresses Test-Time Reinforcement Learning (TTRL) for open-ended generation under a strict fixed-set, label-free transductive setting. In this scenario, a deployed language model must self-evolve on a specific set of test prompts without access to reference answers, human feedback, external reward models, or stronger judges.

Existing TTRL methods rely on answer voting (e.g., majority voting or self-consistency) to generate pseudo-labels. While effective for tasks with extractable or canonical answers (e.g., multiple-choice or symbolic reasoning), this approach fails for open-ended generation. In domains like medical advice or scientific research, valid responses may differ significantly in wording, detail, or structure while conveying the same core safety or factual principles. Conversely, majority voting may reinforce shared omissions or reject valid alternatives that simply do not match the most frequent phrasing. The core challenge is constructing a reliable, informative reward signal from the model's own outputs without external supervision, specifically for tasks where no single "correct" answer exists.

2. Methodology: SERPO

SERPO (Self-Evolving Rubric Policy Optimization) introduces a closed-loop framework that co-evolves three components: query-specific response archives, query-specific rubrics, and the shared actor policy. Unlike methods that use fixed rubrics or external judges, SERPO derives rewards entirely from the model's own evolving outputs.

Core Components

Good–Normal–Bad (G-N-B) Response Evolution:
- For each prompt, SERPO maintains rolling archives of responses categorized as Good, Normal, and Bad.
- Instead of selecting only the absolute best and worst, the system selects a maximally separated ordered triple $(G, N, B)$ based on a separation score that maximizes the distance between the three categories. This ensures the archives contain distinct, ordered evidence of quality differences.
Rubric Evolution:
- A frozen copy of the initial model acts as a rubric generator. It analyzes the contrasts within the G-N-B archives to propose atomic, query-specific criteria (e.g., "checks blood pressure" vs. "mentions rest").
- Utility-based Selection: Criteria are scored based on their ability to discriminate between the G-N-B archives (variance in scores and agreement with the G > N > B order).
- Dynamic Maintenance: Low-utility criteria are eliminated, and new discriminative criteria are admitted. This process ensures the rubric evolves to capture the specific nuances of the current policy's performance.
Probabilistic Reward Interface:
- A frozen probabilistic judge evaluates responses against the active rubric. Instead of hard binary verdicts, it outputs verdict-token probabilities (likelihood of "Pass" vs. "Fail").
- These probabilities are converted into oriented criterion scores (where higher is always better, regardless of whether the criterion is positive or negative).
- Calibration: Scores are normalized based on the mean performance of the Good and Bad archives to ensure the reward signal remains sensitive as the policy improves.
Policy Optimization:
- The actor is updated using Group-Relative Policy Optimization (GRPO). The scalar rewards derived from the evolving rubric are used to calculate advantages, optimizing the policy to maximize the satisfaction of the query-specific criteria.
- New actor rollouts then refresh the G-N-B archives and trigger further rubric evolution, closing the loop.

3. Key Contributions

Formulation of Label-Free Open-Ended TTRL: The paper defines a setting where TTRL operates without reference answers or external judges, addressing the limitations of answer-voting methods in open-ended domains.
SERPO Framework: A novel closed-loop system that co-evolves response archives, query-specific rubrics, and policy parameters. It is the first to combine post-reasoning Boolean verdict probabilities with evolving, query-specific rubrics for policy optimization.
Probabilistic Reward Interface: A mechanism that converts criterion-satisfaction probabilities into group-relative rewards, preserving graded confidence and avoiding the reward ties caused by hard binary verdicts.
Empirical Validation: Comprehensive experiments demonstrating that SERPO achieves significant gains in in-domain performance and transfers effectively to out-of-distribution (OOD) benchmarks without external supervision.

4. Experimental Results

The authors evaluated SERPO on two model configurations (Qwen3-4B and Qwen3.5-9B) across six benchmarks: two in-domain (HealthBench, ResearchQA) and four OOD (MedQA, LLMEval-Med, GPQA-Diamond, RaR-Science).

In-Domain Performance: SERPO significantly outperforms the base models and other baselines (including voting-based TTRL and static rubric methods).
- On HealthBench, SERPO improved the Qwen3.5-9B base model by 20.63 points.
- On ResearchQA, it improved the base model by 20.31 points.
- The six-benchmark macro-average increased by up to 8.06 points over the base model.
OOD Transfer: Policies evolved on HealthBench transferred effectively to medical OOD benchmarks (MedQA, LLMEval-Med), and ResearchQA-evolved policies transferred to science OOD benchmarks. SERPO outperformed baselines in all OOD settings, demonstrating that self-evolved criteria generalize better than frequency-based consensus.
Ablation Studies:
- Removing rubric evolution or freezing the G-N-B archives resulted in significant performance drops (~12–13%), confirming the necessity of both evolving definitions and up-to-date evidence.
- Using probabilistic judges (soft scores) was crucial; hard verdicts reduced performance by 6% due to reward ties.
- Actor updates were the largest driver of improvement; freezing the actor while evolving only the rubric yielded minimal gains.
Long-Horizon Evolution: Extending training to 45 epochs showed continued improvement, suggesting the method does not saturate quickly. Sequential evolution across benchmarks (HealthBench $\to$ ResearchQA) demonstrated that SERPO can learn new domains while retaining and improving performance on previous ones.

5. Significance and Claims

The paper claims that SERPO represents a significant step toward autonomous, label-free self-improvement for language models in complex, open-ended domains.

Beyond Voting: It demonstrates that answer voting is insufficient for open-ended tasks and that criterion-level verification is a necessary alternative.
Self-Sufficiency: SERPO achieves strong performance without external reward models, human feedback, or privileged reference answers, relying solely on the model's internal capacity to distinguish quality via evolving rubrics.
Generalization: The ability to transfer learned criteria to unseen benchmarks suggests that the method learns generalizable principles of quality (e.g., safety, completeness) rather than overfitting to specific prompt patterns.
Future Direction: The authors position SERPO as a foundation for continual, label-free improvement, proposing future work on cyclic benchmark streams and mechanisms to manage saturation and forgetting.

The paper concludes that by closing the loop between response evidence, discriminative criteria, and policy optimization, SERPO enables models to self-evolve effectively even in the absence of external supervision.