🤖 machine learning

OCRR: A Benchmark for Online Correction Recovery under Distribution Shift

This paper introduces OCRR, a novel benchmark for evaluating online correction recovery under distribution shift, and demonstrates that a proposed hash-chained append-only substrate significantly outperforms existing continual-learning and fine-tuning baselines by simultaneously achieving high accuracy on new categories while retaining performance on original data with minimal memory overhead.

Original authors: Adrian Grassi

Published 2026-05-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Adrian Grassi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Tuesday Retrain" Trap

Imagine you work at a customer service desk. A customer asks a question, and your computer system guesses the wrong answer.

The Old Way: You fix the mistake in your head, but the computer doesn't learn. You have to wait until "Tuesday's retrain" (hours or days later) to update the system. In the meantime, the computer keeps making the same mistake for everyone else.
The Goal: You want a system that learns immediately when you correct it, without forgetting how to do everything else it already knew.

The authors of this paper say: "We have no way to measure how fast a computer system learns from these immediate corrections." Existing tests only check how smart a computer is before it goes to work, not how well it adapts while it's working.

The Solution: OCRR (The "Live Correction" Test)

The authors created a new test called OCRR (Online Correction Recovery Rate).

The Analogy:
Imagine a librarian (the AI) who knows 67 types of books perfectly. One day, a customer brings in a brand new type of book the librarian has never seen (a "novel class").

The librarian guesses wrong.
The customer says, "No, that's actually a Sci-Fi book."
The librarian must instantly learn this new category and remember it, while not forgetting how to sort the original 67 categories.

The OCRR test streams thousands of these "mistake-and-correction" moments at the librarian and measures two things:

Novel Accuracy: Did the librarian learn the new book types?
Original Accuracy: Did the librarian forget how to sort the old book types?

The Contenders: Who Played the Game?

The authors tested 9 different "librarian strategies" (algorithms) against their own new strategy, called the Substrate.

The "Static" Librarians: They memorized the 67 books and refused to change. (They failed the test because they couldn't learn new things).
The "Gradient" Librarians (EWC, A-GEM, LwF, River): These try to rewrite their internal rulebook slightly every time they get a correction.
- The Problem: When they try to learn the new book, they accidentally erase the rules for the old books. This is called "Catastrophic Forgetting." It's like trying to write a new chapter in a book by erasing the old pages to make room.
The "LoRA" Librarian: This is a very smart librarian with a massive brain (1.5 billion parameters) who uses a special "fine-tuning" technique.
- The Surprise: Even with a huge brain, this librarian forgot the old rules almost completely (dropping from 67% accuracy to 10%) when trying to learn the new stuff. The authors found that "bigger brains" don't necessarily solve the forgetting problem if they try to rewrite their core logic on the fly.
The "Retrieval" Librarian (kNN-LM): This librarian keeps a physical card catalog. When they see a new book, they just add a new card to the shelf. They don't rewrite their brain; they just look up the card.
The "Substrate" (The Winner): This is the authors' new strategy. It's a digital ledger (like a blockchain) that acts as a permanent, unchangeable record.
- How it works: When a mistake happens, the librarian doesn't rewrite their brain. They simply append (add) a new note to the end of a long, secure chain of notes.
- The Voting System: When a new question comes in, the librarian looks at the 5 most similar notes in the chain and lets them "vote" on the answer. If 3 out of 5 say "Sci-Fi," the answer is "Sci-Fi."

The Results: Why the Substrate Won

The paper claims the Substrate is the only system that did both things perfectly:

It learned the new book types quickly (88.7% accuracy).
It never forgot the old book types (95.4% accuracy).

The Key Comparisons:

Vs. The "Gradient" Librarians: The Substrate was 32.6 percentage points better at learning new things without forgetting old ones, using the same amount of memory.
Vs. The "LoRA" Giant: The Substrate was 84.6 percentage points better at remembering the old stuff. The giant brain forgot almost everything; the simple ledger remembered everything.
The "Sparse" Test: Even if the librarian only got corrected 1 out of 10 times (a very noisy environment), the Substrate still learned, while the others gave up.

The "Magic" of the Substrate

The authors found two surprising things about their winning strategy:

It's Fast: Because it just adds a note to a list rather than rewriting a complex brain, it is 100 times faster per correction than the other methods.
It's Robust to "Bad Search":
- The Analogy: Imagine the librarian has 10 million notes. Finding the exact top 5 most similar notes is hard and slow. Usually, if you use a "fast but fuzzy" search (Approximate Nearest Neighbors), you might grab the wrong 5 notes.
- The Result: Even when the search was fuzzy and only found 23% of the perfect matches, the Substrate's voting system still got the answer right 99% of the time. The "majority vote" of the notes was strong enough to ignore the noise.

The Trade-Off: Storage vs. Speed

The paper admits a catch: The Substrate needs to keep a record of every correction.

Unbounded: If you keep everything, you never forget.
Bounded: If you have a limit (e.g., only keep the last 1,000 notes), you start to forget the very old stuff.
The Finding: Even with a limit of just 5,000 notes, the Substrate was still the best performer, beating all the other complex algorithms.

Summary

The paper argues that for real-world systems that need to learn from mistakes right now, the best approach isn't to try to "rewire" a giant AI brain. Instead, it's better to keep a secure, append-only log of corrections and let a simple voting system decide the answer based on that log. This method learns fast, forgets nothing, and is incredibly robust, even when the search for information isn't perfect.

Technical Summary: OCRR – A Benchmark for Online Correction Recovery under Distribution Shift

1. Problem Statement

Deployed machine learning systems face a critical gap between static benchmark performance and real-world operational needs. While standard benchmarks (e.g., Banking77, GLUE, MMLU) measure a model's performance when frozen at training time, production systems must handle distribution shifts (new categories, paraphrased queries, data drift) and recover from errors via online user corrections.

Current evaluation paradigms fail to capture correction recovery rate:

Static Benchmarks: Measure accuracy on a frozen test split, ignoring the ability to absorb new information without retraining.
Continual Learning (CL): Evaluates models after a full sequence of tasks is completed, not during a live stream of user corrections.
Online Learning: Typically tests cumulative accuracy on a stream but often assumes no historical data storage or lacks held-out class scenarios.

The paper argues that a model achieving 95% static accuracy but failing to absorb a single correction without retraining is operationally inferior to a 90% model that can recover online. There is currently no benchmark to measure how quickly a classifier recovers when corrections arrive in a stream.

2. Methodology: The OCRR Benchmark

The authors introduce OCRR (Online Correction Recovery Rate), a benchmark designed to evaluate systems under streaming constraints with distribution shifts.

3.1 Core Protocol

Held-Out Class Shift: The system is initialized on $C-H$ known classes. A stream of queries from $H$ held-out classes (unknown at initialization) is introduced.
Correction Stream: The system predicts labels. If a prediction is wrong, a policy determines if a correction is applied.
Correction Policies:
- Oracle: Every wrong prediction is corrected.
- Random-50% / Random-10%: Wrong predictions are corrected with probability 0.5 or 0.1, simulating sparse supervision.
Evaluation Axes:
- Novel Accuracy: Performance on the held-out classes (recovery capability).
- Original Accuracy: Performance on the initial known classes (forgetting check).
Metrics: Novel/Original accuracy vs. correction count, and "Corrections-to-N%" (speed of recovery).

3.2 Systems Evaluated

The benchmark evaluates a proposed Substrate against nine baselines across five families:

Strawmen: Static-kNN, Static-linear, Online-linear (SGD).
Continual Learning (CL): EWC (Fisher-weighted penalty), A-GEM (gradient projection with memory), LwF (knowledge distillation).
Online ML Library: river (Logistic Regression).
Retrieval/Parametric Hybrid: kNN-LM (interpolating parametric head with growing datastore).
Parameter-Efficient Fine-Tuning (PEFT): LoRA on DeBERTa-v3-large (1.5B parameters).

3.3 The Proposed Substrate

The Substrate is an append-only ledger with cryptographic hash chaining.

Mechanism: It stores embeddings, labels, and hashes. It does not update parameters; instead, it appends new entries for corrections.
Prediction: Retrieves $k=5$ nearest neighbors via cosine similarity.
Voting: Uses a margin-band majority vote (neighbors within 0.05 cosine of the top hit vote) with max-similarity and recency tiebreaks.
Integrity: The hash chain allows verification that no past entries were tampered with, supporting audit trails.
Bounded Variants: The authors test bounded versions using reservoir sampling and FIFO eviction to probe the storage-vs-recovery Pareto frontier.

3. Key Results

Experiments were conducted on Banking77 (77 classes) and CLINC150 (151 classes) with 3 random seeds.

3.1 Pareto Dominance

The Substrate is the only system that simultaneously achieves high novel-class accuracy and high original-distribution accuracy.

Performance: On Banking77 (Oracle policy), the unbounded Substrate achieved 88.7% ± 2.9% novel accuracy and 95.4% ± 0.8% original accuracy.
Comparison: It outperformed the next-best continual-learning baseline (A-GEM) by 32.6 percentage points on novel accuracy at an equal memory budget (1000 entries).
Retention: It beat LoRA-on-DeBERTa-v3-large by 84.6 percentage points on retention (original accuracy), demonstrating that large parameter counts do not necessarily prevent catastrophic forgetting in per-correction online settings.

3.2 Sparse Correction Policies

Under sparse correction (Random-10%), parametric baselines (EWC, A-GEM, LwF, LoRA) collapsed to near 0% novel accuracy. The Substrate maintained ~65% novel accuracy while retaining original accuracy.

3.3 Scale and Retrieval Robustness

A critical finding concerns scalability. The authors tested the Substrate with Approximate Nearest Neighbors (HNSW) on corpus scales up to 10 million entries.

Recall Degradation: HNSW recall@5 dropped from 0.69 (at 10k entries) to 0.23 (at 10M entries).
Accuracy Stability: Despite the poor retrieval recall, the Substrate's classification accuracy remained stable at 99%.
Implication: The margin-band majority vote mechanism is robust to retrieval imperfections, absorbing noise that pure top-k metrics would predict as failure.

3.4 Compute Efficiency

The Substrate is two orders of magnitude faster per correction than gradient-based methods (~10 µs for ledger append vs. ~50 ms for LoRA backpropagation on a 1.5B model).

4. Significance and Claims

The paper positions OCRR as the first benchmark to directly measure correction recovery rate under online distribution shift.

Honest Trade-offs: Unlike benchmarks that enforce strict "no storage" constraints (excluding retrieval methods) or ignore storage costs, OCRR explicitly reports memory footprints and maps systems on a storage-vs-recovery Pareto frontier.
Non-Parametric Advantage: The results suggest that retrieval-based, non-parametric learning (the Substrate) is significantly more sample-efficient for absorbing new classes via corrections than gradient-based methods, which suffer from catastrophic forgetting or slow adaptation.
Production Relevance: The Substrate's design supports compliance-heavy industries (banking, healthcare) via its hash-chained audit trail, proving data integrity post-correction.
Limitations: The authors acknowledge OCRR currently measures only categorical shifts (new classes), not within-class drift (paraphrasing) or noisy labels, and relies on English-only datasets.

In conclusion, the paper demonstrates that for online correction recovery, a simple, append-only retrieval system with robust voting can outperform complex, parameter-heavy continual learning and fine-tuning approaches, offering a superior balance of speed, accuracy, and data integrity.