Technical Summary: SIFT (Self-Improving, Frozen-gate Training)

1. Problem Statement

The paper identifies that while document classification is technically "solved" in laboratory settings, it remains an unsolved challenge in enterprise production. The primary blockers are not model architecture or accuracy, but rather:

1. The Cold-Start Labeling Project: Deploying a supervised classifier requires a labeled corpus, necessitating a time-consuming and expensive project involving scarce domain experts before the first prediction is served.
2. The Fear of Autonomous Retraining: Enterprises hesitate to allow models to retrain themselves because an automated retraining cycle can silently regress performance on critical classes while improving aggregate accuracy. This leads to models being "frozen" at launch weights, where they eventually decay due to data drift (e.g., shifting document mixes, new sub-types, or OCR changes).

Existing solutions like weak supervision, active learning, and LLM-as-judge approaches address parts of the problem but fail to provide a complete recipe for safe, autonomous, continuous learning without human intervention in every cycle.

2. Methodology: The SIFT Architecture

SIFT (Self-Improving, Frozen-gate Training) is a dynamic classifier service designed to break this standoff by creating a self-feeding loop and a safety mechanism for autonomous updates.

2.1 Cost-Tiered Serving (Model Cascade)

SIFT employs a "cheap model first, expensive model only when needed" architecture:

• The Cheap Model: A CPU-bound pipeline consisting of a SPLADE sparse encoder (generating lexical features based on anchor phrases, regex hits, and length) feeding into a LightGBM head. This model serves the majority of requests in milliseconds.
• The LLM Judge: Only pages where the cheap model's confidence falls below a specific threshold are escalated to a Large Language Model (LLM) acting as a judge. The LLM is provided with the domain's label space, glossary, and policy.
• The Twist: Unlike traditional cascades where the expensive model's output is discarded after inference, the LLM judge's verdicts are written back into a labeled corpus.

2.2 The Self-Feeding Corpus

The system creates a continuous loop where production traffic generates training data:

• Write-Back: Low-confidence pages escalated to the LLM are labeled by the judge (if they meet a minimum confidence bar) and added to the corpus as labeled rows.
• Bootstrap: For new domains with zero labeled data, a bootstrap path runs the LLM judge over raw, unlabeled pages to generate an initial corpus and train a first model.
• Active Learning: Human reviewers are integrated via a queue that surfaces the least-confident, judge-labeled rows. This ensures scarce expert attention is focused on the hardest edge cases, correcting labels to serve as ground truth.
• Economic Effect: As the cheap model learns from the judge's feedback, the escalation rate decreases, causing the marginal cost of labeling to trend toward zero.

2.3 Domains as Declarative Bundles

To eliminate the need for code changes or new labeling projects for new document families, SIFT treats domains as data. A "bundle" is a declarative YAML configuration containing:

• A label space.
• Anchor phrases for the sparse encoder.
• Optional regex packs.
• A glossary and policy block for the LLM judge.
• A list of "critical labels."

2.4 The Frozen-Gate Promotion Mechanism

To enable safe autonomous retraining, SIFT introduces a two-part "promote gate" that vetoes any candidate model that fails specific checks before it can replace the live model:

1. Critical-Label F1 Regression Check: The bundle defines specific classes where misclassification is expensive (e.g., "NDA"). If a candidate model's F1 score on any critical label drops below the current model's score, promotion is blocked.
2. Frozen Golden Regression Set: The bundle includes a small, immutable set of "golden" examples that are never used for training. Every candidate model is scored against this set. A drop in performance on this frozen set (either in overall accuracy or critical-label F1) acts as a hard veto.

This mechanism ensures that a model cannot regress on critical tasks or drift due to the shifting distribution of the training corpus itself.

3. Key Contributions

The paper outlines five primary contributions:

1. Problem Reframing: Shifting the focus of production classification from model accuracy to the real blockers: the up-front labeling project and the unsafety of autonomous retraining.
2. Cost-Tiered Serving: An architecture using SPLADE + LightGBM with LLM escalation, where the escalation rate naturally falls as the corpus grows.
3. Self-Feeding Corpus: A system where judge verdicts and human reviews continuously label raw traffic, enabling a bootstrap path from zero data and an active-learning queue for human-in-the-loop correction.
4. Frozen-Gate Promotion: A safety mechanism combining critical-label F1 checks and a never-trained-on golden set to make autonomous retraining safe and routine.
5. Declarative Onboarding: A bundle-based approach allowing new document families to be onboarded without code changes or labeling projects, with multi-tenant isolation.

4. Results and Illustrative Application

The paper provides an illustrative application involving the classification of legal documents (distinguishing NDAs from supporting pages and miscellaneous content):

• Zero-Label Start: A team created a bundle and used the bootstrap path on hundreds of unlabeled pages. The LLM labeled confident pages, creating a corpus and training a model without human labeling.
• Production Loop: The cheap model handled the majority of traffic. Approximately 15–25% of pages escalated to the LLM.
• Safety in Action: A monthly retrain produced a candidate with improved overall accuracy but a 2-point drop in F1 for the critical "NDA" class. The promote gate blocked the promotion, logging the delta.
• Correction: Humans reviewed the blocked candidate via the active-learning queue, corrected mislabeled "SUPPORTING" pages, and the subsequent candidate passed both gates and auto-promoted.
• Outcome: Over time, the escalation rate decreased as the cheap model absorbed the judge's knowledge, and the team's recurring cost was limited to reviewing occasional vetoed promotions.

5. Significance and Claims

The paper claims that SIFT fundamentally changes the economics of enterprise document classification:

• From Project to Routine: It transforms the "labeling project" from a capital expenditure into a continuous, near-zero marginal cost operation driven by production traffic.
• From Reckless to Routine: It converts "retraining monthly without a human" from a risky proposition into a safe, governed default configuration.
• The Teacher-Student Dynamic: The LLM is positioned not as the primary classifier, but as a "teacher" deployed only on lessons the "student" (the cheap model) has not yet learned, progressively idling as the student improves.
• Governance: The system is designed to be auditable and scoped, making self-improving classifiers deployable in regulated environments where safety and drift detection are paramount.

6. Limitations

The authors acknowledge several constraints:

• Text Only: The system currently relies on OCR-derived text and does not yet utilize image embeddings for visual layout.
• Judge Bias: The system trusts the LLM judge above a confidence bar; a systematically biased judge could teach the cheap model its bias, though the frozen set and active learning provide checks.
• Frozen Set Representativeness: The golden set is a fixed yardstick; it cannot veto regressions on genuinely novel sub-types that were not present in the original set.
• Bundle Dependency: The quality of the cold-start depends heavily on the quality of the anchor phrases and policies supplied in the bundle.
• Operational Footprint: The sparse encoder is the dominant resident cost, and large-scale deployments require shared artifact stores and queues currently on the roadmap.