Technical Summary: RuleChef

Problem Statement

While early Natural Language Processing (NLP) systems relied on transparent, hand-crafted rules, the field shifted toward feature-based classifiers and, more recently, large language models (LLMs) and pretrained transformers. While these modern approaches offer superior coverage and flexibility, they operate as "black boxes" with decision logic embedded in latent parameters, making them difficult to audit, explain, or edit directly. This lack of interpretability is a critical barrier in sensitive, highly regulated domains such as medical, legal, and financial NLP, where predictions must be traceable to explicit patterns to be certified.

Conversely, explicit rules remain attractive for many real-world tasks due to their low inference cost, determinism, and ease of inspection. However, the primary bottleneck for rule-based systems is not accuracy, but the labor-intensive nature of writing and maintaining them manually, which requires substantial domain expertise.

Methodology: The RuleChef Framework

RuleChef is a framework designed to bridge this gap by using LLMs exclusively at learning time to synthesize executable symbolic rules, while keeping inference time deterministic and rule-based. The system separates the generation of rules from their execution, ensuring the final model is fast, inspectable, and editable.

Core Components

1. Rule Synthesis:

   • The system takes a task specification (e.g., Named Entity Recognition, text classification) and supervision signals (labeled examples, corrections, free-text feedback, or observed model behavior).
   • It structures prompts to include task definitions, sampled training examples, data evidence (including regex hints from tools like grex), and format instructions.
   • The LLM generates rules in formats such as regular expressions (regex), spaCy token patterns, or Python code. The paper's evaluation focuses primarily on regex.
   • Generated rules are validated for compilation and schema adherence before acceptance.

2. Iterative Refinement Loop:

   • After initial synthesis, the system evaluates the rule set on a held-out development split.
   • Failures are clustered by "failure modes" (e.g., missed spans, spurious spans, wrong types).
   • The LLM is prompted to generate "patches" (new or modified rules) targeting these specific failure clusters.
   • Crucial Constraint: A patch is accepted only if it improves the quality (F1 score) on the held-out development set. This prevents overfitting and memorization of training data.
   • The loop can be driven by a fixed schedule or an Agentic Coordinator, which analyzes per-class metrics to steer subsequent patches, run periodic critics (to provide rule-level feedback), and perform audits (to merge redundant rules or remove ineffective ones).

3. Conflict Resolution and Pruning:

   • When multiple rules overlap, the system executes the rule with the highest priority. Ties are broken based on the precision achieved on the held-out development set.
   • Precision estimates are adjusted using a Wilson lower bound to discount rules with low support (e.g., a rule correct 1/1 time is not ranked higher than one correct 95/100 times).
   • A leave-one-out pass measures the marginal contribution of each rule to the ensemble F1, dropping rules that do not hurt performance when removed.

4. Observation Mode:

   • For scenarios where an LLM already handles production traffic, RuleChef can bootstrap rules without manual labels. It treats LLM input-output pairs as training examples, periodically synthesizing rules from these observations.
   • A hybrid deployment strategy is supported: high-confidence rules handle traffic, while uncertain cases are delegated back to the LLM.

Key Contributions

• A Hybrid Learning Framework: A system that synthesizes executable rules from supervision signals (examples, corrections, feedback, or model behavior), strictly separating LLM usage during learning from deterministic execution during inference.
• A Refinement Loop: An iterative process that improves rule systems by measuring changes on held-out data, resolving conflicts via precision, and preventing memorization through strict acceptance criteria.
• Human-in-the-Loop (HITL) Integration: The ability for domain experts to review rules and provide explicit feedback, which is used to prompt the LLM for targeted updates.
• Unsupervised Observation: A mode that learns rules from the behavior of an external black-box model, enabling task discovery and rule generation without initial human labeling.
• Open-Source Release: The framework is released under an Apache 2.0 license.

Experimental Results

The authors evaluated RuleChef on two tasks: Text Anonymization (TAB) for Named Entity Recognition (NER) and Banking77 for intent classification.

TAB (Named Entity Recognition)

• Performance on Format-based Entities: On entity types defined by surface form (e.g., dates, case numbers, quantities), RuleChef-generated rules significantly outperformed direct LLM prompting and GLiNER2 baselines in F1 score, achieving high precision (e.g., 89.1% on format entities).
• Performance on Semantic Entities: On semantic types (e.g., Person, Organization), the rule-based approach achieved lower recall than direct LLM prompting but maintained higher precision than GLiNER2 baselines. The authors note this is intended behavior; high-precision rules form a foundation that can be incrementally expanded.
• Inference Efficiency: RuleChef achieved inference latencies of approximately 0.6 ms per document, compared to ~1500 ms for LLM prompting and ~190 ms for GLiNER2.
• Comparison to Neural Baselines: On the official full-document test set, the 22 learned rules outperformed an off-the-shelf RoBERTa NER system by nearly 30 precision points, though they remained below a fine-tuned Longformer system.

Banking77 (Intent Classification)

• Few-Shot vs. Rules: In a 5-way intent classification task, RuleChef's learned rules achieved precision comparable to few-shot LLM prompting and supervised baselines (~97.6% vs. 98.0%), though recall was lower (61.0% vs. 98.0%), a typical trade-off for rule-based systems.
• Observation Mode: When learning from an external LLM's outputs without ground-truth labels, precision increased from 92.7% to 96.0% as the number of observed examples grew from 10 to 50. Recall improved from 19.0% to 49.5%. The authors suggest such a system could serve as a transparent first tier in a hybrid architecture.

Ablation and Repair

• Iterative Learning: A one-shot rule generation prompt was nearly useless for semantic entity types but effective for format types. The iterative refinement loop, particularly with "holdout acceptance," was essential for generalization.
• Human Feedback: A single round of human feedback correcting specific rule defects (e.g., a rule matching fraction-shaped case numbers as quantities) improved the F1 score for the "QUANTITY" class from 5.7 to 35.6, demonstrating the system's responsiveness to expert correction.

Significance and Claims

The paper claims that RuleChef demonstrates the viability of generating fully transparent, inspectable, and editable rule-based systems that are competitive with off-the-shelf neural models, particularly for tasks with regular surface patterns.

The authors emphasize that while the results are preliminary, the framework offers a practical pathway for:

1. Regulatory Compliance: Providing certified, traceable predictions in high-stakes domains.
2. Cost Reduction: Drastically lowering inference costs by replacing LLM calls with regex execution.
3. Hybrid Systems: Serving as a high-precision first layer in cascaded systems, reducing the load on expensive black-box models.

The paper acknowledges limitations, including the reliance on a single LLM family (Kimi-K2) for rule generation, the non-determinism inherent in iterative prompting, and the need for further experimentation across diverse domains and languages. It positions RuleChef not as a replacement for neural models in all contexts, but as a specialized tool for scenarios where explainability, auditability, and deterministic execution are paramount.