Technical Summary: CDR-Bench

Problem Statement

Data refinement is a critical component of modern Large Language Model (LLM) pipelines, transforming noisy raw text into clean, task-ready data for applications ranging from pre-training corpus construction to Retrieval-Augmented Generation (RAG) and privacy-preserving deployments. While traditional pipelines rely on heuristic rules and hand-crafted scripts, the rise of LLMs has introduced the potential for flexible, natural language-driven refinement.

However, a significant gap exists in evaluating whether LLMs can faithfully execute these refinement tasks. Real-world data refinement is rarely a single, isolated operation; it involves compositional and order-sensitive pipelines where multiple operators (mappers and filters) are applied sequentially to an evolving text state. Existing benchmarks either focus on isolated text editing (ignoring composition) or entangle text processing with code generation and tool execution (obscuring the model's intrinsic procedural capability). Consequently, it remains unclear if LLMs can directly and faithfully execute complex, multi-step refinement recipes, particularly when the order of operations dictates the final outcome.

Methodology: CDR-Bench

To address this gap, the authors introduce CDR-Bench, a comprehensive benchmark designed to evaluate the faithful execution of compositional, order-sensitive data refinement recipes.

Benchmark Construction

CDR-Bench comprises 3,462 high-quality tasks spanning four real-world domains:

1. Web Refinement (WR): Cleaning crawled web text (e.g., removing HTML, normalizing whitespace).
2. LaTeX Refinement (LR): Processing scientific documents (e.g., expanding macros, removing comments).
3. RAG Preparation (RP): Preparing text for retrieval (e.g., removing repeated sentences, fixing Unicode).
4. Privacy Redaction (PR): Removing personally identifiable information (PII) like emails, IPs, and secrets.

The benchmark utilizes 29 distinct operators (mappers for transformation and filters for KEEP/DROP decisions) organized into 63 recipe templates. The construction pipeline involves:

• Data Collection & Activation: Sourcing heterogeneous corpora and identifying records where specific operators produce non-trivial changes.
• Recipe Mining: Mining frequent operator co-occurrence patterns to create realistic recipe families.
• Track Materialization: Creating three distinct evaluation tracks:
  • Atomic: Single operator execution (baseline).
  • Order-Agnostic (Agnostic-M): Multi-step recipes where order does not affect the outcome.
  • Order-Sensitive: Recipes where the execution order fundamentally changes the result. This includes Order-M (permuting mapper orders) and Order-F (inserting filters at Pre, Mid, or Post positions relative to mappers).
• Deterministic References: All tasks are evaluated against ground-truth outputs generated by deterministic operator execution, avoiding reliance on LLM-as-a-judge paradigms.

Evaluation Metrics

Standard metrics like exact match or edit distance are insufficient for this task. The authors propose three specific metrics:

1. Recipe Success (RS): Measures exact match between the model's output and the normalized reference text, including the correct KEEP/DROP decision.
2. Order-Consistent Success (OCS): Evaluates order-sensitive groups, requiring the model to correctly solve every variant within a group (e.g., all permutations of a recipe) simultaneously.
3. Refinement Gain (RG): Captures partial progress by measuring the reduction in edit distance toward the reference, even if the exact output is not achieved.

Key Results

Experiments were conducted on over 10 state-of-the-art LLMs (including GPT-5.4, Claude Opus 4.5/4.6, Qwen3.6, DeepSeek-V4, and Llama-4) under non-thinking modes.

1. Atomic-to-Recipe Degradation

Models exhibit a sharp performance drop when moving from atomic to compositional settings.

• GPT-5.4 drops from 30.30% RS on Atomic-M to 22.74% on Agnostic-M, and further collapses to 13.99% on Order-M.
• Similar trends are observed across all models, indicating that procedural composition is a major bottleneck independent of model size or architecture.

2. Order Sensitivity and Faithfulness

Order-sensitive settings expose a critical failure mode where models produce plausible-looking outputs but violate the intended execution order.

• Order-Consistent Success (OCS) is significantly lower than individual Recipe Success. For GPT-5.4, while RS@3 on Order-M is 13.99%, the OCS@3 (solving all permutations correctly) is only 4.90%.
• Deferred Filtering: Filtering decisions degrade drastically when applied after transformations. GPT-5.4's success rate on filter placement drops from 62.04% (Pre-filter) to 25.03% (Mid) and 14.97% (Post).

3. Impact of Prompting Strategies

• Few-Shot and Plan-First strategies offer marginal or inconsistent improvements.
• State-Aware prompting (explicitly identifying intermediate text states) yields the most consistent gains, improving Order-F OCS by up to 9 percentage points for some models. However, absolute performance remains low, suggesting prompt engineering alone cannot resolve the fundamental limitation.
• Thinking Mode: Enabling thinking/reasoning modes substantially improves performance, particularly on Order-F tasks (e.g., +19.6 pp for Qwen3.6-35B-A3B), highlighting the benefit of explicit multi-step reasoning for filter evaluation.

4. Real-World Semantic Validation

To ensure these findings are not artifacts of rule-based operators, the authors evaluated four semantic domains (PII redaction, hallucination processing, safety tagging, rubric scoring) with human-annotated references.

• The atomic-to-compositional gap persists, averaging 28.0 percentage points across domains.
• The severity of the gap varies by task structure: Rubric scoring shows the largest degradation (up to 61.1 pp), while hallucination processing shows a smaller gap because the subtasks form a coarse-to-fine hierarchy where the finest steps are already the bottleneck.

Significance and Claims

The paper claims that CDR-Bench reveals a fundamental limitation in current LLMs: procedural faithfulness.

• Procedural Faithfulness: Current models lack the ability to reliably execute a latent sequence of operations in the correct order. They often generate superficially plausible outputs while silently violating execution constraints.
• Beyond Surface Editing: Progress in data refinement requires a shift from surface-level text editing to genuine procedural faithfulness.
• Benchmark Utility: CDR-Bench provides a reproducible testbed to measure this specific capability, isolating it from code generation and tool use.
• Generalizability: The compositional bottleneck is not limited to deterministic rule-based operators but extends to higher-level semantic reasoning tasks, suggesting a structural limitation in how LLMs handle multi-step, state-dependent workflows.

The authors conclude that while LLMs show promise in individual operations, their reliability in complex, compositional data refinement pipelines remains insufficient for deployment without significant architectural or training advancements focused on state tracking and procedural execution.