💬 NLP

UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

UltraX is a function-calling framework that enhances large-scale pre-training data refinement by introducing adaptive programmatic editing with insertion, deletion, and modification capabilities, thereby overcoming the efficiency and reliability limitations of existing rule-based and LLM-based approaches to achieve superior data efficiency and model performance.

Original authors: Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, Zhiyuan Liu

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xinlong Zhao, Dongsheng Liu, Hengyu Zhao, Zixuan Fu, Zheng Wang, Jie Cai, Jie Zhou, Qiang Ma, Xuanhe Zhou, Xu Han, Yudong Wang, Zhiyuan Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to teach a super-smart robot how to speak human. For a long time, the secret sauce was just more data. You'd throw a mountain of books, websites, and articles at the robot, hoping it would learn everything by sheer volume. But now, we've hit a wall. We've scraped almost every useful bit of text off the internet. The "more is better" rule is starting to fade, like a battery running low.

So, what do we do? We stop hoarding and start cleaning.

Enter UltraX, a new tool designed to act like a microscopic, super-fast editor for the robot's training books. But here's the twist: instead of just deleting bad sentences or rewriting whole paragraphs (which is slow and risky), UltraX uses a clever trick called function calling.

The "Lego Master" vs. The "Demolition Crew"

Think of the old ways of cleaning data like a demolition crew.

Rule-based methods are like workers with a hammer and a checklist. They smash anything that looks like an ad or a weird symbol. But they're clumsy; sometimes they knock down a beautiful sculpture (valuable info) just because it looked a bit like a pile of bricks.
Big AI editors are like artists who rewrite the whole story to make it perfect. But they're slow, expensive, and sometimes they change the plot so much the story doesn't make sense anymore.

UltraX is different. It's like a Lego master with a specific set of tools. Instead of rewriting the whole book, it gives the robot a tiny instruction list:

Delete this specific line of garbage (like an ad).
Swap this weird typo for the right word.
Insert a missing piece of information that got lost in the mess.

The paper explicitly argues against the idea that you need to rewrite entire texts to fix them. It shows that just deleting things (like some other tools do) isn't enough because you might accidentally throw away good stuff. It also argues that trying to generate whole new paragraphs is too slow and unreliable for the massive scale of data needed to train these robots. UltraX proves that precise, surgical edits are the key.

How It Works: The "Recipe" and the "Chef"

The process happens in two main acts:

Act 1: The Training (Teaching the Chef)
First, the researchers needed to teach their small "refiner" model how to be a good editor. They didn't just guess; they used a "smart chef" (a powerful AI) to write perfect, clean versions of messy web pages. Then, they used a special mapping trick to turn those clean versions into a recipe of instructions (like "remove line 5," "fix word 12"). They filtered out the confusing recipes and taught their small model to follow these instructions perfectly.

Act 2: The Cleanup (The Scale-Up)
Now, they take this trained "refiner" and let it loose on billions of web pages. It reads a chunk of text, predicts the exact list of Lego moves needed to fix it, and then a computer program executes those moves instantly. Because the model only has to output a short list of commands (like "delete line 3") instead of writing a whole new story, it's incredibly fast and doesn't get tired.

The Results: Faster, Smarter, and More Efficient

The researchers tested this by training a 1-billion-parameter robot from scratch using different types of cleaned data. Here is what they found:

Better Scores: On a test of 10 different skills (like answering science questions or understanding jokes), the robot trained with UltraX data scored higher than robots trained with raw data or data cleaned by other methods. In fact, UltraX was the top performer on all five different types of internet data they tried.
The "2%" Boost: On several datasets, UltraX gave a relative improvement of over 2%. In the world of AI, that's a huge jump.
Doing More with Less: This is the coolest part. The UltraX-trained robot reached the same high performance level using fewer training tokens (fewer words) than the others. It suggests that UltraX makes the data "denser" with useful information, so the robot learns faster.

What They Didn't Do (And What They're Not Sure About)

It's important to know the limits of this story. The paper does not claim this solves everything forever.

They only tested this on English web data. They admit they haven't tried it on Chinese or other languages yet, where the "noise" might look totally different.
They trained a 1-billion-parameter model. They haven't proven yet if this works exactly the same way on the massive, trillion-parameter models that are coming soon.
They suggest that the gains come from balancing noise removal with keeping good info, but they don't claim to have a perfect formula for every single type of internet mess.

The Bottom Line

UltraX suggests that the future of teaching AI isn't about finding more data, but about being smarter with the data we already have. By using a precise, instruction-based approach that can delete, fix, and insert specific parts of text, it cleans up the robot's library faster and better than the old ways. It's a shift from "throwing everything at the wall" to "surgically removing the trash," and the results show that a little bit of smart cleaning goes a long way.

Technical Summary: UltraX

Problem Statement

As the physical limit of available training data approaches, the performance gains derived from Scaling Laws are diminishing. Consequently, the focus for enhancing Large Language Models (LLMs) has shifted from mere data expansion to improving data utilization efficiency through higher quality data. However, existing refinement methodologies for large-scale corpora face significant limitations:

Rule-based approaches are constrained by fixed heuristics, struggling with instance-level variations and often rejecting documents that contain salvageable high-value information.
Model-based approaches (including LLM-based end-to-end refinement) often incur substantial computational costs, suffer from low throughput, and struggle to scale to pre-training corpora. Furthermore, end-to-end generation in long-context scenarios can introduce semantic fragmentation and structural inconsistencies due to difficulties in delineating segmentation points.
Existing programmatic refinement methods (e.g., ProX, RefineX) exhibit incomplete function spaces (lacking insertion capabilities), rely on unreliable seed supervision, and suffer from execution issues such as duplicate string matching and interference between consecutive operations.

The core challenge is how to retain the benefits of model-driven, fine-grained refinement while reliably scaling the process to large pre-training corpora with high efficiency and stability.

Methodology

UltraX is a function-calling refinement framework designed to address these challenges through a two-stage workflow: Refinement Model Construction and Program Execution at Scale.

1. Task Definition and Function Space Design

UltraX formulates data refinement as a programmatic text transformation process. Instead of generating text directly, a lightweight model predicts a structured sequence of function calls ( $Z$ ) which a deterministic executor applies to the original document ( $d$ ) to produce the refined document ( $\hat{d}$ ).

To overcome the limitations of prior work, UltraX introduces a complete editing function space ( $\Omega$ ) that includes:

Deletion: remove_lines (for noise like ads, navigation, templates) and remove_all (for valueless documents).
Modification: replace_str (for localized string replacement, e.g., fixing HTML remnants or inline noise).
Insertion: add_line (to supplement missing structure or restore semantic content).
Preservation: keep_all (for documents requiring no changes).

Crucially, operations are anchored by line numbers and local strings rather than raw character indices, facilitating execution by lightweight models while ensuring auditability.

2. Program-Supervision Generation Pipeline

To train the refinement model, UltraX constructs a reliable seed supervision pipeline:

Dataset-Adaptive Prompt Optimization: An automatic agent profiles each dataset's noise patterns and content types, iteratively optimizing prompts to guide an expert LLM (DeepSeek-V3.2) in producing high-quality end-to-end refined texts.
Hierarchical Text-to-Operation Mapping:
- Line Alignment Mapping: Aligns original and refined lines based on content, context, and position to identify deletions, insertions, and modifications.
- Dynamic Context Replacement: Analyzes intra-line edit spans and dynamically expands surrounding context to convert edits into uniquely locatable replace_str operations, avoiding ambiguity from repeated substrings.
Filtering and Sampling: Low-confidence examples (e.g., excessive operations, overly long insertions) are filtered. The training data is constructed using ratio-controlled sampling by operation-combination type to stabilize the training distribution.

3. Large-Scale Execution Pipeline

During inference, UltraX processes large-scale corpora through:

Segment-wise Prediction: Documents are split into sliding windows (with overlap) to handle length limits. The model predicts operation sequences for each segment.
Operation Aggregation: Local operations are mapped back to a global line-number space. Overlapping regions are handled by the preceding window to prevent duplicate modifications.
Systematic Post-Processing: The pipeline includes ambiguous replacement filtering, merging of adjacent/interfering operations on the same line, and detection of repetitive patterns (e.g., cyclic insertions) to ensure execution stability.
Deterministic Execution: A deterministic executor applies the validated program to the original text, ensuring traceability and reproducibility.

Key Contributions

UltraX Framework: A function-calling refinement framework that introduces insertion alongside deletion and modification, completing the editing function space for fine-grained instance-level editing.
Reliable Supervision Pipeline: A hierarchical text-to-operation mapping system utilizing dataset-adaptive prompt optimization, Line Alignment Mapping, and Dynamic Context Replacement to convert end-to-end refined texts into structured program supervision.
Robust Execution Mechanism: A large-scale execution pipeline featuring sliding-window prediction, global operation aggregation, and systematic post-processing (including duplicate pattern detection) to normalize and validate model outputs.
Empirical Validation: Extensive experiments demonstrating that UltraX improves downstream performance across multiple heterogeneous corpora while achieving superior data efficiency.

Experimental Results

The authors conducted from-scratch pre-training experiments using a 1B-parameter MiniCPM model on five corpora: FineWeb, RedPajama-V2, AICC, Ultra-FineWeb, and FineWeb-ProX-Doc.

Performance Gains: UltraX achieved the highest average performance across all five corpora. On multiple datasets, it showed relative improvements exceeding 2% compared to raw data and the ProX-C baseline.
Data Efficiency: UltraX matched or surpassed baseline methods with fewer training tokens. For instance, on FineWeb, UltraX trained with 16B tokens achieved a higher average score (45.49) than the 20B-token baselines (Raw: 45.08, ProX-C: 45.05).
Refinement Quality: Analysis of 80K sampled documents showed UltraX achieved a higher average quality score (9.60 vs. 9.17 for ProX-C) and a significantly lower rate of low-score samples (0.38% vs. 2.59%). It demonstrated superior performance in noise removal, content preservation, and format integrity.
Token Distribution: Unlike aggressive deletion methods that drastically reduce token counts, UltraX preserves more non-empty documents and useful content, balancing noise removal with information preservation.

Significance and Claims

The paper claims that UltraX provides an efficient, reliable, and scalable approach to improving pre-training data quality. By shifting from end-to-end text generation to structured function calls, UltraX overcomes the efficiency bottlenecks of LLM-based refinement while avoiding the rigidity of rule-based methods.

The authors emphasize that the performance gains stem not from simply deleting more content, but from fine-grained, instance-level programmatic refinement that better balances noise removal with the preservation of valuable training signals. The framework successfully scales function-calling refinement to large pre-training corpora, offering a viable path to enhance LLM performance as data scaling laws begin to diminish.

The paper modestly acknowledges limitations, noting that experiments were conducted under limited token budgets and model scales, and that the current focus is on English web corpora. Future work is suggested for multilingual settings, model compression, and specialized refiners for specific data pathologies.