💻 computer science

Taming CATS: Controllable Automatic Text Simplification through Instruction Fine-Tuning with Control Tokens

This paper introduces a domain-agnostic framework for Controllable Automatic Text Simplification (CATS) using instruction fine-tuning with control tokens, demonstrating that while smaller models can achieve consistent readability control, reliable compression and evaluation depend critically on training data variability and the adoption of error-based metrics to address limitations in existing corpora and standard evaluation practices.

Original authors: Hanna Hubarava, Yingqiang Gao

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanna Hubarava, Yingqiang Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot librarian. This robot can read any book, from complex medical journals to dense government laws, and rewrite them so a child can understand. But here's the catch: you want to tell the robot exactly how simple you want the rewrite to be. Maybe you want it to sound like a 4th-grade textbook, or maybe you want it to be half the length of the original.

This paper is about teaching that robot librarian to listen to your specific instructions using a new method called Taming CATS.

Here is the story of what they found, explained with some everyday analogies:

1. The Problem: The Robot Was Guessing

Before this study, trying to get a robot to simplify text to a specific level was like asking a chef to "cook something salty" without telling them how much salt to use. The chef (the AI) might guess, but you can't guarantee the result. Also, the people testing the chefs were using the wrong tasting spoons—they were checking if the food tasted good, but not if it actually had the right amount of salt.

2. The Solution: The "Magic Control Tokens"

The researchers gave the robot a new set of tools: Control Tokens.
Think of these like dials on a radio.

Instead of just saying "Simplify this," you say: "Simplify this, but set the Readability Dial to 4.0" or "Set the Length Dial to 0.5 (half size)."
They trained the robot by showing it thousands of examples where the "dial" was set, and the robot learned to turn the text until it matched that setting.

3. The Big Discovery: It's All About the Recipe Book (Data)

The most surprising thing they found wasn't about how big the robot was (its brain size), but about the cookbook they used to train it.

The Readability Dial Works Great: They found that if the cookbook had a wide variety of texts ranging from "very hard" to "very easy," the robot learned perfectly how to hit the target. It was like having a cookbook with recipes for every skill level.
The Length Dial Failed: However, when they tried to teach the robot to shorten text (compression), it struggled. Why? Because the cookbooks they used mostly had "complex" and "simple" versions that were almost the same length. It was like trying to teach a chef to make a small cake when every recipe in the book only showed how to make a cake of the exact same size. The robot had no examples of shrinking text, so it couldn't learn the trick.

4. The "Cut-and-Paste" Trap

The researchers also discovered that how you slice up your cookbook matters.

If you just grab random pages for training and testing, you might accidentally give the robot a "hard" book to study and a "medium" book to test on. The robot would look confused and fail.
They found that you have to stratify (carefully sort) your data, ensuring the training and testing books have a mix of difficulties that match perfectly. Otherwise, the robot thinks it's a genius during practice but fails the real exam.

5. Bigger Isn't Always Better

Usually, in AI, we think "bigger model = smarter." But here, a smaller robot (1 billion parameters) could sometimes do just as well as a giant one (14 billion parameters) if the training data was good.

Analogy: It's like a small, focused student who studies the right textbook vs. a giant encyclopedia that was never taught how to read the specific instructions. The focused student wins.

6. The Wrong Scorecard

Finally, they realized the old ways of grading the robot were broken.

Old tests asked: "Does the new text sound like the old text?" or "Is it grammatically correct?"
The new test asks: "Did the robot actually follow the dial?"
They found that a robot could produce perfect English that was totally the wrong complexity level. You need a scorecard that specifically checks if the robot hit the target number you gave it.

The Takeaway

To build a truly helpful text-simplifying robot, we can't just throw more money at bigger computers. We need:

Better Cookbooks: Datasets that actually show the robot how to change length and difficulty in many different ways.
Better Dials: Using those "control tokens" to give precise instructions.
Better Grading: Checking if the robot actually listened to the instructions, not just if it wrote good sentences.

In short: Garbage in, garbage out. If the training data doesn't show the robot how to shrink text, no amount of "fine-tuning" will make it a master of shortening stories. But if you give it the right data, even a small robot can become a master of simplification.

1. Problem Statement

Automatic Text Simplification (ATS) aims to reduce linguistic complexity while preserving meaning to improve accessibility. While Large Language Models (LLMs) have renewed interest in Controllable Automatic Text Simplification (CATS)—where systems adapt outputs to user-specified complexity levels—current approaches face significant limitations:

Data Constraints: Existing datasets often lack sufficient variation in target attributes (e.g., compression ratios), leading models to fail at learning fine-grained control.
Evaluation Gaps: Controllability is often treated as a decoding problem, but standard metrics (BLEU, SARI, LENS) measure similarity or fluency rather than alignment with specific target constraints (e.g., a specific readability score).
Distributional Mismatch: Naive data splits (random or native) often introduce distributional mismatches between training and evaluation sets, confounding results.
Model Scalability: It is unclear if larger models inherently perform better at control or if data quality and fine-tuning strategies are the dominant factors.

2. Methodology

The authors propose a domain-agnostic CATS framework based on Instruction Fine-Tuning (IFT) using discrete control tokens.

A. Control Attributes

The system targets five numerical attributes:

Readability: Flesch-Kincaid Grade Level (FKGL), Automated Readability Index (ARI), and Dale-Chall.
Compression: Word count ratio and Character count ratio.

B. Data Processing & Preprocessing

Datasets: Four domain-specific corpora were used: Med-EASi (Medical), SimPA (Public Administration), WikiLarge (Encyclopedic), and Newsela (News).
Harmonization: Data was converted to a unified JSONL schema.
Stratified Sampling: To prevent distributional mismatch, the authors employed stratified sampling based on readability metrics (specifically FKGL) rather than random splitting. They used the Kolmogorov–Smirnov (KS) test to ensure splits were statistically representative of the full dataset.
Filtering: A "monotonic" filter was applied to remove instances where complexity increased from source to reference, though the authors note this may reduce training diversity.

C. Fine-Tuning Framework

Prompt Design: The model is instructed via a dynamic prompt containing the source text and a control token in the format <METRIC=VALUE>.
- Example: <FKGL=4.0> indicates the target readability level.
- The token is prepended to the assistant's response, teaching the model the conditional relationship between the metric value and linguistic features.
Models: Experiments covered three model families (Llama, Mistral, Qwen) ranging from 1B to 14B parameters.
- Models $\le$ 4B were fine-tuned fully.
- Models > 4B used LoRA (Low-Rank Adaptation) to manage computational constraints.
Training Objective: Standard causal language modeling (cross-entropy loss) with prompt tokens masked.

D. Evaluation Metrics

The authors argue for a multi-dimensional evaluation:

Controllability: Measured via Mean Absolute Error (MAE) between the target attribute value and the generated text's actual attribute value.
Simplification Quality: SARI and LENS.
Similarity: BLEU and BERTScore.
Robustness: Multiple inference runs (3–5 seeds) to account for LLM non-determinism.

3. Key Contributions

Discrete Control Token Framework: Demonstrated that open-source decoder-only LLMs can be steered to specific readability and compression levels using simple instruction fine-tuning with discrete tokens.
Data-Centric Insights:
- Signal Variability: Controllability is heavily dependent on the variation in the training data. Readability control is learned well because datasets show high variation in FKGL/ARI. Compression control fails in sentence-aligned datasets (like SimPA/WikiLarge) because complex-simple pairs often have similar lengths (compression ratio $\approx$ 1.0), providing a weak learning signal.
- Stratification Importance: Naive splits lead to distributional mismatches that undermine both training and evaluation. Stratified sampling by readability is critical for reliable benchmarks.
Model Scaling Analysis: Found that larger model size does not guarantee better controllability. Smaller models (1–3B) can be competitive with larger ones (8–14B) if the data signal is strong. Performance gains are non-monotonic and vary by dataset and metric.
Evaluation Critique: Highlighted that traditional metrics (SARI, LENS) are insufficient for measuring control. They often reward copying or fail to penalize deviations from the target constraint. Error-based metrics (MAE) are indispensable for evaluating CATS.

4. Key Results

Readability vs. Compression: Models consistently learned to target specific readability levels (FKGL, ARI, Dale-Chall). However, compression control was underperformed across all models due to the lack of length variation in existing sentence-level corpora.
Model Performance:
- Qwen models showed the most stable performance across datasets and metrics.
- Mistral achieved strong SARI scores on specific datasets (Med-EASi, SimPA).
- Llama models produced well-formed outputs but showed more variance.
- Scaling: Increasing model size improved SARI (edit-based) and MAE (error-based) for readability control but offered limited gains for compression control.
Data Filtering Impact: Filtering for strictly monotonic readability reduction (removing non-simplifying pairs) did not consistently improve results and sometimes degraded performance, suggesting that retaining diverse attribute values is better than strict filtering.
Metric Correlation: Strong negative correlations were observed between SARI/LENS and MAE, indicating that high similarity to references does not necessarily equate to hitting the target control attribute.

5. Significance and Future Directions

Paradigm Shift: The paper shifts the focus of CATS from purely architectural choices to data curation and evaluation design. It argues that "controllability" is a data problem as much as a modeling problem.
Benchmarking Standards: The authors call for future CATS benchmarks to:
- Report split creation methods and verify representativeness (using KS/EMD tests).
- Include dedicated datasets with explicit, diverse compression ratios.
- Prioritize error-based metrics (MAE) alongside traditional quality metrics.
Practical Application: The proposed IFT framework is lightweight and flexible, enabling the transformation of open-source LLMs into steerable simplification tools for domains like medicine and public administration, provided the training data contains sufficient signal variation.

In conclusion, "Taming CATS" demonstrates that while LLMs can learn to control text complexity, their success is fundamentally bounded by the richness of the control signal in the training data and the rigor of the evaluation methodology, rather than just model scale.