💬 NLP

Controlling Language Difficulty in Dialogues with Linguistic Features

This paper proposes a framework that controls language difficulty in educational dialogues by training large language models on linguistically annotated data using readability, syntactic, and lexical features, thereby achieving more stable and flexible proficiency modulation than prompt-based methods while introducing the Dilaprix metric for evaluation.

Original authors: Shuyao Xu, Wenguang Wang, Handong Gao, Wei Kang, Long Qin, Weizhi Wang

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shuyao Xu, Wenguang Wang, Handong Gao, Wei Kang, Long Qin, Weizhi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a language teacher trying to talk to a student. If you speak like a university professor to a 5-year-old, the child will get confused and give up. But if you speak like a toddler to a teenager, they will get bored. The goal is to find the "Goldilocks zone"—the perfect level of difficulty that matches the student's current skills.

This paper is about teaching Large Language Models (LLMs)—the smart AI chatbots—to be that perfect teacher. The researchers wanted to solve a problem: AI is great at talking, but it's hard to tell it, "Speak exactly at a 3rd-grade level," or "Speak at a high-school level," without it accidentally slipping into a different difficulty.

Here is how they did it, explained simply:

1. The "Recipe" for Difficulty

Instead of just telling the AI "be easy" or "be hard," the researchers gave the AI a specific recipe made of three types of ingredients. They measured the AI's output using these "linguistic features":

Readability (The "Speed Bumps"): They used classic formulas (like the Flesch-Kincaid test) that count how long sentences are and how many syllables words have. Think of this as measuring how bumpy the road is for a reader.
Syntax (The "Tree Structure"): They looked at the grammar trees of sentences. Imagine a sentence as a tree. A simple sentence is a small sapling. A complex sentence with many clauses is a giant oak with deep, twisting roots. They measured how deep and complex these "trees" were.
Lexical (The "Word List"): They checked the vocabulary against two lists: a "Simple" list (words a child knows) and an "Intermediate" list (words a middle-schooler knows). They calculated what percentage of the AI's words came from the simple list.

2. The New Ruler: "Dilaprix"

The researchers realized that existing ways to measure difficulty were like trying to measure a conversation using a ruler meant for essays. Conversations are messy and back-and-forth.

So, they invented a new ruler called Dilaprix.

What it is: A single number that combines all those ingredients (readability, grammar trees, and word lists) into one score.
Why it matters: It's like a "difficulty thermostat." If the score is low, the AI is speaking simply. If the score is high, the AI is speaking complexly.
The Proof: They asked human experts to guess which of two AI sentences was harder. The Dilaprix score matched the human experts' guesses almost perfectly (95% correlation). It's a reliable way to know exactly how hard the AI is talking.

3. Training the AI: The "Coach" Approach

How did they teach the AI to hit these specific numbers?

The Old Way (Prompting): You might try to tell the AI, "Please speak at a B1 level." The paper found this is like asking a student to "be polite" without giving examples; the AI often guesses wrong or gets inconsistent.
The New Way (Fine-Tuning): The researchers created a massive dataset of conversations where every response was tagged with those specific "ingredients" (the 11 linguistic features). They then "trained" the AI on this data.
- Think of it like a coach giving a player a specific drill: "Today, you must use exactly 18 words, keep the sentence tree depth at 9, and use 80% simple words."
- They also used a technique called DPO (Direct Preference Optimization), which is like a coach reviewing game tape and saying, "That response was good, but this one was better because it stuck to the rules."

4. The Results: Precision and Stability

The experiments showed that their new method was much better than the old ways:

More Control: The AI could smoothly dial the difficulty from very easy to very hard, covering a wide range. The old methods only had a few "preset" levels (like A1, A2, B1) and couldn't fine-tune in between.
More Stable: When the AI tried to be "easy," it stayed easy. When it tried to be "hard," it stayed hard. The old methods would sometimes start simple and then accidentally get too complex in the middle of a sentence.
Still Good Quality: Crucially, making the AI speak simply didn't make it sound robotic or fail to answer the question. It kept the conversation natural while strictly controlling the difficulty.

5. A Surprising Discovery: The "Word Count" Trap

The researchers found something interesting about Utterance Length (how many words are in a sentence).

When they forced the AI to strictly control the number of words, the quality of the conversation actually dropped.
Why? Imagine a teacher asking a student, "Can you tell me why you like drawing mountains and what you think about the park?" If the AI is forced to keep the sentence short, it might cut off the question or answer it poorly.
The Fix: When they removed the strict "word count" rule but kept the other rules (grammar and vocabulary), the AI actually did a better job at controlling difficulty and kept the conversation flowing naturally.

Summary

This paper presents a new way to teach AI chatbots to speak at the exact right level for language learners. Instead of vague instructions, they use a precise "recipe" of grammar and vocabulary rules, measured by a new tool called Dilaprix. The result is an AI that can act as a perfect language tutor, adjusting its speaking style to match the student's skills without losing its ability to have a natural conversation.

Technical Summary: Controlling Language Difficulty in Dialogues with Linguistic Features

Problem Statement

Large Language Models (LLMs) have become powerful tools for second language (L2) acquisition, particularly in simulating interactive dialogues for speaking practice. However, a significant challenge remains: adapting the language difficulty of LLM-generated responses to precisely match a learner's proficiency level. Existing methods often rely on prompt-based instructions using broad frameworks like the Common European Framework of Reference (CEFR) (A1–C2). While effective to a degree, these approaches face limitations due to the inherent ambiguity of CEFR definitions and a lack of fine-grained control. Furthermore, standard complexity metrics often rely on proficiency-labeled corpora (e.g., essays or reading passages) that are unsuitable for conversational data. Consequently, there is a need for a framework that can dynamically adjust LLM output complexity based on specific, quantifiable linguistic features to align with target L2 learner proficiency while maintaining high dialogue quality.

Methodology

Linguistic Feature Extraction

The proposed framework leverages three categories of linguistic features to quantify and regulate text complexity:

Readability Features: Four established scores are utilized: Flesch Reading Ease (FR), Flesch-Kincaid Grade Level (FG), Gunning Fog Index (GF), and Coleman-Liau Index (CL). These rely on sentence length, syllable counts, and character counts.
Syntactic Features: Five metrics derived from constituency parsing trees are employed: Tree Depth (TD), Leaf Node Count (LN), Non-terminal Diversity (ND), Subtree Complexity (SC), and Utterance Length (UL). These capture structural complexity.
Lexical Features: Two ratios based on expert-curated word lists (elementary and intermediate levels) are used: Simple Word Ratio (SW) and Intermediate Word Ratio (IW).

Dataset Construction

The authors constructed a specialized dataset using user conversation data from a textbook dialogue application (students in Grades 3–9).

Instruction Tuning Data: Approximately 100,000 instances were generated. An LLM (Qwen-plus) produced utterances conditioned on conversational context and instructions specifying one of six CEFR proficiency levels. From these, 11 linguistic features were extracted. Each training instance paired the original context with these extracted features, ensuring each dialogue context was associated with six distinct responses reflecting unique linguistic configurations.
DPO Data: For Direct Preference Optimization (DPO), 30,000 instruction-response pairs were sampled. Negative samples included responses that were contextually correct but linguistically inaccurate, or responses that failed the dialogue task.

Model Training

Experiments were conducted on Qwen2.5-7B-Instruct and LLaMA3.1-8B-Instruct.

Instruction Tuning: Models were fine-tuned on the linguistically annotated dialogue data to learn the mapping between specific feature constraints and generated text.
Direct Preference Optimization (DPO): Models were further optimized using DPO to align outputs with human preferences regarding task completion and linguistic constraints.

Evaluation Metrics

Response Success Rate (RSR): Measures dialogue quality by evaluating if a response is both natural and successfully fulfills the specified dialogue task.
Dialogue Language Proficiency Index (Dilaprix): A novel, interpretable metric designed to quantify dialogue complexity without relying on pre-assigned proficiency labels. It integrates the 11 linguistic features into a unified score (0 to 1) using a normalization and aggregation function. Dilaprix was validated against expert judgments, achieving a Pearson correlation coefficient (PCC) of 0.950.

Key Results

Controllability and Stability

Superior Control: Models trained on linguistically annotated data (Linguistic Feature Controlled) demonstrated significantly better controllability than prompt-based methods or CEFR instruction-tuned baselines.
Range and Stability: The QWEN-DPO model achieved a Dilaprix range from 0.073 to 0.883, compared to a narrower range (0.211–0.685) for CEFR-based methods. Furthermore, the feature-controlled models exhibited lower standard deviation in Dilaprix scores (e.g., 0.084 for QWEN-DPO vs. 0.155 for QWEN-CEFR), indicating greater stability in generating consistent complexity levels.
Dialogue Quality: The fine-tuned models maintained high dialogue quality, with QWEN-CEFR achieving an average RSR of 0.966 across CEFR levels, significantly outperforming the prompt-based baseline (0.908). The linguistic feature-controlled models maintained comparable or superior RSR across the continuous Dilaprix spectrum.

Ablation Studies

Feature Importance: Removing individual features generally degraded performance. The Simple Word Ratio (SW) was found to be the most critical feature; its removal caused the most significant performance drop.
Utterance Length (UL): Surprisingly, removing the Utterance Length constraint improved overall performance (AUC and RSR). The authors suggest that strict length constraints can conflict with dialogue tasks requiring longer, more complex utterances, thereby hindering task completion.
Granular Control: The framework allows for assigning different target values ( $t$ ) to different feature groups (e.g., lexical vs. syntactic). Experiments showed this enables fine-grained control, such as generating text with simple vocabulary but complex sentence structures, while maintaining high quality.

Significance and Claims

The paper claims to address the challenge of precise language proficiency control in educational dialogue systems by moving beyond broad, ambiguous labels (like CEFR) to specific, quantifiable linguistic features.

Novel Metric: The introduction of Dilaprix provides a reliable, interpretable metric for assessing dialogue difficulty that correlates strongly with expert judgment, overcoming the limitations of existing corpora-based metrics.
Effective Framework: The study demonstrates that training LLMs on linguistically annotated data enables precise modulation of language proficiency. This approach offers superior flexibility and stability compared to prompt-based methods.
Pedagogical Utility: By enabling the generation of responses that align with a learner's current proficiency level, the proposed framework enhances the adaptability and pedagogical value of AI-driven language learning tools, potentially preventing learner disengagement due to overly complex language.

The authors maintain a modest stance, acknowledging that while the approach is effective, it requires substantial training data which may not always be available. They also note that Dilaprix is specific to dialogues and may require adaptation for other text types like essays.