💬 NLP

Towards Empowering Consumers through Sentence-level Readability Scoring in German ESG Reports

This paper enhances German ESG report analysis by crowdsourcing sentence-level readability annotations and demonstrating that while LLM prompting shows promise, a small finetuned transformer achieves the highest accuracy in predicting human readability perceptions.

Original authors: Benjamin Josef Schüßler, Jakob Prange

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Josef Schüßler, Jakob Prange

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to choose a new bank or an electricity provider. You want to pick the one that is truly "green" and sustainable. To help you, companies publish ESG Reports (Environmental, Social, and Governance reports). These are like the "nutrition labels" for a company's soul.

But here's the problem: These reports are often written in a confusing, jargon-heavy language that feels like reading a legal contract written in a foreign language. If you can't understand them, you can't make a good choice.

This paper is like a team of detectives trying to build a smart translator that can instantly tell you: "Hey, this sentence is easy to understand," or "Whoa, this sentence is a mess of confusing words."

Here is the story of their investigation, broken down simply:

1. The Mission: Empowering the "Regular Joe"

The researchers wanted to help regular people (not just financial experts) understand these reports. They asked two big questions:

Question 1: Are German sustainability reports actually easy to read?
Question 2: Can we build a computer program to grade them automatically?

2. The Evidence: Crowdsourcing the "Human Opinion"

To train their computer, they needed a "gold standard" of what humans think is easy or hard to read.

The Experiment: They took thousands of sentences from real German ESG reports and asked regular German speakers (via an online crowd) to rate them on a scale of 1 (confusing) to 4 (crystal clear).
The Surprise: They found that, surprisingly, most sentences were rated as quite easy to read. However, there was a lot of disagreement. One person might think a sentence is clear, while another thinks it's nonsense. This proved that "readability" is subjective—like how one person might love a spicy curry while another finds it too hot.

3. The Tools: Three Different "Detectives"

The team built three different types of computer models to see which one could best predict how humans would rate a sentence. Think of them as three different detectives:

Detective A: The "Rulebook" (Traditional Formulas)
- How it works: This detective just counts things. "How long is the sentence? How many big words are there?" It's like a teacher grading an essay based only on word count.
- Verdict: It was okay, but a bit dumb. It missed the nuance.
Detective B: The "Grammar Police" (The Whitebox Model)
- How it works: This detective looks at the structure of the sentence. Is it in the passive voice? Is the sentence tree too deep and tangled? It's like a mechanic looking under the hood of a car to see if the engine is too complex.
- Verdict: This was the winner for accuracy. It was small, fast, and made the fewest mistakes when guessing what humans would think. It proved that if you understand the grammar structure, you can predict readability very well.
Detective C: The "Super-Intelligent AI" (Large Language Models)
- How it works: These are the giant AIs (like the ones you chat with today). They are incredibly smart and can understand context and meaning deeply.
- Verdict: They were great at spotting the difference between "easy" and "hard" sentences (ranking them correctly), but they were bad at giving the exact score. They were like a genius who knows the answer is "right" but can't explain why or give the precise number. Also, they were slow and expensive to run.

4. The Big Takeaway: The "Goldilocks" Solution

The researchers found that while the giant AI is impressive, the small, specialized "Grammar Police" model was the most practical tool.

It was fast (like a sports car).
It was accurate (it didn't make many mistakes).
It was transparent (we know why it gave a score because it looks at grammar rules).

They also found that if you combine the "Grammar Police" with the "Super AI," you get a tiny bit better accuracy, but it slows down the process significantly.

5. Why This Matters for You

Imagine a future where you are shopping for a bank. You open an app, and instead of reading a 50-page PDF, the app instantly scans the bank's report.

It highlights the sentences that are clear and honest.
It flags the sentences that are confusing or trying to hide something (greenwashing).
It simplifies the hard parts for you.

In short: This paper builds the foundation for a world where companies can't hide behind confusing language. By teaching computers to spot "clear" vs. "confusing" text, we can empower regular consumers to make smarter, more sustainable choices without needing a PhD in finance.

The Moral of the Story: You don't need a giant, expensive brain to understand a sentence; sometimes, a small, focused tool that understands grammar is all you need to make the complex simple.

1. Problem Statement

With the increasing regulatory and societal demand for Environmental, Social, and Governance (ESG) reporting, companies are generating massive amounts of sustainability data. While these reports are often legally mandated (e.g., EU CSRD), they are frequently written for financial experts rather than layperson consumers.

The Core Issue: Consumers need to make informed, sustainable purchasing decisions based on these reports, but the language is often too complex or ambiguous.
The Gap: Existing readability assessment methods are often document-level (e.g., Flesch Reading Ease) or designed for English educational texts. There is a lack of fine-grained, sentence-level readability scoring specifically for German ESG reports that aligns with non-expert (layperson) perceptions.
Objective: To develop and evaluate automatic readability assessment (ARA) models that can identify clear vs. unclear sentences in German ESG reports to empower consumer decision-making.

2. Methodology

A. Data Collection and Annotation

Source: The authors extended the SustainEval 2025 dataset, which contains excerpts from German Sustainability Code reports.
Structure: Each data point consists of three context sentences and one target sentence.
Crowdsourcing: Native German speakers (laypeople) were recruited via Prolific to rate the readability of the target sentence on a 4-point Likert scale (1: Not at all clear to 4: Very clear).
Data Characteristics:
- Imbalance: The dataset is heavily skewed toward high readability (scores of 4), likely due to the native proficiency of annotators and potential "social desirability" bias in crowdsourcing.
- Agreement: Inter-annotator agreement was moderate (Mode Agreement ~60–70%). The authors used majority voting to aggregate labels, handling ties by taking the mean.
- Preprocessing: Underrepresented classes were oversampled in the training set to mitigate class imbalance.

B. Model Architectures

The study compares three distinct approaches to modeling sentence-level readability:

Whitebox Models (Interpretable, Syntax-based):
- Baseline 1 (Sentence Length): Simple linear regression using word count.
- Baseline 2 (Readability Formulae): An XGBoost model trained on scores from established formulas adapted for German (Flesch-Amstad, HKPS, Polysyllabic Proportion, Vienna Formula, LIX).
- Proposed Model (Syntactic Features): A feed-forward neural network using specific linguistic features extracted via spaCy:
  - POS-tag n-grams (bigrams/trigrams).
  - Dependency tree depth.
  - Mean dependency distance.
  - Root POS-tag.
  - Binary flags for passive voice and subordination.
- Goal: To determine if explicit syntactic features can predict readability without the "black box" nature of LLMs.
Blackbox Models (Deep Learning):
- XLM-RoBERTa: A transformer encoder fine-tuned as a regression head. Tested in base and large variants.
- Generative LLMs: Instruction-tuned models (Qwen 3, Gemma 3, Llama 3) prompted with few-shot examples to classify readability into 4 classes. Prompts were in English to mitigate language bias in multilingual models.

C. Evaluation Metrics

MSE (Mean Squared Error) & MAE (Mean Absolute Error): To measure prediction accuracy against human ratings.
Kendall $\tau$ : To measure rank-order correlation (ability to distinguish easy vs. hard sentences, regardless of absolute score bias).
Inference Time: To evaluate computational efficiency.

3. Key Results

A. Model Performance

Best Overall (Error): The fine-tuned XLM-RoBERTa (base) model achieved the lowest MSE (0.0295) and MAE (0.1114), outperforming all other individual models.
Best Ranking (Correlation): Qwen 3 (LLM) achieved the highest Kendall $\tau$ (0.2822), indicating it was best at distinguishing between easy and hard sentences, despite having a systematic bias in absolute scoring (assigning lower scores than humans).
Whitebox Performance: The syntactic feature-based model performed comparably to the traditional readability formulae baseline (MSE ~0.039) but was significantly slower than the formulae baseline.
LLM Limitations: While Qwen showed high ranking potential, Llama 3 and Gemma 3 performed poorly, failing to estimate readability accurately.
Ensembles: Averaging predictions (e.g., Syntax + XLM-RoBERTa + Qwen) yielded marginal improvements in MSE but increased inference time significantly.

B. Feature Importance (Ablation Study)

In the syntactic model, trigrams were the most critical feature; removing them caused the largest performance drop. Passive voice and sentence depth were the next most important features. Interestingly, removing bigrams slightly improved error metrics, suggesting redundancy with trigrams.

C. Human vs. Model Insights

Subjectivity: Human ratings showed significant variation, confirming that readability is subjective even among native speakers.
Context Dependency: Models struggled with sentences that were syntactically simple but semantically dependent on preceding context (which was provided to humans but not to the models in the "no-context" setup).

4. Key Contributions

Dataset Extension: Creation of a crowdsourced, sentence-level readability dataset for German ESG reports, addressing a gap in non-expert perception data.
Methodological Comparison: A comprehensive evaluation of whitebox (syntactic), traditional formula, and blackbox (Transformer/LLM) approaches for a specific domain.
Empirical Findings:
- Demonstrated that small, fine-tuned transformers (XLM-RoBERTa) offer the best trade-off between accuracy and speed for this task.
- Showed that while LLMs have high potential for ranking (distinguishing clear vs. unclear), they suffer from calibration issues (score bias) without specific fine-tuning.
- Highlighted that traditional readability formulas are insufficient for ESG reports, but syntactic features alone can capture a significant portion of readability variance.
Consumer Empowerment Framework: Proposed a practical pipeline where readability scoring acts as a filter in Retrieval-Augmented Generation (RAG) systems to prioritize clear sentences or trigger simplification for complex but relevant ones.

5. Significance and Implications

Regulatory Compliance: Supports the EU's "Empowering Consumers Directive" by providing tools to ensure sustainability information is accessible to the general public, not just experts.
Greenwashing Mitigation: By flagging unclear or overly complex sentences, these tools can help consumers identify potential obfuscation or "greenwashing" in corporate reporting.
Future Systems: The results suggest that for real-time consumer applications, a fine-tuned XLM-RoBERTa is the optimal choice, balancing high accuracy with low latency.
Limitations & Future Work: The study notes the challenge of class imbalance (too many "easy" sentences) and the difficulty of assessing readability without context. Future work should explore personalized readability models and document-level coherence.

In conclusion, the paper establishes that while LLMs show promise in ranking sentence difficulty, small, domain-fine-tuned transformer models currently provide the most reliable and efficient solution for automating readability scoring in German ESG reports, directly contributing to more transparent and consumer-friendly sustainability reporting.