💬 NLP

Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

This study evaluates the performance of various large language models in restoring Romanian diacritics, finding that advanced models like GPT-4o achieve high accuracy while others show greater variability, thereby highlighting the influence of architecture, training data, and prompt design on this NLP task.

Original authors: Mihai Nadas, Laura Diosan

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mihai Nadas, Laura Diosan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a pile of Romanian text messages, but someone accidentally scrubbed all the special "accent" marks off the letters. In Romanian, these marks (like ă, î, ș, ț, and â) are the difference between a word meaning "to eat" and "to sleep," or "a village" and "a city." Without them, the text is like a map with all the street names erased—confusing and hard to read.

This study asked a big question: Can the super-smart AI chatbots we see everywhere today (called Large Language Models, or LLMs) fix these messy texts and put the accents back in the right places?

The Great Accent Restoration Race

The researchers set up a massive race track. They gathered 2,000 Romanian sentences and stripped them of all their accents, turning them into a "blank slate." Then, they invited a lineup of famous AI models to the starting line to see who could restore the accents best.

The lineup included:

The Heavyweights: OpenAI's GPT-3.5, GPT-4, and the newest GPT-4o.
The Google Contender: Gemini 1.0 Pro.
The Open-Source Crew: Meta's Llama 2 and 3 (in various sizes), MistralAI's Mixtral, and a few others.
The "Dummy" Baseline: A simple trick called the "Echo" model. This model didn't try to fix anything; it just copied the messy text back exactly as it was. It was the "zero effort" score everyone had to beat.

The Winners and the Losers

The Champions:
The results were clear: OpenAI's GPT-4o was the undisputed star of the show. When given a specific set of instructions (a "prompt") that included three examples of how to fix the text, it achieved a Total Average Score (TAS) of 0.9639.

To put that in perspective, the "Echo" baseline (the lazy copier) only scored 0.8100. This means GPT-4o didn't just do a little better; it outperformed the baseline by 19%. In fact, the best-performing model (GPT-4o) was 1.190 times better than the baseline. It was like a master chef perfectly seasoning a dish, while the baseline just served it plain.

The Surprising Strugglers:
Here is where it gets interesting. You might think bigger or more famous open-source models would do well, but the paper suggests otherwise.

Meta's Llama 2 7B was a total flop. It scored a 0.002, which is extremely low. It was so bad that it was 0.002 times the performance of the baseline. While it wasn't literally zero, it was effectively useless for this task, failing to beat the baseline by a massive margin.
Meta's Llama 2 70B (the bigger version) did slightly better but still failed to beat the baseline, scoring 0.146.
Mixtral 8x7B and RoLlama 2 7B also scored below the baseline, meaning they were actually worse at fixing the text than just copying the errors.

The paper suggests that size doesn't always equal success. Just because a model is huge or open-source doesn't mean it knows how to handle the tricky rules of Romanian accents.

The Secret Weapon: The "Prompt"

The study found that how you ask the AI matters just as much as the AI itself.

The "3-Shot" Trick: The best results came when the researchers gave the AI three examples of "messy text" and the "fixed text" right before asking it to do the job. This is called a "3-shot" prompt.
The Result: Using this method, GPT-4o's score jumped up. The paper suggests that giving the AI a few examples (like showing a student a few solved math problems before a test) helps it understand the rules much better than just giving it a simple command.

Where Did They Go Wrong? (The Error Analysis)

Even the winners made mistakes, and the researchers looked closely at where the trouble happened:

The â vs. î Confusion: The most common error was mixing up the letters â and î. In Romanian, â is used in the middle of words, and î is used at the start or end. The AI sometimes put î in the middle (like writing romîn instead of român). About 21.3% of all errors came from this specific mix-up.
Capital Letters: The AI struggled more with words at the start of a sentence that were capitalized. For example, it got the accent right on lowercase ș 98.7% of the time, but only 94.6% of the time when it was a capital Ș.
Over-Enthusiasm: Some models, like Mixtral, got too excited. They added accents where they didn't belong. The paper notes that Mixtral added an average of 16.35 extra accents per 10-word sentence, creating "hallucinations" (fake accents) that made the text worse.

What the Paper Rules Out

The paper is very clear about what doesn't work:

It rules out the idea that "any" LLM is good at this. The study explicitly shows that several popular models (like Llama 2 7B) are actually worse than doing nothing.
It rules out the idea that "bigger is always better." The 70-billion-parameter Llama 2 model performed terribly, while the smaller but well-tuned GPT-4o performed brilliantly.
It rules out the idea that simple copying is a valid solution. The "Echo" baseline was used specifically to show that just copying the text without fixing it is a low bar that many models failed to clear, proving that copying is not an effective strategy for restoration.

How Sure Are We?

The authors are measured and specific about their findings. They didn't just guess; they ran the models against a dataset of 1,000 statements from two different sources (one from a dictionary project and one from web crawlers).

They suggest that the gap between the best models (GPT-4o) and the worst (Llama 2 7B) is real and significant.
They suggest that the "3-shot" prompt strategy is a key factor in success.
They note that their results are based on a specific subset of data (post-1993 Romanian rules) and that they didn't test historical texts or other languages.

The Bottom Line

If you want to fix Romanian text with accents, don't just grab any AI. The paper suggests that OpenAI's GPT-4o, when given a few examples to follow, is currently the best tool for the job. However, many other popular models are still learning the ropes and might actually make the text messier than it was to begin with.

The study concludes that while AI is powerful, it still needs a little help (like good instructions and examples) to master the subtle, beautiful details of the Romanian language.