Temporal Reliability of ChatGPT-Generated Bilingual Texts: Implications for AI-Assisted Language Learning and Multilingual Information Access

Imagine you have a super-smart, all-knowing robot friend who speaks every language on Earth. You ask it to translate a tricky sentence, and it gives you a perfect answer. But what if you ask the exact same question a month later, after the robot has had a "software update"? Does it still give the same answer, or has it forgotten something, changed its mind, or started speaking in a different style? This is the big question behind a new study in the world of Artificial Intelligence (AI) and language.

Scientists call these super-smart robots "Large Language Models" (LLMs). They are like digital brains that can write, translate, and chat. Usually, when we buy a new video game or a new phone, we expect the newest version to be better than the old one. We assume that if a company releases "Version 5.0" and then "Version 6.0," the second one is smarter, faster, and more accurate. But with these AI robots, things aren't always that simple. They are constantly being tweaked and updated behind the scenes, sometimes changing how they think or speak without us even knowing. This study asks a very important question: If you use an AI to help you learn a language or read news in a foreign tongue, can you trust that the "newest" version is actually the best one? Or does the robot sometimes get worse at certain things even as it gets better at others?

The Experiment: Testing the Robot's Memory

To find out, a researcher named Zhihan Fu decided to play a game of "spot the difference" with four different versions of a popular AI chatbot (named GPT-5.0, GPT-5.4, GPT-5.5, and GPT-5.6 Sol). Think of these versions like four different snapshots of the same person taken over a few months.

The researcher didn't just ask the robot to chat; he gave it a very specific challenge. He took three real-world articles—one about money (finance), one about solar power technology, and one about government rules (public policy)—and asked every single version of the robot to translate them from Chinese into English. He used the exact same instructions for every version and compared their answers to a "gold standard" translation made by a human expert.

He used two main ways to judge the translations:

1. The "Meaning" Check: He used a tool called COMET to see how close the robot's meaning was to the human expert's meaning. Did it keep all the facts right?
2. The "Readability" Check: He measured how easy the text was to read. Was it too hard? Were the sentences too long? Did it use the right words?

The Surprise: The Robot Isn't Getting Better in a Straight Line

The results were a bit like watching a rollercoaster that goes up and down in different directions for different passengers. The big surprise was that newer versions of the AI were not automatically better. In fact, the robot's performance changed in weird, unpredictable ways depending on what topic it was talking about.

Here is what happened in each "world" the robot visited:

• The Government Rules World (Public Policy): This was the only place where the robot got steadily better at keeping the meaning correct. With each new version, the "Meaning" score went up, reaching its highest point with the newest version, GPT-5.6 Sol. However, there was a catch: as the meaning got more accurate, the text became harder to read. The newer versions started writing in a more complex, clunky style that felt less like a smooth story and more like a stiff legal document.
• The Money World (Finance): Here, the robot was pretty steady for a while, but then it peaked with version GPT-5.5 before getting slightly worse again with the newest version. The newest version wasn't the champion here.
• The Solar Power World (Technology): This was the wildest ride. The robot started strong, got even better with version GPT-5.4, and then got worse with every update after that. By the time the newest version (GPT-5.6 Sol) arrived, its ability to keep the meaning correct had dropped below even the very first version! However, the newest version did something interesting: it broke the long, confusing sentences into shorter, punchier ones, making the text look easier to read, even though it was actually missing some of the important technical details.

The Big Takeaway: "Newer" Doesn't Mean "Better"

The study suggests that we cannot just assume that the latest AI update is the best tool for the job. It's like a chef who gets a new kitchen. Maybe the new oven makes their cakes taste perfect, but the new knives make their vegetable chopping slower.

• For Government Texts: The newest AI is the most accurate, but it's also the hardest to read.
• For Tech Manuals: The newest AI writes in shorter sentences (which looks nice), but it actually loses important meaning compared to older versions.
• For Money Texts: The middle version (GPT-5.5) was the best, not the newest one.

The researcher found that the robot's ability to keep the meaning correct and its ability to be easy to read often moved in opposite directions. Sometimes, making the text more accurate made it harder to understand. Other times, making it shorter and simpler made it less accurate.

Why This Matters for You

If you are a student using AI to help you learn a language, or if you are trying to read news from another country, this study is a warning label. It tells us that we shouldn't just grab the "newest" version of an AI and assume it's the smartest. The "best" version depends entirely on what you are trying to do.

If you need to understand a complex government rule, the newest version might give you the right facts but in a confusing way. If you are reading a technical manual, the newest version might look simple but might have missed a crucial safety warning. The study suggests that we need to keep testing these AI tools over and over again, checking them for specific jobs, because the robot changes its personality every time it gets an update. The "perfect" AI translator doesn't exist yet; instead, we have a robot that is constantly shifting, and we have to be careful about which version we trust for which task.