Recovering Engineering-Change Traceability from Legacy CNC Work Orders: A Pydantic Schema and Few-Shot LLM Extraction Baseline

Imagine you are a detective trying to solve a mystery, but the clues aren't neatly written on index cards. Instead, they are scribbled in the margins of old, crumpled receipts, written in a mix of two languages, and some of the paper has been chewed off by a hungry dog. This is the world of Engineering Change Management (ECM). In the high-stakes game of building machines, things rarely go exactly to plan. A part might break, a rule might change, or a material might need swapping. In big, fancy factories, these changes are tracked in massive, digital databases that act like perfect, organized filing cabinets. But in smaller workshops, the "history" of these changes often lives in tiny, messy notes left by workers on old computer systems. These notes are the "free text" of the manufacturing world: short, chaotic, and often cut off mid-sentence because the old software they were written in had a tiny memory limit.

The challenge for scientists is to turn these messy, bilingual scribbles (a mix of Traditional Chinese and English) into a clean, structured list that a computer can actually read and use. This is called Information Extraction. It's like trying to teach a robot to read a teenager's chaotic text messages and turn them into a formal report. The goal is to find out what changed, when it changed, and how one job order is connected to another, all without losing the story in the process. If we can't do this, companies lose the ability to trace their mistakes or improvements, which is like trying to drive a car without knowing where the road goes.

---

The Mystery of the 255-Character Limit

In this study, researchers from Taiwan tackled a very specific, very messy problem. They went to a CNC machinery company (a place that builds precise metal parts) and looked at their old work orders. They found 399 of these orders, but only 217 had any notes at all. The notes were a chaotic mix of Traditional Chinese and English, full of industry jargon, and—here's the kicker—125 of them were literally chopped off at exactly 255 characters.

Why 255? It turns out the company was using an old database system called Microsoft Access, which had a rule that a text box could only hold 255 letters. If a worker wrote a longer note, the computer just sliced it off, often right in the middle of a sentence or a crucial detail. It's like trying to read a recipe where the instructions for "add the eggs" are cut off, leaving you with just "add the..." and then nothing.

The team wanted to see if they could use a Small Language Model (SLM)—a type of AI that is smart enough to understand language but small enough to run on a regular computer without needing the internet—to fix this mess. They didn't want to send the company's secret data to a giant cloud server; they wanted to keep it local and private.

The Detective's Toolkit: Pydantic and the "Grounding" Test

To solve this, the researchers built a special "template" called a Pydantic schema. Think of this as a very strict, digital cookie cutter. No matter how messy the dough (the text) is, the cutter forces it into a perfect shape. The AI was trained to look at the messy notes and force the information into this specific shape, identifying things like "Component" (what part changed), "Substitution" (what it was swapped for), and "Traceability Links" (how this job connects to a previous one).

They used a 3-billion-parameter model (a "small" AI in the world of large language models) and taught it to fill out this cookie cutter. But there was a catch: the AI is a bit of a daydreamer. When it sees a note about a machine part, it sometimes gets so excited about what could be true that it invents details that aren't actually there. This is called hallucination. It's like a student who, when asked to summarize a story, makes up a character who wasn't in the book because they think it makes the story better.

The Results: Good at Finding, Bad at Filtering

The results were a mix of success and a very specific kind of failure.

First, the good news: The AI was incredibly good at following the rules. 96.9% of the time, it produced a perfectly formatted answer that fit the cookie cutter. It successfully found most of the changes and could even spot when a note was cut off.

However, the AI had a major personality flaw: it was over-generous.

• For finding things (Recall): It was great. It found 80% of the actual connections between work orders.
• For being accurate (Precision): It was terrible. Only 10% of the connections it claimed to find were actually real.

In plain English: The AI was shouting, "I found a link! I found a link!" and it was right only 1 time out of 10. It was shouting "I found a link!" when there was no link at all 9 times out of 10. It was making up connections to be helpful, which is dangerous in a factory where you need to know exactly what happened.

The Two Fixes: A Warning vs. A Filter

The researchers tried two different ways to stop the AI from making things up.

Attempt 1: The "Just the Facts" Prompt
They tried to talk to the AI, adding a special instruction to the prompt: "Only write down things you can see in the text. If you aren't sure, leave it blank."

• The Result: This backfired. The AI didn't just stop making things up; it stopped writing anything useful. It became so scared of making a mistake that it deleted real information too. The number of correct links dropped from 80% to 40%. The "Just the Facts" instruction made the AI too timid to do its job.

Attempt 2: The "Truth Filter"
Instead of asking the AI to be careful, the researchers built a separate, automatic filter that ran after the AI finished its work. This filter had a simple rule: "If the AI wrote a word or a number, check if that exact word or number appears in the original text. If it doesn't, delete it."

• The Result: This was a huge improvement, but not a magic cure. The filter didn't change how many things the AI found (Recall stayed at 80%), and it wiped out all the fake words that weren't in the text, driving the "hallucination" count of made-up strings to zero. However, the AI's accuracy (Precision) only jumped from 10% to 13.3%.

Why didn't it get to 100%? Because the AI was still making a different kind of mistake. Even when the words it used were real and present in the text, it sometimes attached them to the wrong relationship. For example, it might see two work order numbers in a note and guess they are connected, when they are actually just mentioned side-by-side for a different reason. The filter can't catch this because the words are there; the error is in the logic of the connection, not the words themselves.

The Takeaway

The paper concludes that for this specific, messy type of data, you can't just tell an AI to "be honest." At this size, the AI doesn't really understand the difference between "what is in the text" and "what I think is in the text." It needs a deterministic filter—a rigid, unfeeling rule-checker—to clean up its mess.

The study shows that we can recover the history of engineering changes from these old, chopped-up notes, but we have to accept that the AI will still make logical errors even after we fix the word-fabrication ones. The "grounding" instruction (telling it to be careful) makes it worse, but a "post-hoc filter" (checking its work afterwards) successfully removes all the made-up words and improves accuracy, though it cannot fix the AI's confusion about how things are related. It's a reminder that sometimes, the best way to handle a daydreaming robot isn't to lecture it, but to give it a fact-checker who can spot the lies, even if the robot still gets the plot twists wrong.