Semantic Reverse Engineering Legacy Software Applications with ChatGPT, Gemini AI, and Claude AI
This research paper presents findings on utilizing ChatGPT, Gemini, and Claude AI to semantically reverse engineer legacy database software applications.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a very old, dusty library of books written in a language that no one speaks anymore. You don't have the dictionary, and the authors are long gone. Your goal isn't just to copy the letters on the page (that's easy); your goal is to understand the story, the logic, and the rules the author was trying to follow.
This paper is about a team of researchers trying to use three different "super-intelligent robots" (ChatGPT, Gemini, and Claude) to read these old computer programs and explain what they actually do.
Here is the breakdown of their experiment using simple analogies:
The Mission: Decoding the "Ghost" in the Machine
The researchers had old software (called "legacy code") that acted like a bouncer at a club. This bouncer had specific rules:
- The Geography Rule: "If a city is the capital of a country, it must belong to that country."
- The Royal Rule: "Two kings can't rule the same country at the same time, unless they are married or parent/child."
The problem? The code was written in an old style (VBA) with no comments explaining why it was written that way. The researchers wanted to see if AI could look at the code and say, "Ah, I see! This code is enforcing the rule that capitals must match countries."
The Three Robots (The AI Tools)
The researchers tested three different AI "detectives":
1. ChatGPT: The Thoughtful Tutor
- How it worked: ChatGPT was like a smart student who really wanted to get it right. When the researchers gave it the first simple rule, it understood perfectly.
- The Mistake: When they gave it the complex "Royal Rule," ChatGPT initially got the story wrong. It thought the code was rejecting a marriage, when actually, the code was rejecting a change to an existing marriage.
- The Recovery: When the researchers corrected it, ChatGPT didn't just say "Okay." It said, "Oh! I see my mistake. I was looking at the wrong part of the logic. You are right: it's not about the state of the marriage, but about the transition of changing the spouses." It learned from the correction and explained why it was wrong.
- Verdict: The best at understanding the deep meaning and admitting when it messed up.
2. Claude: The Elegant Poet
- How it worked: Claude was very polite and appreciated the mathematical beauty of the code. It understood the simple rules immediately and praised the researchers for their clever logic.
- The Mistake: Like ChatGPT, it also got the complex "Royal Rule" wrong at first. It made a confusing claim that "concurrent reigns are impossible for a married couple," which didn't make sense.
- The Recovery: When corrected, it admitted the error and finally understood the logic. However, it was very slow to talk to (the researchers had to wait hours between questions), making the process feel like a very slow, formal tea party.
- Verdict: Very good at understanding, but slow and a bit rigid.
3. Gemini: The Over-Enthusiastic Artist
- How it worked: Gemini was the most dramatic. Instead of saying "This code checks if a city belongs to a country," it would say things like, "This is a Bio-Logic Self-Healing sensor detecting an Ethical Rupture!" It used too many metaphors (bacteria, limestone, labyrinths) and tried to sell the researchers on using different, modern tools instead of analyzing the old code.
- The Mistake: It got the simple math wrong (calling a function a "surjection" when it was an "injection") and, when corrected, it pretended to understand but kept making the same error.
- Verdict: Too flowery and verbose. It got lost in its own metaphors and couldn't stick to the facts.
The Big Discovery: The "State" vs. The "Change"
The most important lesson from the paper is a subtle but huge difference in logic:
- State: Is the current situation valid? (e.g., "Is this marriage legal?")
- Transition: Is the action of changing the situation valid? (e.g., "Can I swap the husband for someone else?")
The researchers found that the AI tools were great at spotting the "State" (the simple rules). But when it came to the complex "Transition" rules (the Royal Rule), they all got confused. They thought the code was stopping a marriage from happening, when it was actually stopping a change to a marriage.
The Final Conclusion
The researchers concluded that:
- AI is a great assistant for small puzzles: If you give the AI a tiny piece of code, it can explain it very well.
- AI is not a replacement for experts yet: If the code is a bit bigger or more complex, the AI might confidently explain the exact opposite of what the code is doing.
- Math matters: The researchers noted that the AI tools worked best when the humans guiding them knew their math (like set theory and logic). The AI needs a human who speaks "logic" to keep it on the right track.
In short: These AI tools are like very bright interns. They can read the manual and explain the basics perfectly, but if you give them a tricky, real-world scenario, they might confidently tell you the wrong story. You still need a skilled human engineer to double-check their work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.