Can LLMs Test Terminal User Interfaces?

Imagine you are a video game developer. You've built a beautiful, complex world with mountains, cities, and characters, but you forgot to test if the doors actually open or if the player gets stuck in a wall. In the world of software, this is a nightmare. To prevent it, programmers use "automated testers"—robotic assistants that click buttons, type commands, and try to break the software to find bugs before real users do.

For a long time, we've had two main types of software to test. First, there are Graphical User Interfaces (GUIs), like the apps on your phone or computer. These are like fancy, colorful rooms with labeled doors, windows, and buttons. Testers can easily ask the computer, "Where is the 'Start' button?" and click it. Second, there are Command-Line Interfaces (CLIs), which are like old-school telegraphs. You type a secret code, and the computer replies with text. These are easy to test because they are simple: type this, get that.

But there is a third, sneaky type of software called a Terminal User Interface (TUI). Think of these as "retro-futuristic" apps. They look like the old telegraphs (just text on a black screen), but they act like the fancy rooms. They have moving cursors, pop-up menus, and complex states that change as you type. They are popular with hackers, system administrators, and even AI coding assistants. The problem? We don't really know how to test them properly. They are too complex for the simple telegraph testers, but they lack the "labeled buttons" that the fancy room testers need. This paper asks a big question: Can modern Artificial Intelligence (AI) robots learn to test these tricky text-based apps, or are they just as confused as we are?

---

The Great TUI Detective Story

The researchers behind this paper decided to play detective. They gathered a massive collection of 197 real-world TUI applications—everything from file managers to system monitors—and put them through a rigorous stress test. But first, they had to solve a mystery: How well are these apps tested right now?

They peeked inside the code of these 197 apps and found a shocking secret. Only about 12% of the testing code actually tried to interact with the screen. Even worse, nearly half of the tests that did touch the screen never actually typed a single key! They just checked if the screen looked right when the app first opened, like checking a car's paint job but never turning the engine on. It turns out, most TUIs are being tested with their eyes closed.

So, the team built their own tools to see what would happen if they let AI and random chance loose on these apps. They created a "headless" lab (a computer with no screen, just a virtual terminal) and packaged each app into a special container. They then set up a race between four different strategies:

1. The Random Monkey: A robot that just types random keys as fast as it can.
2. The AI Guide: A smart Large Language Model (LLM) that looks at the screen and decides what to type next.
3. The AI Guide with a Map: The same smart robot, but this time it also figured out the correct "launch codes" (arguments) to get the apps started properly.
4. The AI Scriptwriter: A robot that reads the source code and writes a test plan before starting.

They ran these strategies against the 197 apps, giving each one exactly 600 seconds (10 minutes) to find bugs.

The Results: Who Wins?

The results were surprising and a bit counter-intuitive.

1. The "Smart" Robot isn't the fastest.
When the researchers looked at how many bugs each strategy found per run, the Random Monkey actually found the most crashes. Why? Because it was incredibly fast. In 600 seconds, the random robot could type hundreds of keys. The AI robots, being thoughtful and "smart," spent most of their time thinking and waiting for their answers, so they only managed to type about a dozen keys in the same amount of time.

2. But the "Smart" Robot is much more efficient.
Here is the twist: If you measure how good they are per key press, the AI robots were 13 times better than the random monkey. The random monkey was just throwing darts in the dark and hitting a few by luck. The AI robots were aiming carefully. They were uniquely able to find "input-gated" bugs—crashes that only happen if you type a specific sequence of keys to unlock a hidden menu. The random monkey never figured out the code, but the AI did.

3. The "Launch Code" was the real hero.
The biggest breakthrough wasn't the AI's intelligence; it was its ability to figure out how to start the apps. Many TUIs need specific files or arguments to even open their interface. Without these, they just quit immediately. The strategy that used the AI to automatically derive launch inputs found the most bugs and covered the most code. It turns out, you can't test a car if you don't know how to turn the key.

4. The "Crash" Trap.
The researchers also discovered a major trap in how we usually count bugs. They found that 82% of the "crashes" (programs stopping unexpectedly) were actually just the programs saying, "Hey, I need a file!" or "I was told to stop!" These aren't real bugs; they are just normal behavior. If you just count every time a program stops, you get a false alarm. The team had to build a special "crash detector" that looks at the actual text on the screen to see if it's a real error or just a polite exit. Once they filtered out the noise, they found 179 real, valid bugs across the 197 apps.

5. More Code Coverage $\neq$ More Bugs.
In software testing, people usually believe that if you cover more lines of code, you find more bugs. The paper suggests this is not true for TUIs. They found that the tests that found the most crashes often had lower code coverage. Why? Because finding a crash stops the test immediately! The test gets cut short, so it never gets to cover the rest of the code. This means that for TUIs, counting how many lines of code were touched is a bad way to measure how good a test is.

The Bottom Line

The paper concludes that automated testing for these text-based interfaces is possible, but it's far from solved. No single AI model was the clear winner; in fact, a simple random tester was competitive because it was faster. The real key to success was hybrid strategies: using AI to figure out how to start the app and navigate to the right state, and then using high-speed random testing to stress-test the interface.

The authors also warn us that we need better tools. The current way of testing (just checking if the program stops) is full of false alarms. They released their own tools, tuicov and tuibot, to help others test these apps properly. The message is clear: TUIs are a huge, growing part of our software world, but they are currently the "wild west" of testing. We have the tools to start taming them, but we need to stop relying on old rules and start designing tests that understand the unique, text-based nature of these interfaces.