Cached accessibility-tree control and code-enforced verification for design entry through a desktop application's user interface

In the world of software, there is a persistent divide between how humans interact with computers and how machines talk to one another. Humans use graphical interfaces, clicking buttons and typing into boxes that appear on a screen. Machines, however, often prefer to speak a direct, structured language, bypassing the visual layer entirely to send commands straight to the underlying code. When a piece of software lacks this direct language, it becomes a "black box" to automation tools. This is a common hurdle for robotic process automation, a field where software bots are trained to perform repetitive tasks by mimicking human actions on a screen. The challenge is twofold: first, the bot must find the right button or menu item in a complex, shifting landscape of windows; second, it must know if the action it just took actually worked. Without a direct line of communication, the bot is essentially blind, guessing whether a click landed in the right spot or if a new window appeared as expected.

A researcher named Tiago Miranda de Leao tackled this problem by building a system to automate a specific, complex task: designing kitchen cabinets using a popular Windows application called Mozaik Enterprise. This software does not offer a direct way for outside programs to talk to it, so the researcher's system had to operate entirely through the user interface, just as a human would. The system was designed to solve the two main problems of automation: control and correctness. To handle control, the system first spent time mapping out the entire structure of the application's interface, creating a mental map of where every button and menu lived. Once this map was built, the system stopped looking for things and instead acted on a cached list of coordinates, clicking directly on the known locations of the interface elements. To handle correctness, the system used a vision model to "read" a design drawing and then used strict, pre-written code rules to verify if the computer had actually built what was requested. The code, not the artificial intelligence, made the final decision on whether the work was successful.

The results of this experiment were a mix of promising speed and significant limitations. The system successfully mapped the application's interface, identifying nearly a thousand distinct elements, and stored the positions of about six hundred of them. When the system tried to switch between different tabs in the software, the difference between looking for the tab and clicking a cached location was stark. Finding a tab through the standard method took anywhere from twenty-nine to fifty-seven seconds, a process that involved searching through the application's internal structure every single time. Once the location was cached, however, switching to that same tab took less than a third of a second. This demonstrated that for a stable environment, remembering where things are is vastly more efficient than searching for them every time.

However, the system's ability to adapt was fragile. The cached positions were tied to the exact size and shape of the window when the map was first made. If the window size changed, or if the screen resolution was different, the cached coordinates would point to the wrong place, and the system would fail. The researcher found that the system could not automatically repair itself; if the layout shifted, the entire map had to be rebuilt from scratch. Furthermore, the system did not check if its clicks actually worked. It would report success even if the application did not respond, because it had no way to verify the result without looking at the screen again, which would defeat the purpose of caching.

The verification part of the system, which checked if the design matched the drawing, worked well in theory but faced hurdles in practice. The system was tested with five hundred synthetic cases where errors were deliberately injected into the data. In these tests, the code correctly identified every intended error, such as missing cabinets or wrong appliance types, and never raised a false alarm. It successfully rejected designs that did not match the requirements. Yet, when the system was tested on real, unscripted attempts to build cabinets, the results were less clear. Five recorded attempts to build a full design all failed at an early stage, stopping before the final product could be verified. The logs showed that the system could not correctly count the cabinets it was supposed to build, leading to an immediate halt. The researcher noted that the system had not been tested on a wide variety of real-world drawings, and the few real attempts recorded were incomplete.

The study also highlighted the role of artificial intelligence in this process. The system used an AI model to interpret the design drawings, but the final decision on whether the build was successful was left to the code. This separation was intentional; the code acted as a strict gatekeeper, ensuring that the AI's interpretation was checked against hard rules. The researcher found that while the AI could describe the drawings, it could not be trusted to judge the outcome on its own. The system was designed so that if the AI made a mistake, the code would catch it. However, the records showed that in the few real-world attempts, the system failed before it could even reach the stage where the AI's description was compared to the final result.

Ultimately, this work provides a clear picture of what is possible when automating software that refuses to talk directly to machines. The approach of mapping the interface once and then acting on those memories is fast and effective, but it is brittle. It works only as long as the software looks exactly the same as it did when the map was made. The system proved that code-enforced checks can reliably verify results, but it also showed that without a robust way to handle changes in the interface or to recover from errors, the automation cannot yet replace a human in a dynamic environment. The researcher concluded that while the method works for a specific version of the software on a specific machine, it is not a general solution for all applications. The system remains a proof of concept, demonstrating that automation is possible even without direct access, but it also underscores the difficulty of making such systems robust enough for the real world.