Tactile: Giving Computer-Using Agents Hands and Feet
Tactile is an open-source tool layer that enhances computer-using agents by converting heterogeneous UI evidence into semantic, verifiable action objects, thereby replacing brittle coordinate-based interactions with a reliable observe-ground-act-verify loop that significantly improves task success rates across multiple models.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a super-smart robot how to use a computer. Right now, most of these robots are like a person trying to drive a car while wearing thick, foggy goggles and blindfolds. They can only see the screen as a flat picture (a screenshot). To click a button, they have to guess the exact X and Y coordinates, like shouting "Click at 450, 200!" and hoping they hit the right spot. If the button moves or the screen changes slightly, the robot gets confused and crashes. This is the current state of "computer-use agents": they are smart, but their hands are clumsy because they are working in the dark, relying only on what they can see in a photo.
But what if the computer could whisper its secrets directly to the robot? Modern operating systems already have a hidden layer of information designed for humans who need help, like screen readers for the blind. This layer tells you exactly what a button is, what it does, and whether it's ready to be pressed. The big question scientists are asking is: Can we give our robot agents a direct line to this secret layer so they don't have to guess? If we can, it might mean robots that don't just "look" at screens, but actually "understand" them, making them much better at doing real work like scheduling meetings or organizing files.
Giving Robots Hands and Feet: The TACTILE Project
Meet TACTILE, a new open-source tool that acts like a translator and a set of super-hands for computer-using agents. The researchers behind TACTILE noticed that while robots are getting smarter, their ability to actually do things on a computer is still a bit shaky. They often treat a computer screen like a painting: they look at it, guess where to poke, and hope for the best. TACTILE changes the game by giving the robot a direct connection to the computer's internal "accessibility" system—the same system that helps humans with disabilities navigate screens.
Think of the old way of doing things like trying to find a specific book in a library by looking at a blurry photo of the shelves. You might guess the book is on the third shelf, second from the left, but if someone moved it an inch, your guess is wrong. TACTILE is like giving the robot a magical list that says, "The book you want is 'History of Space,' it's currently on Shelf 3, it's a hardcover, and you can pull it out." It turns the messy, visual chaos of a screen into a clean, organized list of things the robot can actually touch and use.
How TACTILE Works: The "Ladder" of Clues
The paper introduces a clever strategy called an "operating ladder." Instead of forcing the robot to choose between "seeing" (looking at the screen) and "reading" (using the computer's internal data), TACTILE lets the robot climb a ladder of clues to find the best answer:
- The Top Rung (Accessibility): If the computer says, "I have a button here called 'Send' that is ready to be pressed," TACTILE grabs that info immediately. This is the strongest, most reliable clue.
- The Middle Rung (OCR): If the computer is silent about a specific part, TACTILE looks at the screen, reads the text with its eyes (OCR), and says, "Okay, I see the word 'Send' right here." It's a good backup, but not as precise as the internal data.
- The Bottom Rung (Visual Fallback): If the computer is totally opaque (like a video game or a custom drawing), TACTILE falls back to just looking at the picture and guessing coordinates, just like the old robots did.
The magic is that TACTILE doesn't just pick one; it combines them. It prefers the internal "whisper" from the computer but knows when to switch to "looking" if the computer is being stubborn. It also keeps a detailed diary of every step, so if something goes wrong, we can look back and see exactly what the robot saw, what it guessed, and where it got stuck.
What They Found
The researchers tested TACTILE on a set of 96 different computer tasks, ranging from simple clicks to complex workflows, using four different types of AI agents (Codex, Claude Code, OpenCode, and Goose). The results were promising but not a magic cure-all.
When they added TACTILE to the mix, the robots got significantly better at finishing their tasks. For the Codex agent, the success rate jumped from 41.1% to 50.0% overall. The improvement was even bigger on tasks where the computer was friendly and shared its internal secrets (accessibility-adapted tasks), where success rose from 45.2% to 55.3%. Even on the harder tasks where the computer hid its secrets, the robots still improved, going from 27.8% to 33.3%.
The study suggests that giving robots access to this structured, "hands-on" layer makes them much more reliable. It proves that we don't just need smarter brains (better AI models); we also need better bodies (execution tools) that let those brains interact with the world in a meaningful way.
What TACTILE Is NOT
It's important to note what TACTILE doesn't do. It doesn't replace the need for robots to look at screens; sometimes the computer's internal data is broken, missing, or just plain wrong. In those cases, TACTILE knows to fall back to visual clues. It also isn't a "learned" policy that teaches the robot how to think; it's a tool layer that makes the robot's existing thoughts easier to act upon. The researchers are careful to say these results are preliminary and suggest that while this is a big step forward, there is still work to be done to make these agents perfect.
Why It Matters
The paper ends with a playful but profound idea: "Agent-ready software should be human-accessible software." If we build computers that are easy for robots to use, we are also building computers that are easier for humans with disabilities to use. By treating accessibility data as a shared infrastructure, TACTILE shows that helping robots and helping humans can be the same goal. It turns the computer screen from a flat, confusing picture into a world full of objects that can be touched, understood, and verified.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.