SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis
This paper introduces SwipeGen, a tool for synthesizing diverse human-like swipe interactions, and SwipeBench, a benchmark for evaluating them, to address the rigid swipe execution that limits current GUI agents' performance in real-world scenarios.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where your phone could understand your spoken commands and navigate your apps for you, tapping buttons and typing text just as you would. This is the promise of the graphical user interface agent, a type of artificial intelligence designed to interact with screens the way humans do. For years, researchers have taught these systems to recognize icons, read text, and click on specific targets. However, a critical gap remained: while these agents could point and click with increasing accuracy, they struggled profoundly with the most common gesture of all—the swipe. Whether scrolling through a news feed, adjusting a volume slider, or revealing hidden menu options, the digital hand of the machine often moved with a rigid, mechanical precision that failed to trigger the screen's response. This limitation meant that even the smartest agents could get stuck, unable to complete simple tasks that required the fluid motion of a human finger.
A team of researchers at Fudan University set out to solve this specific problem by asking a fundamental question: why do machines fail at swiping when humans find it so natural? They discovered that the issue was not a lack of intelligence, but a lack of data and a misunderstanding of how swipes work. Existing systems were trained on datasets dominated by simple clicks, treating every interaction as a single point on a screen. But a swipe is not a point; it is a journey. It involves a starting place, a direction, a distance, and a speed. The researchers found that current agents, trying to guess a single coordinate, often missed the mark entirely because they did not understand that the speed of a swipe or the exact starting point could change the outcome. To fix this, they built a new tool called SwipeGen, an automated system designed to teach machines how to swipe by generating thousands of examples of human-like gestures.
The process began by letting the system explore mobile apps on its own, just as a user would. It navigated through screens, identifying areas that could be scrolled, such as lists of photos or rows of icons. Once it found a scrollable area, the system did not just guess; it systematically tried different ways to swipe. It tested swipes starting from the center of a screen, from the edges, moving up, down, left, or right, and at different speeds. Crucially, the system watched the screen before and after each attempt. If the swipe caused the content to move or change, the system recorded it as a success, capturing the exact coordinates, direction, and duration that made it work. If the screen did not react, the attempt was discarded. This cycle of trying, watching, and recording allowed the team to build a massive library of successful swipe interactions, complete with natural language descriptions explaining what each gesture achieved.
With this new library of data in hand, the researchers trained a new version of their artificial intelligence model, which they named GUISwiper. They taught the model not just to find a button, but to understand the physics of a swipe. The results were striking. When tested on a new set of apps that the model had never seen before, GUISwiper succeeded in performing swipe actions more than two and a half times better than previous state-of-the-art models. While older models managed to complete only about a quarter of the swipe tasks correctly, the new model succeeded in over 60 percent of them. More importantly, the researchers proved that teaching the machine to swipe did not make it worse at other tasks. The model remained just as good at identifying buttons and reading text, showing that it could learn complex gestures without losing its ability to understand the screen's layout.
The study also highlighted that the biggest hurdle for these agents was not understanding what to do, but knowing how to do it. The researchers found that when the model failed, it was usually because it started the swipe in the wrong place or moved too slowly to trigger the screen's response, rather than because it chose the wrong direction. By focusing on the execution details—the precise start, end, and speed of the motion—they bridged the gap between a machine's rigid logic and the fluid intuition of a human user. This work suggests that for artificial intelligence to truly master the digital world, it must learn not just to see, but to move with the same nuance and timing as the people it is designed to serve. The ability to swipe effectively is no longer just a minor feature; it is a fundamental requirement for any agent that hopes to navigate our screens with the ease of a human hand.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.