⚡ electrical engineering

Freehand Sketching for End-User Programming of Robot Swarms

This paper presents and evaluates a freehand sketching interface that enables non-expert users to intuitively program robot swarms to form specific geometric shapes, achieving high usability scores in a human study without requiring robotics or programming expertise.

Original authors: Riwa Karam, Ian Kuo, Magnus Egerstedt

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Riwa Karam, Ian Kuo, Magnus Egerstedt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of small, autonomous robots working together not as a single machine, but as a collective mind. These swarms can move through forests to find lost hikers, monitor air quality in cities, or perform complex tasks that would be impossible for a single unit. For these groups to be truly useful, however, they need to be able to take instructions from ordinary people, not just from engineers who speak the language of code and control theory. The challenge has always been how to tell a swarm of dozens of independent machines exactly what shape to form or where to go without getting bogged down in technical details. If a person wants the robots to arrange themselves into a specific pattern, they should not have to program each robot individually or memorize a list of complex commands.

A new study explores a surprisingly simple solution to this problem: freehand drawing. The researchers asked whether a person could simply sketch a shape on a tablet and have a swarm of robots instantly understand the intent and arrange themselves accordingly. This approach treats the drawing not as a picture to be displayed, but as a set of instructions for the robots to follow. The system is designed to look at the lines a person draws, figure out the key points that define the shape, and then automatically decide which robot should go where to recreate that shape in the real world. The goal is to create a bridge between human imagination and robotic execution that requires no prior training in robotics or computer science.

To test this idea, the research team built a system that turns a sketch into a physical formation. When a user draws a shape, such as a letter or a symbol, the computer first identifies the important corners and curves of that drawing. It then translates these points into a set of target locations within the physical space where the robots operate. The system must then solve a logistical puzzle: it has a specific number of robots available, and it needs to assign each robot to one of the target points in the most efficient way possible. The software calculates the best match so that the robots do not have to travel unnecessarily far to get into position. Once the assignments are made, the robots move toward their targets, communicating with their neighbors to maintain the correct distances and angles, effectively locking themselves into the desired shape.

One of the most subtle but critical parts of this process involves the direction of the shape. In mathematics, a shape and its mirror image often look identical if you only measure the distances between points. For example, if a person draws the letter "a," a mirror image of that letter also satisfies the same distance rules between its parts. However, a mirrored "a" looks different and might not be what the user intended. The researchers developed a method to ensure the robots form the shape with the correct orientation, preventing them from accidentally creating a reflected version of the drawing. This ensures that the final formation looks exactly like the sketch the user made, preserving the meaning of the symbol.

The team put this system to the test with twenty human participants. These individuals, ranging from undergraduate students to researchers, were asked to draw whatever shapes they wanted on a tablet while watching a group of robots on a testbed floor. The participants received only basic instructions on how to use the drawing tool; they were not taught how the robots worked, nor were they given a list of shapes to draw. They simply sketched, and the swarm responded. Across the session, the participants generated forty-two different geometric shapes. In every single case, the robots successfully moved from their starting positions to form the exact shape the user had drawn. The system handled the translation of the drawing into robot movements automatically, filtering out points that were too close together and assigning the available robots to the correct spots.

The results of the study suggest that this method is highly effective for non-experts. After interacting with the system, the participants filled out a standard questionnaire designed to measure how easy a system is to use. The group achieved an average score of 84.25 out of 100, a number that indicates a high level of perceived usability. Most participants found the system easy to learn and felt confident using it without needing technical support. Many of them chose to draw more than the required number of shapes, indicating that the experience was engaging and intuitive. The study showed that people could communicate complex spatial ideas to a robot swarm simply by drawing, without needing to understand the underlying mechanics of how the robots coordinate or move.

While the study was conducted with a group that included many people with backgrounds in science and engineering, the interface itself required no such expertise. The researchers noted that the system worked well even when the drawings were imperfect or the shapes were complex. The robots were able to converge on the target formations, maintaining the correct geometry and orientation every time. This success points to a future where human-robot collaboration could become much more accessible. Instead of requiring users to learn a new language of commands, they could simply use the universal language of drawing to direct a swarm of machines. The study confirms that freehand sketching is a viable and powerful way to program robot swarms, turning a simple drawing into a precise, physical reality.

Technical Summary: Freehand Sketching for End-User Programming of Robot Swarms

Problem Statement
As robot swarms expand into domains requiring interaction with non-expert users (e.g., education, public demonstrations, art), a significant challenge arises: how can individuals without expertise in robotics, control theory, or programming effectively communicate collective intent to a multi-robot system? Existing Human-Swarm Interaction (HSI) interfaces often focus on navigation, trajectory generation, or task allocation, or they rely on predefined commands and gestures. There is a lack of interaction paradigms that allow users to directly specify the geometric shape a swarm should collectively realize through intuitive, unconstrained inputs. Furthermore, while computational methods exist to translate sketches into formations, the efficacy of freehand sketching as an interaction abstraction for end-users remains unvalidated, particularly regarding usability and learnability for non-experts.

Methodology
The paper proposes and evaluates a "sketch-to-swarm" framework that integrates human spatial specification with autonomous swarm realization. The methodology consists of three primary technical components:

Sketch Processing and Formation Construction:
- User input is captured as a freehand drawing image ( $I$ ).
- The system applies Laplacian of Gaussian (LoG) edge detection to extract contours, reducing sensitivity to input variations.
- Shi-Tomasi feature detection identifies critical points (corners) along these contours to generate a compact set of candidate formation nodes.
- These points are scaled from image coordinates to the physical robot workspace and filtered based on a minimum separation distance ( $d_{min}$ ) to ensure compatibility with physical robot constraints.
- The resulting $n$ target points ( $q_i$ ) form a target configuration matrix $Q$ . A rigid formation graph $G=(V, E)$ is constructed where edges represent desired inter-agent distances ( $d_{ij}$ ). The graph is designed to be infinitesimally rigid to preserve shape up to global translation and rotation.
Robot-to-Formation Assignment:
- The system solves a binary linear assignment problem to map $N$ available physical robots to the $n$ target nodes.
- The cost function minimizes the Euclidean distance between robot positions ( $p_j$ ) and target points ( $q_i$ ).
- The Hungarian algorithm is employed to obtain a globally optimal assignment, ensuring each robot is assigned to exactly one node.
Reflection-Free Distributed Formation Execution:
- Once assigned, robots execute a distributed distance-based formation control law. Each robot adjusts its position based on relative distances to neighbors to satisfy the edge constraints of the rigid graph.
- Critical Innovation: Since distance constraints are invariant under reflection, the swarm could theoretically converge to a mirrored (reflected) version of the intended shape (e.g., a backward "a"). To prevent this, the assignment process incorporates a reflection-free result. By preserving the sign of the orientation angles formed by triplets of non-collinear points during the assignment, the system guarantees that the realized formation maintains the correct semantic orientation of the user's sketch, excluding unintended reflections while allowing global translation and rotation.

Key Contributions
The paper makes three specific contributions:

Interaction Abstraction: It introduces a sketch-based interface where non-expert users communicate formation-level intent via freehand drawings, eliminating the need for conventional programming or predefined shape libraries.
Human Study: It presents a user study with 20 participants evaluating the perceived usability and learnability of the system.
Physical Validation: It demonstrates the system's efficacy through 42 self-specified formations on the physical Robotarium testbed, all of which successfully converged to the user-specified geometries.

Results
The study involved 20 participants (students and researchers from various engineering and science backgrounds) who generated 42 geometric shapes.

Usability Metrics: The interface achieved a mean System Usability Scale (SUS) score of 84.25 (SD = 15.24), significantly exceeding the standard reference value of 68 for average usability. This indicates high perceived usability.
Learnability: Participants rated the system as easy to learn (Item 7 mean: 4.55/5) and reported high confidence in using it (Item 9 mean: 4.40/5).
System Performance: All 42 submitted formations converged to the target configuration within a strict error tolerance ( $| \|p_i - p_j\|_2 - d_{ij} | \le 0.025$ m). No unintended reflected realizations were observed, confirming the effectiveness of the reflection-free assignment strategy.
Engagement: 65% of participants voluntarily performed more than the single required execution, generating multiple formations.

Significance and Claims
The paper claims that freehand sketching serves as an intuitive interaction abstraction for human-swarm collaboration, enabling non-experts to specify collective behaviors without reasoning about resource assignment, swarm dynamics, or low-level control parameters. The results support the feasibility of this paradigm for accessible end-user programming of robot swarms.

The authors maintain a modest scope regarding their claims:

The evaluation was conducted with a population primarily composed of university students and researchers, which may not fully represent the broader non-technical public.
The study focused on static planar formations; the paper does not claim applicability to dynamic behaviors or 3D formations without further work.
The system successfully bridges the gap between high-level human intent and low-level swarm execution, but the authors note that the interface may still benefit from refined interaction cues for users who found it less intuitive.

Future work is identified as extending this interaction to broader non-technical populations, swarm behaviors beyond formation control, and continuous human-in-the-loop integration.

More like this