MotionDLO: Hybrid Event- and Frame-Based Tracking of Deformable Linear Objects

In the world of robotics, there is a persistent gap between what a machine can see and what it needs to do. Robots excel at moving heavy, rigid objects like metal boxes or car parts, but they struggle with anything soft, flexible, or unpredictable. Think of a wire harness in a car factory, a cable in a server room, or a suture thread in a hospital. These objects are known as deformable linear objects. They twist, bend, and tangle in ways that are impossible to predict perfectly in advance. To manipulate them, a robot must watch them move in real time, updating its understanding of their shape hundreds of times per second. Traditional cameras, which take pictures like a standard film camera, are often too slow for this task. By the time a standard camera captures a clear image of a fast-moving wire, the wire has already moved, leaving the robot with a blurry, outdated picture. This lag causes the robot to miss its target or, worse, to lose track of the object entirely.

To solve this, researchers have begun using a different kind of sensor called an event camera. Unlike a standard camera that takes a full picture at fixed intervals, an event camera works like a nervous system. It only reports changes. If a pixel on the sensor sees light change, it sends a signal immediately. If the scene is still, the sensor stays silent. This allows the camera to react to motion in microseconds, capturing the speed of a moving wire without any blur. However, while these cameras are incredibly fast, they are often poor at telling exactly where an object is or what it looks like in detail. They see the movement, but they struggle to define the shape. The challenge for engineers has been to combine the lightning-fast reaction of the event camera with the precise, detailed vision of a standard camera, creating a system that is both quick and accurate enough for a robot to handle delicate tasks.

A team of researchers has developed a new system called MotionDLO to bridge this gap. Their work focuses on tracking these flexible wires and cables as they move, ensuring the robot knows exactly where every part of the object is at all times. The system works by using two different "eyes" simultaneously. One eye is a standard high-resolution camera that takes pictures to get a clear, detailed view of the wire's shape. The other eye is the event camera, which watches for any movement. The researchers designed the software to switch between these two views based on what the wire is doing. When the wire is still, the system relies on the standard camera to build a perfect, detailed map of its shape. The moment the wire starts to move, the system instantly switches to the event camera. Because the event camera reacts so quickly, it can track the wire's motion as it happens, updating the robot's understanding of the wire's position every 12 milliseconds.

The brilliance of the system lies in how it keeps these two different streams of information working together without getting confused. The researchers found that if they relied only on the fast event camera, the robot might lose track of which wire it was following, especially if there were several wires in the scene. If they relied only on the standard camera, the robot would be too slow to catch up with fast movements. MotionDLO solves this by using the standard camera to establish a solid, accurate starting point and to confirm the identity of the object. Then, as the object moves, the event camera takes over, providing a continuous stream of updates that keep the robot's knowledge fresh. The system is smart enough to know when the object has stopped moving and switch back to the detailed camera to correct any small errors that might have crept in during the fast motion. This creates a seamless loop where the robot never loses sight of the object, regardless of how fast it is moving.

The team tested their system in a realistic industrial setting, using a robot arm to grab and move different types of cables and tubes. They used three specific types of objects: a plastic tube used for air pressure, a shielded communication cable, and a thin power wire. The robot moved these objects at various speeds, including very fast movements that would cause a standard camera to see only a blur. The results showed that MotionDLO could track the objects with remarkable precision. Even when the robot moved the cables at speeds of up to 2,255 millimeters per second, the system maintained an average error of less than half a millimeter. This level of accuracy is crucial because it means the robot can manipulate the wire without damaging it or dropping it. In contrast, other existing methods that rely on standard cameras or simpler event-based tracking failed under these conditions. They either lost track of the object entirely or made mistakes so large that the robot could not complete the task.

One of the most significant findings was that the system could keep track of a single moving wire even when it was tangled with other stationary wires. In a crowded scene with multiple cables, other tracking methods often got confused, mixing up which wire was which or losing the moving one entirely. MotionDLO, however, used the unique ability of the event camera to see only the moving parts. Since the stationary wires did not trigger the event camera, the system could isolate the moving wire from the background clutter and follow it steadily. This ability to maintain a consistent identity for the object over time is a major step forward, as it allows robots to perform complex tasks that require long periods of attention without getting distracted or losing their place.

The researchers also demonstrated that their system works without needing to be trained on specific types of wires. Many modern robot vision systems require thousands of hours of training data to learn how to recognize a specific object. If the robot encounters a new type of cable it has never seen before, it often fails. MotionDLO avoids this problem by using a "zero-shot" approach. It uses a general-purpose model that can understand what a wire looks like based on simple descriptions, rather than having memorized specific examples. This means the system can be deployed in a factory with a new type of cable and start working immediately, without the need for a lengthy training period. This flexibility makes the technology much more practical for real-world industrial use, where the types of materials and tools change frequently.

In their experiments, the team measured the time it took for the system to process each update. They found that the entire process, from seeing the movement to updating the robot's plan, took only 12 milliseconds. This speed is fast enough to allow the robot to react in real time, adjusting its grip or path as the wire moves. The system was able to maintain this performance across all three types of cables tested, from the thick plastic tube to the very thin power wire. The accuracy remained high even when the wires were moving quickly, proving that the hybrid approach successfully combines the speed of event-based sensing with the precision of standard imaging.

The implications of this work extend beyond just moving wires. The ability to track flexible objects in real time opens the door for robots to handle a much wider range of tasks. In the future, this technology could be used to automate the assembly of complex wiring harnesses in cars, route cables in tight server racks, or even assist in delicate medical procedures like stitching. The researchers noted that while their current system works in two dimensions, the next step would be to add depth perception so the robot can understand the full three-dimensional shape of the object. They also plan to test the system in environments with changing light conditions, such as those found in welding or laser-cutting shops, to ensure it remains robust in all industrial settings.

The success of MotionDLO demonstrates that the future of robotic perception lies not in choosing between speed and accuracy, but in combining the strengths of different technologies. By listening to the rapid signals of an event camera while keeping the steady gaze of a standard camera, the system achieves a level of performance that neither could reach alone. This approach offers a practical solution to a problem that has long hindered the automation of tasks involving flexible materials. As robots become more common in factories and homes, the ability to handle the unpredictable and the delicate will be just as important as the ability to lift heavy loads. MotionDLO provides a clear path toward that future, showing that with the right combination of sensors and software, robots can finally learn to handle the world as it really is: messy, moving, and full of flexible things.