💻 computer science

Vision-Force Admittance Learning for Peg Insertion into a Movable Hole

This paper proposes a Vision-Force Admittance Learning (VFAL) framework that fuses asynchronous visual feedback from foundation models with high-frequency force control to enable robust, millimeter-precision peg-in-hole insertion into moving holes in dynamic environments.

Original authors: Yuzhong Chen, Yongqing Liang, Yunzhi Xu, Irving Fang, Chase Kidder, Hui-ping Wang, Raihan Haque, Yubiao Zhang, Chen Feng

Published 2026-09-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuzhong Chen, Yongqing Liang, Yunzhi Xu, Irving Fang, Chase Kidder, Hui-ping Wang, Raihan Haque, Yubiao Zhang, Chen Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet, controlled world of a factory floor, a robot arm can be a marvel of precision. It can pick up a screw and drive it into a hole with a steadiness no human hand can match, provided the hole does not move. This is the realm of static manipulation, where the target sits still, and the robot's challenge is simply to find its way to a fixed point. But the real world is rarely so still. Imagine a car part moving along a conveyor belt, or a robot mounted on a vehicle that is driving over uneven ground. In these dynamic environments, the target is in motion, and the robot must not only find the hole but chase it, all while maintaining the millimeter-level accuracy required to fit a peg into a tight space. This is a fundamental conflict: the need for extreme precision clashes with the chaos of a moving target. Traditional robots struggle here because they rely on either slow, high-level vision to see where the hole is, or fast, local touch sensors to feel the moment of contact. Vision is too slow to react to sudden shifts, while touch alone cannot see far enough ahead to plan a smooth path.

Researchers at New York University and General Motors have developed a new way to solve this problem, creating a system that allows a robot to insert a peg into a moving hole with remarkable success. They call their method Vision-Force Admittance Learning. Instead of forcing the robot to choose between seeing and feeling, this system lets the two senses work together in a specific, asynchronous rhythm. The robot uses a camera to estimate where the hole is, but because cameras are slow, this information is treated as a general guide rather than a strict command. Simultaneously, the robot relies on a force sensor at its wrist, which reacts hundreds of times per second to the physical pressure of the peg touching the hole. The innovation lies in how these two streams of data are combined. The fast force sensor makes the immediate, split-second adjustments needed to keep the peg aligned, while the slower camera updates the robot's understanding of the hole's overall position. This visual update acts as a gentle correction, or a regularizer, that keeps the robot from drifting too far off course, even as the force sensor handles the rapid, chaotic vibrations of the insertion.

To test this, the team built a real-world experiment involving a robotic arm holding a peg and a base with a hole that could move. They used a special type of fastener, a pushnut with flexible fins, which is common in industrial assembly. The setup included a camera to watch the scene and a force sensor to feel the contact. They tested the robot in three different scenarios: a hole that stayed still, a hole moving in a straight line, and a hole moving in three dimensions on a motorized slider. The results were striking. When the hole was moving, a robot relying only on vision failed to insert the peg most of the time, while a robot relying only on force sensors also struggled, often losing its way or applying too much force. However, the new system, which fused the two senses, achieved a success rate of 80 percent in the most difficult three-dimensional moving scenario. In the straight-line moving scenario, it succeeded every single time.

The researchers found that the key to this success was not just having both sensors, but how they were timed. They compared their method to a version where the vision and force data were forced to wait for each other, a process that slowed everything down. By letting the fast force sensor drive the motion while the slower vision system updated the goal in the background, the robot could react instantly to physical contact without losing its global sense of direction. They also discovered that the system could recover from mistakes. If the camera lost track of the hole, the robot would safely back away and wait for the vision to return. If the peg got stuck or the force readings indicated a bad angle, the robot could lift the peg and try again. This ability to adapt and recover made the system robust enough to handle the unpredictable nature of a moving target.

The study also explored how different settings affected the outcome. They found that the balance between trusting the camera's position and the force sensor's feedback was critical. If the robot trusted the camera too much, it ignored the physical reality of the peg hitting the side of the hole. If it trusted the force sensor too much, it would drift aimlessly when the hole moved. By finding a middle ground, the robot learned to use the camera to stay on the right path and the force sensor to make the fine adjustments needed to slide the peg in. This approach worked even when the target moved in complex, non-linear ways, proving that the system could handle more than just simple, predictable motion.

Ultimately, this work demonstrates that robots can be taught to perform delicate tasks in environments that are not perfectly still. By combining the slow, broad view of a camera with the fast, sensitive touch of a force sensor, the researchers created a method that mimics how humans might handle a moving object: using our eyes to know where to go and our hands to feel the way. The system does not require the robot to be perfect from the start; it allows for errors and corrects them in real-time. While the current system assumes the robot has already picked up the peg, the ability to insert it into a moving hole is a significant step toward robots that can work alongside humans in dynamic, real-world settings, from assembly lines to mobile vehicles, without needing the environment to stop for them.

Technical Summary: Vision-Force Admittance Learning for Peg Insertion into a Movable Hole

Problem Statement

Precise manipulation in dynamic environments, such as mobile manipulation or assembly on moving production lines, presents a significant challenge in robotics. While static peg-in-hole tasks are well-studied, dynamic scenarios introduce substantial uncertainty due to relative motion between the robot and the target. This uncertainty conflicts with the millimeter-level precision required for successful insertion. Existing approaches face a trade-off: force-based methods offer high-frequency, low-latency control but struggle with large initial misalignments and global context, while vision-based methods provide global context but suffer from high latency and lower precision during contact-rich phases. Furthermore, many industrial robots rely on position control rather than direct force control, complicating the implementation of compliant strategies.

Methodology

The authors propose Vision-Force Admittance Learning (VFAL), a multimodal framework designed to fuse asynchronous visual feedback with a high-frequency force-based model. The system operates in two distinct stages:

1. Approaching Stage

Before contact, the robot utilizes a vision-based pose estimation module (adapted from FoundationPose) to guide the robot toward the moving target. To address the latency inherent in visual processing, the authors design a cross-coupled PD tracker. This tracker accepts target velocity estimates (predicted via a Kalman filter) and outputs expected robot poses and velocities, enabling the robot to track the moving hole with reduced error before contact is established.

2. Contact-Rich Manipulation Stage

Once contact is detected, the system switches to a force-based control loop running at 125 Hz. This stage employs a Linear Model Learning (LML) approach to adaptively model the contact dynamics online.

Force-Based Adaptive Contact Modeling: The system maintains a linear model $G$ that maps the current system state (robot pose, force readings, control input) to predicted forces. This model is updated stepwise using a Kalman filter to handle the non-linearities of dynamic environments within small time intervals.
Asynchronous Visual Referencing: To prevent the force-based model from drifting due to motion shifts or latency, the system incorporates visual pose estimates as a regularization term. Although visual updates occur at a lower frequency (20 Hz), a Kalman filter predicts the target's pose and velocity at the current control cycle.
Control Objective: The control input $u_i$ is optimized to minimize the discrepancy between the expected force (based on the desired insertion profile) and the predicted force, while simultaneously penalizing the positional error between the expected robot pose and the visually estimated target pose. The objective function is:
$\arg\min_{u_i} \|F^{expected}_i - F^{predicted}_i\|^2_2 + \lambda \| \hat{p}^{expected}_i - p_i \|^2_2$
where $\lambda$ balances force exploration and adherence to the visual prior.

Robustness Mechanisms

VFAL includes failure recovery strategies:

Tracking Loss: If visual tracking is lost, the robot retreats to a safe position until tracking is restored.
Insertion Failure: If force peaks indicate a failed engagement (e.g., no corresponding force signature for fastener locking), the robot lifts and reinitializes the process.

Key Contributions

Novel Multimodal Framework: Introduction of VFAL, which asynchronously fuses force and vision to enhance precision and adaptability in peg-in-movable-hole tasks.
Asynchronous Fusion Pipeline: A pipeline that leverages predicted target pose as a regularization term for high-frequency admittance-based control, allowing the system to maintain local linearity assumptions while adapting to global motion.
Real-World Deployment: Successful deployment of the framework on a position-controlled robot (FANUC CRX-10iA/L) demonstrating robust peg insertion into moving bases with varying stiffness and motion patterns.

Experimental Results

The authors evaluated VFAL on a FANUC robot with a force/torque sensor, using pushnuts (deformable fasteners) inserted into fixed, 1D mobile, and 3D mobile bases.

Success Rates:
- 1D Mobile Base: VFAL achieved a 100% success rate, outperforming vision-only (40%), force-only admittance (60%), and synchronous fusion (90%).
- 3D Mobile Base (Stiffer, more demanding): VFAL achieved an 80% success rate. This significantly outperformed vision-only (20%), force-only methods (0-10%), and synchronous fusion (60%).
Precision (Full Wrench MSE): VFAL demonstrated the lowest error in both 1D (13.95) and 3D (9.22) mobile base scenarios compared to all baselines.
Ablation Studies:
- Online Learning: Removing online learning reduced the success rate on the 3D base from 80% to 70%.
- Regularization Weight ( $\lambda$ ): A value of $\lambda=0.1$ yielded optimal performance. Higher values ( $\lambda=1$ ) suppressed necessary force feedback, while lower values ( $\lambda=0.01$ ) weakened visual guidance.
- Controller Frequency: Increasing the robot controller frequency alone did not replicate VFAL's performance gains, confirming that the asynchronous fusion strategy is the primary driver of success.

Significance and Claims

The paper claims that VFAL effectively bridges the gap between the high precision of force control and the global adaptability of vision. By treating visual pose estimates as a regularization term rather than a direct control signal, the system avoids the latency issues that typically plague vision-based dynamic control. The authors demonstrate that this approach allows robots to perform precise assembly tasks on moving platforms without requiring direct force control hardware, making it suitable for industrial position-controlled robots. The work highlights that asynchronous fusion, combined with online model adaptation and failure recovery, is a viable path toward robust manipulation in dynamic, uncertain environments.