💻 computer science

Hybrid Offline-Online Reinforcement Learning for Sensorless, High-Precision Force Regulation in Surgical Robotic Grasping

This paper presents a sensorless control framework for surgical robotic grasping that combines a first-principles digital twin with a hybrid offline-online reinforcement learning pipeline to achieve high-precision distal force regulation using only proximal measurements, successfully validating its performance in both simulation and hardware experiments.

Original authors: Edoardo Fazzari, Omar Mohamed, Khalfan Hableel, Hamdan Alhadhrami, Cesare Stefanini

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Edoardo Fazzari, Omar Mohamed, Khalfan Hableel, Hamdan Alhadhrami, Cesare Stefanini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pick up a delicate grape with a pair of robotic tongs, but there's a catch: you can't see the grape, and you can't feel it.

The tongs are controlled by motors located far away at the base (like the handle of a long stick), connected to the tips by thin, stretchy wires (tendons). As you move the handle, the wires stretch, rub against the inside of the tube, and twist. By the time the motion reaches the tips, it's messy and unpredictable. If you squeeze too hard, you crush the grape. If you squeeze too lightly, it slips away.

This is the exact problem surgeons face with robotic surgery tools. The paper you shared presents a brilliant solution: Teaching the robot to "feel" without actually having sensors.

Here is how they did it, broken down into simple steps:

1. Building a "Digital Twin" (The Virtual Simulator)

First, the researchers didn't just guess how the robot works. They built a super-accurate virtual clone (a "digital twin") of the surgical tool inside a computer.

The Analogy: Think of this like a flight simulator for a pilot. They programmed the computer to know exactly how the wires stretch, how the friction feels, and how the motors spin. It's so realistic that if you tell the virtual robot to squeeze, the computer knows exactly how much force the tip will apply, even though there are no sensors there.

2. The "Oracle" (The Perfect Teacher)

To teach the robot, they needed a perfect example to follow. But the robot is so complicated that if you just let it try to learn by trial and error, it would break things or fail constantly.

The Analogy: Imagine trying to teach a child to ride a bike. You wouldn't just let them fall over a thousand times. Instead, you'd have a professional cyclist (the Oracle) ride perfectly alongside them first.
In the computer, they used a super-smart math algorithm (called CMA-ES) to act as this "Oracle." It calculated the perfect sequence of electrical signals to send to the motors to squeeze the grape exactly right, every single time. It generated thousands of hours of "perfect practice" data.

3. The "Student" (The AI Learner)

Now, they needed a student to learn from the Oracle's data without ever needing to touch the real robot again (at first).

The Analogy: This is like a student watching a master chef cook a million times on video, memorizing every move, but never actually stepping into the kitchen.
They used a technique called Offline Reinforcement Learning. The AI studied the Oracle's perfect data and learned a "policy" (a set of rules). It learned: "When I see the motor is moving this fast and the wire feels this tight, I should send this specific voltage to get the perfect squeeze."

4. The "Fine-Tuning" (The Real-World Practice)

The student was great at the video game, but the real world is messy. The wires might be slightly different, or the friction might change.

The Analogy: This is like the student finally stepping into the kitchen to cook the dish. They start with the knowledge from the video, but they make tiny adjustments based on how the real pan feels.
They let the AI interact with the real robot just a little bit to "fine-tune" its skills. It learned to adapt to the tiny differences between the virtual world and the real world.

The Result: Superhuman Precision

The final result is a tiny, smart computer program (the "brain" of the robot) that is only about the size of a small app.

No Sensors Needed: It doesn't need a pressure sensor at the tip. It just looks at the motors and the wires and knows how hard it's squeezing.
Incredible Accuracy: In tests, the robot kept the squeezing force within 1% of the target. That's like holding a grape so gently that it doesn't bruise, but firmly enough that it doesn't roll away, even while the robot is moving around quickly.
Fast: It thinks 26,000 times a second, which is fast enough for real-time surgery.

Why This Matters

Previously, to get this kind of control, engineers had to add expensive, fragile sensors to the tiny tips of the tools, which made them harder to sterilize and more likely to break.

This paper shows that if you understand the physics well enough and use smart AI, you don't need extra hardware. You can teach a robot to have a "golden touch" just by giving it a perfect virtual model and letting it learn from the best. It's a cheaper, simpler, and safer way to make surgical robots that are gentle enough to handle the most delicate tissues in the human body.

1. Problem Statement

The paper addresses the critical challenge of precise grasp force regulation in tendon-driven surgical robotic instruments (specifically the da Vinci Xi end-effector).

Core Difficulty: In proximally actuated systems, the relationship between motor commands and distal grasp force is governed by complex, nonlinear couplings involving motor dynamics, transmission compliance (tendon elasticity), friction, and distal mechanics.
Limitations of Existing Solutions:
- Distal Sensing: Adding force sensors at the instrument tip increases hardware complexity, sterilization difficulties, and cost, while often failing to fit within confined surgical geometries.
- Analytical Compensation: Model-based approaches often rely on simplified assumptions (e.g., constant friction, linear elasticity) that degrade performance under dynamic motion and varying loads.
Goal: Achieve high-precision distal force control (maintaining a specific grasp force) during dynamic jaw motion using only proximal measurements (motor currents, positions, voltages) without distal sensors or complex analytical inversion.

2. Methodology

The authors propose a sensorless control framework that integrates high-fidelity physics modeling with a structured Hybrid Offline–Online Reinforcement Learning (RL) pipeline.

A. Physics-Consistent Digital Twin

A first-principles digital twin of the da Vinci Xi grasping mechanism was developed to serve as the training environment.

Modeling: It is formulated as a coupled differential-algebraic system capturing:
- Electrical Dynamics: DC motor equations (inductance, resistance, back-EMF).
- Transmission Mechanics: Tendon compliance modeled as linear axial springs with lumped mass to capture inertial effects at high bandwidth.
- Jaw Dynamics: Rotational inertia, friction (viscous and Coulomb), and force balance.
Consistency: The model ensures energy flow consistency from electrical input to mechanical output, avoiding artificial tuning terms.
Simulation: To handle system stiffness, a fixed timestep of $2 \times 10^{-6}$ s is used, with a control frequency of 1 kHz (500 internal steps per control action).

B. Hybrid Learning Pipeline

To safely learn control policies in this stiff, nonlinear system, the authors introduced a three-stage pipeline:

Oracle Generation (CMA-ES):
- A receding-horizon Covariance Matrix Adaptation Evolution Strategy (CMA-ES) acts as an "oracle."
- It optimizes voltage sequences over a short horizon to minimize a weighted cost of trajectory tracking error and force regulation error.
- Purpose: Generates high-quality, dynamically feasible "expert" trajectories to overcome the instability of learning from scratch.
- Data: This process generated a dataset of ~1 million episodes (3 million transitions) stored in the Minari framework.
Offline Policy Learning (Implicit Q-Learning - IQL):
- IQL is used to train a policy on the static oracle dataset.
- Advantage: IQL avoids out-of-distribution (OOD) action evaluation by using expectile regression, ensuring a stable initialization without unsafe exploration in the real system.
- Outcome: A policy that closely imitates the oracle's behavior.
Online Refinement (Twin Delayed DDPG - TD3):
- The IQL policy is used to initialize a TD3 agent for online fine-tuning.
- Purpose: Adapts the policy to on-policy dynamics and corrects for any distributional shifts between the simulated oracle data and the true system behavior.
- Process: The agent interacts with the environment (simulated or real) to refine the policy while maintaining grasp stability.

C. Control Objective

Inputs: Proximal measurements (motor positions, velocities, currents, jaw angle/velocity, and force error).
Outputs: Motor voltage commands ( $V_1, V_2$ ).
Reward Function: Penalizes deviations from the reference grasp force ( $F_{ref} = 12.57$ N) and the reference jaw trajectory (multi-harmonic functions emulating surgical motions).

3. Key Contributions

High-Fidelity Digital Twin: Development of a unified differential-algebraic model of the da Vinci Xi end-effector that captures coupled electromechanical and tendon dynamics, formulated specifically as an RL environment.
Hybrid RL Pipeline: Introduction of a novel three-stage training strategy (CMA-ES Oracle $\to$ Offline IQL $\to$ Online TD3) that enables safe, stable, and data-efficient learning in stiff, nonlinear systems without unsafe exploration.
Sensorless Precision: Demonstration that high-precision distal force regulation can be achieved without distal force sensors, relying solely on model-informed learning and proximal actuation.
Real-Time Deployability: The resulting policy is compact (~71k parameters) and executes at kilohertz rates, making it suitable for embedded surgical controllers.

4. Results

Simulation Performance

Force Regulation: The learned policy maintained grasp force within 1% of the desired reference during multi-harmonic jaw motion.
Tracking: Achieved an average angular tracking error of ~5.16%.
Comparison: The policy performed nearly as well as the CMA-ES oracle but with significantly lower computational cost during inference (feedforward mapping only).

Hardware (Physical) Experiments

Setup: Experiments were conducted on a physical da Vinci Xi setup using a load cell to measure grasp force via a Mylar leaf slip test.
Accuracy:
- Average Force Error: 3.84% (initial) and 3.24% (after minor parameter tuning).
- Worst-Case Error: 5.67% (initial) and 5.08% (tuned).
- Best-Case Error: 1.84% / 1.81%.
Sim-to-Real Transfer: The results validate that the policy trained in simulation transfers effectively to the physical robot without distal sensing.

Computational Efficiency

Latency: 0.0386 ms per step on a CPU (MacBook Pro M4), corresponding to an inference rate of ~26 kHz.
Memory: ~279 KB (FP32), well within the constraints of modern microcontrollers.

5. Significance

This work represents a significant advancement in surgical robotics by demonstrating that model-informed learning can replace hardware augmentation for complex control tasks.

Clinical Impact: It offers a scalable, mechanically compatible solution that preserves the compactness and sterilizability of surgical instruments by eliminating the need for distal force sensors.
Paradigm Shift: It bridges the gap between purely analytical control (which struggles with nonlinearities) and purely data-driven approaches (which require massive data or unsafe exploration).
Generalizability: The proposed hybrid offline-online framework provides a generalizable paradigm for controlling other coupled, compliant robotic mechanisms where high-fidelity modeling is available but analytical control is intractable.