Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation

Problem Statement

Dexterous in-hand manipulation—repositioning or articulating a grasped object using only finger motion—is a significant challenge in robotics. Current learning-based approaches face distinct bottlenecks:

• Reinforcement Learning (RL): While effective in simulation, RL relies on heavy domain randomization to bridge the sim-to-real gap, which is difficult due to the complex, dynamic nature of contact-rich interactions.
• Imitation Learning (IL): This approach requires extensive demonstration data. However, capturing high-quality in-hand demonstrations is difficult because fingers occlude the object and each other, making it impossible for purely visual recordings (even egocentric views) to capture all contact points. Furthermore, retargeting human data to robots with different kinematics remains challenging.

Both approaches often demand massive modeling, simulation, or data-collection efforts to achieve dexterity. The authors propose an alternative: a lightweight, data-efficient method that learns the relationship between finger actuation and object motion directly on the physical robot without pre-collected data or analytic contact models.

Methodology

The proposed system utilizes an online task-Jacobian estimation approach to control a 17-DoF anthropomorphic robotic hand (the ORCA hand) for in-hand pen writing. The system operates without an analytic hand-object kinematic model, simulation training, or prior demonstrations.

1. System Architecture & Perception

• Hardware: A tendon-driven ORCA hand grasps a pen held in a custom soft TPU sleeve (increasing diameter for a secure grip). The wrist, ring, and pinky fingers are fixed; control is limited to the 10 joints of the thumb, index, and middle fingers.
• Perception: A standard webcam tracks an ArUco marker on the pen and markers on the desk to define the paper frame. The pen-tip position is recovered via a fixed offset and smoothed using a Kalman filter with outlier rejection.
• Control Loop: The system runs at a perception-limited rate of approximately 15 Hz. It operates in a "perceive–estimate–act" loop where the pen-tip trajectory is compared to a reference, and joint commands are generated based on the estimated Jacobian.

2. Online Jacobian Estimation
The core of the method is a recursive least-squares (RLS) estimator that learns the command-space task Jacobian ($J$), which maps joint velocities ($\dot{q}$) to pen-tip velocities ($\dot{x}$).

• Update Mechanism: The estimator updates $J$ using commanded joint increments ($\Delta q_{cmd}$) rather than measured velocities, as the command-to-motion mapping is operationally relevant and less noisy.
• Adaptivity: A forgetting factor ($\lambda$) allows the estimator to track slowly changing dynamics (e.g., grip slippage). The covariance matrix is diagonalized to reduce computational load.
• Excitation Phase: To bootstrap the estimation, the system performs a 18-second initialization phase where the hand sweeps through six predefined grip poses. This ensures the estimator observes a well-conditioned set of $(\dot{q}, \dot{x})$ pairs before tracking begins.

3. Control Law
The controller uses a damped right pseudo-inverse of the estimated Jacobian ($J^+$) to compute joint commands:
$$\Delta q = (J^+ v_{cmd} + k_{pb} N^\perp (q_0 - q)) \Delta t$$

• Task Tracking: The first term ($J^+ v_{cmd}$) drives the pen tip along the desired trajectory using a PID controller with feedforward.
• Grip Stabilization: The second term projects a posture stabilization force through the approximate nullspace ($N^\perp$) of the Jacobian. This pulls the hand back toward an initial stable grip pose ($q_0$) without interfering with the primary writing task, preventing the grasp from destabilizing.

Key Contributions

1. Embodied In-Hand Writing: The first demonstration of an anthropomorphic hand writing arbitrary single-stroke trajectories (letters and shapes) on paper through purely in-hand finger motion, achieving sub-millimeter precision.
2. Lightweight, Model-Free Control: A control architecture that requires no analytic contact models, simulation training, or pre-collected demonstrations. It relies solely on real-time interaction and online estimation.
3. Generalizability: The same estimator/controller formulation was successfully applied to three different robotic hand systems (one physical ORCA hand and two simulated hands: Shadow Hand and Wuji Hand 2), demonstrating embodiment independence.
4. Open Source: The code and simulation implementations are released as open source.

Results

• Tracking Accuracy: Across 38 runs (22 in air, 16 on paper), the system achieved a mean in-plane tracking error of 0.64 ± 0.10 mm. The best runs reached ~0.39 mm (air) and ~0.57 mm (paper). The 95th percentile error was approximately 1.4 mm.
• Robustness: The system maintained accuracy over long horizons (e.g., writing the entire alphabet in a continuous 31-minute run) without performance degradation. It could recover from large transient errors (e.g., a ~20 mm spike) without re-initialization.
• Ablation Studies:
  • Jacobian Freezing: Freezing the Jacobian estimate after initialization caused rapid divergence or degraded accuracy. Continuous online updating was essential for robustness against disturbances like grip slippage.
  • Grip Regularization: Disabling the nullspace posture stabilization term led to grasp instability and failure in 60% of runs, proving its necessity for maintaining a secure grip.
  • Speed: Writing speed was deliberately kept slow at a base rate of $8 \times 10^{-4}$ m/s (0.8 mm/s). Increasing the speed to 2×, 3×, or 4× nonlinearly degraded accuracy and robustness, indicating a trade-off between throughput and stability.
• Simulation: The method was ported to MuJoCo for the Shadow Hand and Wuji Hand 2. Despite controlling 3D coordinates (unlike the 2D physical control) and lacking the compliant sleeve, the Shadow Hand achieved 0.17 mm RMSE and the Wuji Hand 2 achieved 1.48 mm RMSE, confirming the approach's applicability across different kinematics.

Significance and Claims

The authors claim this work demonstrates that online, real-time learning is a viable route to dexterous in-hand manipulation, particularly where lightweight, adaptive controllers are required.

• Alternative to Heavy Methods: The paper positions this approach as a computationally simple and data-efficient alternative to RL and IL, which typically require massive modeling or data-collection efforts.
• No Precise Model Needed: The results suggest that a lack of a precise analytic model or large datasets is not a fundamental barrier to achieving real dexterity; rather, rapid embodied learning can suffice.
• Practicality: The system runs on a standard laptop CPU using a simple webcam, making it promising for applications with limited compute resources, such as onboard prosthetic control.

Limitations Acknowledged by the Authors:

• The system currently controls only 2D in-plane motion; out-of-plane ($z$) drift is uncontrolled and absorbed by a compliant paper mount. It cannot yet write on rigid surfaces or perform multi-stroke tasks (like the dot on an 'i') without arm assistance.
• It cannot handle contact-discontinuous tasks (e.g., re-grasping the pen).
• Writing speed is currently slow compared to human writing, and accuracy is bounded by vision system jitter.