Robustness-Aware Tool Selection and Manipulation Planning with Learned Energy-Informed Guidance

This paper introduces a robustness-aware framework that jointly selects tools and plans contact-rich manipulation trajectories by leveraging an energy-based metric to optimize for disturbance resilience in robotic tool-use tasks.

The Gist

Imagine you are at a barbecue, trying to serve a slippery meatball. You have a flat spatula and a deep ladle in front of you. Instinctively, you grab the ladle. Why? Because even if your hand shakes or the meatball wobbles, the ladle's deep bowl keeps it safe. The flat spatula? One tiny slip, and the meatball goes flying.

Humans do this kind of "robust" thinking automatically. We subconsciously pick the tool and the grip that is least likely to fail if things go wrong. But for robots, this is incredibly hard. Most robots are programmed just to "get the job done," not to "get the job done without dropping it if the wind blows."

This paper introduces a new way for robots to think like that barbecue chef. It's a system that helps a robot choose the best tool and plan the safest movement to handle objects, even when the world is messy and unpredictable.

Here is how it works, broken down into simple concepts:

1. The Core Idea: "Energy" as a Safety Score

The researchers invented a special way to measure "safety." They call it an Energy-Informed Metric.

Think of it like a video game health bar, but for stability.

• If a robot is holding a fish with a flat spatula, the "health bar" is low. A tiny push (a disturbance) gives the fish enough energy to escape, and it falls.
• If the robot uses a deep shovel, the "health bar" is high. The fish is trapped in a deep pocket. To make it fall, you'd need to apply a massive amount of force (energy) to lift it out.

The robot's goal is to find the tool and the hand position that gives the object the highest possible health bar against falling.

2. The Two-Step Dance

The robot doesn't just guess; it follows a smart, two-step process:

• Step 1: The "Freeze-Frame" Check (Tool Selection)
  Before moving, the robot pauses and asks: "If I had to hold this object right now, which tool and which grip would be the hardest to break?"
  It looks at all its available tools (like a shovel, a hook, or a hanger) and simulates thousands of scenarios to find the one "perfect pose" where the object is most secure. It picks the winner.

• Step 2: The Safe Path (Trajectory Planning)
  Once it knows the "perfect pose," the robot plans a path to get there. But it doesn't just take the shortest route. It plans a path that keeps the safety score high the whole time. It avoids moves where the object might wobble dangerously, even if those moves are faster.

3. The "Cheat Sheet" (Machine Learning)

Calculating this "safety score" for every single move is like trying to solve a million math problems in a second. It's too slow for a real-time robot.

To fix this, the researchers trained a neural network (a type of AI) to be a "cheat sheet."

• Training: They fed the AI millions of examples of tools holding objects and told it, "This is safe, this is unsafe."
• Result: Now, instead of doing complex math, the robot just asks the AI: "How safe is this?" The AI answers instantly. This makes the robot fast enough to plan in real-time.

4. Real-World Tests

The team tested this on three tricky tasks:

• Pulling Tape: Using a hanger or umbrella to pull a roll of tape.
• Scooping Fish: Using different spoons/shovels to lift a wobbly, squishy fish.
• Hanging Scissors: Hooking a pair of scissors onto a wall hook.

The Results:

• The "Smart" Robot (This Paper): Consistently picked the deep shovel for the fish and the triple-hook for the scissors. When they pushed the robot or shook the table, the objects stayed put.
• The "Dumb" Robot (Old Methods): Often picked the wrong tool or held the object loosely. When disturbed, the fish fell, and the scissors slipped off.

Why This Matters

Think of this as teaching a robot common sense.
Before, robots were like a person trying to walk a tightrope while blindfolded—they could do it if nothing moved. This new method is like giving them a safety net and a steady hand. It allows robots to work in messy, real-world environments (like a kitchen or a factory floor) where things bump, shake, and go wrong, without dropping everything they are holding.

In short: It teaches robots to choose the tool that won't let them fail, even when the world gets a little chaotic.

Technical Summary

Here is a detailed technical summary of the paper "Robustness-Aware Tool Selection and Manipulation Planning with Learned Energy-Informed Guidance."

1. Problem Statement

Robotic tool use often prioritizes task completion over resilience to external disturbances. While humans subconsciously select tools and manipulation strategies that are robust to uncertainty (e.g., choosing a ladle over a spatula for meatballs), current robotic planning methods struggle with this. The core challenges are:

• Tool Variability: Tools have diverse geometries and physical properties, making manipulation outcomes highly sensitive to the combined configuration of the robot, tool, and object.
• Joint Optimization: Robustness requires not just planning a trajectory but also selecting the optimal tool from a set of candidates, accounting for complex contact dynamics in uncertain environments.
• Computational Cost: Evaluating robustness (e.g., via caging analysis) is computationally expensive, making real-time planning difficult.

The paper addresses the problem of jointly selecting the most robust tool and planning a contact-rich manipulation trajectory that maintains robustness against external force disturbances throughout execution.

2. Methodology

The authors propose a hierarchical optimization pipeline guided by a learned, energy-informed robustness metric. The approach consists of three main components:

A. Energy-Informed Robustness Metric

The core of the method is a metric derived from caging analysis, specifically the Minimum Escape Energy (MEE).

• Definition: MEE ($Q_{mee}$) quantifies the energy required for an object to "escape" from a tool's grasp or constraint under a specific force field (gravity, pushing, or elastic deformation).
• Energy Functions: The energy function $E(s_{obj})$ varies by task:
  • Gravitational: $m_{obj}gz_{obj}$ (vertical position).
  • Pushing: Adds a term for pushing force magnitude and direction.
  • Deformable: Includes an elastic term based on deformation angle.
• Baseline: The method compares against Partial Caging Clearance (PCC), which measures the geometric dilation needed to prevent escape, but MEE provides a more dynamic, energy-based assessment.

B. Learned Guidance (Offline Training)

To overcome the computational bottleneck of calculating MEE online (which requires sampling-based algorithms like BIT*), the authors train a Multi-Layer Perceptron (MLP) offline.

• Data Collection: A dataset is generated by randomizing tool-object configurations and computing MEE/PCC values using BIT* in a simulator.
• Network Architecture: A 3-layer MLP (128-128-64 neurons) predicts:
  1. Cage Status ($\hat{q}_c$): A binary classification (sigmoid) indicating if the object is fully caged.
  2. MEE Value ($\hat{Q}_{mee}$): A regression value for the escape energy.
• Unified Metric: During planning, a unified metric $\hat{Q}_{mee}$ is used, assigning a high robustness value if the cage status confidence exceeds a threshold, otherwise relying on the continuous energy estimate.

C. Hierarchical Optimization Pipeline

The planning process is split into two stages to handle the mixed-integer nature of the problem (discrete tool choice + continuous trajectory):

1. Keyframe Optimization (Tool Selection):

   • Uses CMA-ES (Covariance Matrix Adaptation Evolution Strategy) to jointly optimize the tool identity and the tool-object configuration at a specific keyframe timestep ($T_k$).
   • Objective: Maximize the robustness metric $Q$.
   • Output: The best tool ($o^*_{tool}$) and the optimal intermediate configuration ($s^*_{tool}, s^*_{obj}$).

2. Trajectory Planning:

   • Uses VPSTO (Via-Point-Based Stochastic Trajectory Optimization), a variant of CMA-ES.
   • Objective: Minimize the distance to the goal while enforcing the trajectory to pass through the optimal keyframe configuration found in step 1.
   • Constraint: The trajectory must maintain high robustness throughout execution, guided by the learned metric.

3. Key Contributions

1. Robustness-Aware Optimization: A unified framework that jointly selects tools and plans manipulation trajectories, explicitly optimizing for disturbance resilience rather than just task completion.
2. Learned Energy-Informed Metric: The introduction of a data-driven method to approximate Minimum Escape Energy (MEE) using neural networks, enabling efficient online robustness guidance without expensive real-time sampling.
3. Hierarchical Planning Architecture: A two-stage optimization process that decouples discrete tool selection from continuous trajectory generation, making the problem tractable.
4. Comprehensive Evaluation: Validation across three distinct tasks involving rigid, articulated, and deformable objects, with both simulation and real-world experiments.

4. Experimental Results

The method was evaluated on three tasks: Tape Pulling, Fish Scooping (deformable object), and Scissors Hanging (articulated object). It was compared against:

• PCC: A clearance-based baseline.
• NoRob: Planning without robustness guidance ($\beta=0$).
• VLM: A Vision-Language Model (GPT-5) used for tool selection.

Key Findings:

• Tool Selection: The MEE-based planner consistently selected the most robust tools (e.g., a shovel over a fish slice for scooping, a treble hook over a coat hook for hanging). VLMs showed some zero-shot capability but failed to identify the most robust geometric configurations for complex tasks.
• Robustness Performance:
  • In Tape Pulling, the MEE planner kept the tape in close contact with the tool handle, whereas PCC allowed the tape to slide out due to lack of robustness guidance.
  • In Fish Scooping, MEE successfully constrained the fish between the shovel and a table fixture, while PCC failed to find such stable configurations.
  • Quantitative Metrics: MEE resulted in significantly lower positional deviations under random force disturbances compared to PCC and NoRob (e.g., 0.054 vs. 0.237 deviation in tape pulling).
• Real-World Transfer: In real-world Scissors Hanging experiments, the MEE method achieved a 83% success rate, compared to 50% for the PCC baseline. Failures were primarily attributed to pose estimation errors rather than planning flaws.
• Efficiency: The learned model reduced inference time from 3 seconds per configuration (using BIT*) to **0.057 ms**, enabling real-time planning (total planning time ~29 seconds vs. ~17 CPU-days without the network).

5. Significance

This work bridges the gap between theoretical robustness (caging/energy analysis) and practical robotic manipulation. By integrating a learned energy metric into the planning loop, the system achieves human-like robustness, selecting tools and strategies that naturally resist external disturbances. The method demonstrates that explicit optimization for robustness, rather than relying solely on reactive control or semantic reasoning, significantly improves reliability in uncertain, contact-rich environments. The approach is particularly significant for tasks involving deformable or articulated objects where traditional grasp metrics fail. Future work aims to extend the dataset for zero-shot generalization to novel tools and objects.