Technical Summary: AeroGrab – A Unified Framework for Aerial Grasping in Cluttered Environments

Problem Statement

Reliable aerial grasping in cluttered environments remains a significant challenge due to occlusions, collision risks, and the dynamic constraints of aerial platforms. Existing aerial manipulation pipelines often suffer from three primary limitations:

1. Centroid-Based Simplification: Many systems target the geometric center of objects, which fails for everyday items where stable grasps lie on functional components (e.g., handles, rims) rather than the centroid.
2. Lack of Active Perception: Most systems assume the target is clearly visible. In cluttered scenes, objects are frequently occluded, making grasp feasibility highly viewpoint-dependent. Current pipelines rarely integrate active repositioning with grasp generation.
3. Disconnected Planning: There is a lack of tight integration between language-level task specification, grasp pose generation, and collision-aware feasibility analysis, leading to gaps in end-to-end operation reliability.

Methodology

The proposed AeroGrab framework presents a unified pipeline that translates natural language instructions into collision-safe aerial grasps. The system operates through four integrated stages:

1. Language-Guided Target Localization

The system converts natural language commands (e.g., "Grasp the green bottle from the steel table") into structured visual targets using a Large Language Model (LLM).

• Scene Grounding: The platform first identifies the scene context (e.g., "steel table") using SAM-3 for real-time segmentation.
• Active Orbital Search: Once the scene is localized, the drone performs a stationary yaw scan followed by an active orbital exploration around the scene centroid. This motion is regulated to maintain the scene within the camera's Field of View (FoV) while seeking diverse viewpoints to reveal occluded targets.
• Target Identification: As the drone orbits, the system detects the specific target object ($T_{obj}$) once it enters the visible region, transitioning from scene exploration to object-specific grasp planning.

2. Target Synthesis and Grasp Generation

Upon isolating the target, the system extracts a Region of Interest (RoI) from the RGB-D stream.

• Hybrid Architecture: A Hybrid CNN-ViT network, trained on the GraspNet-1Billion dataset, predicts a set of 6-DoF grasp candidates (translation, orientation, gripper width, and confidence).
• Aerial Feasibility Filtering: Since standard grasp datasets are designed for fixed-base manipulators, the system applies specific constraints:
  • Gravity-Aligned Approach: Discards grasps requiring access from below the support surface.
  • Stability-Aware Scoring: Penalizes grasps far from the object centroid to reduce destabilizing moment arms during lift-off, while retaining them if necessary for cluttered scenarios.

3. Collision-Aware Feasibility Evaluation

To ensure safety, candidate grasps undergo a rigorous filtering process before execution:

• Kinematic Reachability: Inverse kinematics are solved to verify the manipulator can reach the pose without violating joint limits.
• Batched Trajectory Collision Evaluation: Instead of sequential checking, the system uses a batched GPU-accelerated process to evaluate approach paths for all candidates simultaneously.
  • Convex Hull Approximation: The full aerial manipulator body is approximated as a set of convex hulls.
  • Two-Stage Check: A coarse KD-tree spatial query filters safe path segments, followed by an exact GPU intersection test against the robot's convex hulls and the environment's point cloud.
  • Scoring: A collision penalty is applied to the visual score, heavily penalizing paths that penetrate obstacles while allowing for minor grazing contacts.

4. Decision Logic and Execution

• Active Re-Positioning: If no safe grasp is found (score $\le$ threshold $\delta$), the system autonomously continues orbiting around the target centroid to explore new viewpoints until a collision-free corridor is revealed.
• Execution: Once a valid grasp is selected, the system generates a minimum-snap trajectory and executes it using an adaptive controller that compensates for modeling uncertainties and aerodynamic disturbances.

Key Contributions

1. Language-Guided Aerial Grasping in Clutter: A unified pipeline that translates natural language into executable actions by combining language grounding, active perception, and aerial manipulation.
2. Active Perception for Viewpoint-Dependent Grasping: An autonomous repositioning strategy that allows the aerial manipulator to explore cluttered scenes to improve target observability and expose feasible grasp approaches.
3. Collision-Aware 6-DoF Grasp Feasibility: A feasibility ranking module that evaluates candidates by explicitly considering full-body approach clearance, platform reachability, and surrounding clutter, enabling safe selection via lightweight onboard computation.

Experimental Results

The framework was validated through extensive simulation and hardware experiments (N=30 hardware runs) across three scenarios: tabletop clutter, window-constrained access, and shelf reachability.

• Performance Metrics:

  • Collision-Free Grasp Rate (CFGR): Achieved 80% in dense tabletop clutter (hardware), significantly outperforming baselines (Bauer et al.: 10%, Ramon-Soria et al.: 23.33%).
  • Grasp Accuracy (GA): Achieved an average error of 3.1 cm in dense clutter.
  • Latency: The perception-to-plan compute time (including scoring and feasibility evaluation) was 41 ms, enabling real-time operation.
  • Failure Rates: Object Search Failures (OSF) and Collision-Induced Failures (CIF) were reduced to 10% each in dense clutter scenarios.

• Ablation Study:

  • Removing obstacle awareness increased collision-induced failures to 53.34%.
  • Disabling real-time perception updates (open-loop) increased grasp error to 4.5 cm and reduced CFGR to 46.66%.
  • Replacing GPU-accelerated collision checking with CPU-only methods increased latency to 145 ms and reduced CFGR to 53.34%.

Significance and Claims

The paper claims that AeroGrab addresses the critical gap in existing aerial manipulation systems by tightly coupling language-guided semantic perception with platform-aware kinematic and collision constraints. The authors assert that the system enables reliable, end-to-end operation in unstructured, cluttered environments where traditional centroid-based or non-active approaches fail.

The significance lies in the demonstration that active perception combined with real-time, full-body collision checking is essential for reliable aerial grasping. The framework successfully bridges the gap between high-level task specification and low-level safe execution, achieving robust performance on lightweight onboard hardware (NVIDIA Orin Nano) without relying on external motion capture for control (though used for evaluation). The authors conclude that future work will focus on fully onboard deployment (including segmentation) and operation in GPS-denied environments without external infrastructure.