Lang2Lift: A Language-Guided Autonomous Forklift System for Outdoor Industrial Pallet Handling

This paper presents Lang2Lift, an end-to-end autonomous forklift system that leverages natural language instructions to guide perception, pose estimation, and motion planning for successful pallet pick-up operations in diverse, unstructured outdoor industrial environments.

The Gist

Imagine a busy construction site or a lumber yard. It's messy, the lighting changes from bright sun to gloomy shadows, and pallets (those wooden platforms used to stack goods) are scattered everywhere, some holding bricks, others holding steel beams, and some covered in snow.

In the past, if you wanted a robot forklift to pick up a specific stack, you had to be a programmer. You'd have to tell the robot exactly where to go using coordinates, like "Move 5 meters forward, turn 30 degrees left." If the pallet moved or the robot got confused by a new object, the whole system would freeze, and a human would have to take over.

Enter "Lang2Lift."

Think of Lang2Lift as giving the robot forklift a superpower: the ability to understand human conversation. Instead of speaking in math coordinates, a human worker can just point and say, "Hey robot, pick up that pallet with the concrete blocks on top, the one near the crane."

Here is how the system works, broken down into simple steps using everyday analogies:

1. The "Smart Eyes" (Vision & Language)

Imagine the robot has a pair of glasses that can read your mind.

• The Translator: When you speak, a small computer chip translates your words ("pallet with concrete blocks") into a visual search query.
• The Detective: The robot uses a massive, pre-trained "brain" (called a Foundation Model) that has seen millions of images. It doesn't need to be taught what a "concrete block" is; it already knows. It scans the scene and highlights the specific pallet you are talking about, ignoring the steel beams or empty pallets nearby.
• The Refiner: Once it finds the right pallet, it uses another tool (like a digital scalpel) to trace the exact outline of the pallet, pixel by pixel, so it knows exactly where the edges are, even if the pallet is half-hidden behind a truck.

2. The "3D GPS" (Pose Estimation)

Knowing where the pallet is isn't enough; the robot needs to know how to grab it.

• The Puzzle Solver: The robot calculates the pallet's 6D pose. Think of this as solving a 3D puzzle to figure out exactly how the pallet is tilted, rotated, and positioned in space.
• The Fork Alignment: Because pallets are symmetrical (they look the same from the front and back), the robot has to guess which way the "fork holes" are facing. It uses a clever trick to figure out the correct angle so the forks slide in smoothly, not hitting the wood.

3. The "Driver" (Planning & Control)

Now that the robot knows what to grab and where it is, it has to drive there.

• The Navigator: The robot plans a path that avoids obstacles, just like a human driver looking for a clear lane.
• The Steady Hand: As it approaches, it drives slowly and carefully. It uses sensors to make tiny adjustments, ensuring the forks line up perfectly with the pallet's holes. It's like threading a needle while driving a car.

Why is this a big deal?

• Flexibility: Old robots were like rigid robots in a factory; they could only do one thing. Lang2Lift is like a human worker who can adapt. If you say, "Grab the red one," it grabs the red one. If you say, "Grab the one behind the truck," it figures that out too.
• Real-World Toughness: The researchers tested this in the real world, not just in a clean lab. They tried it in the snow, in low light, and in messy, cluttered yards. It worked surprisingly well, successfully picking up pallets about 60% of the time in the best setups, which is a huge leap for outdoor automation.
• No Coding Required: The biggest win is that construction workers don't need to learn to code. They can just talk to the machine naturally.

The Catch (Limitations)

It's not magic yet. The paper admits there are still some hiccups:

• Confusing Instructions: If you say something vague like "Pick up the thing," the robot gets confused. You have to be specific.
• Total Hiding: If a pallet is completely buried under a pile of debris, the robot's "eyes" can't see it, and it can't grab it.
• Speed: It's not lightning fast. The thinking process takes about a second or two per cycle. This is fine for a slow-moving forklift, but it's not for high-speed racing.

The Bottom Line

Lang2Lift is a bridge between human intuition and robotic precision. It takes the "brain" of modern AI (which understands language and images) and puts it into the "body" of a heavy-duty forklift. It turns a rigid, pre-programmed machine into a helpful assistant that can listen to a foreman and get the job done, even in the messy, unpredictable world of outdoor construction.

Technical Summary

1. Problem Statement

Autonomous pallet handling in outdoor industrial and construction environments faces significant challenges compared to controlled warehouse settings. Key issues include:

• Unstructured Scenes: Cluttered environments with variable lighting, weather conditions (snow, low-light), and occlusions.
• Dynamic Cargo: Pallets carry diverse loads (concrete blocks, steel beams, lumber) with different orientations and configurations.
• Rigidity of Current Systems: Existing automated forklifts rely on pre-programmed paths or rigid geometric detection. They lack the flexibility to interpret natural language commands (e.g., "pick up the pallet with the concrete block on top") to distinguish between multiple similar-looking pallets.
• Generalization Gap: Traditional CNN-based detection methods often fail to generalize across diverse outdoor conditions and cannot classify specific load types without extensive retraining.

The core problem is enabling an autonomous forklift to dynamically select and manipulate specific pallets based on natural language instructions in unstructured outdoor settings without prior knowledge of every specific scene configuration.

2. Methodology: The Lang2Lift Framework

Lang2Lift is an end-to-end system integrating Foundation Models (FMs) for perception with traditional motion planning and control. It operates on the ADAPT autonomous outdoor forklift platform.

A. Perception Pipeline (Three Stages)

1. Language-Driven Object Segmentation:
   • Input: Natural language commands (e.g., "Pick up the lumber pallet near the crane").
   • Processing: A semantic parser extracts object types, visual descriptors, and spatial relationships to create a structured "referring-expression prompt."
   • Detection: The prompt is fed into Florence-2 (a unified Vision-Language Foundation Model) to generate bounding boxes for target objects.
   • Refinement: SAM-2 (Segment Anything Model 2) is applied to the bounding boxes to generate pixel-level segmentation masks, ensuring precise boundaries for pose estimation.
2. Pose Processing & Geometric Refinement:
   • 6D Pose Estimation: The system uses FoundationPose (a model-free, training-free foundation model) combined with RGB-D data and a CAD model of the pallet to estimate the initial 6D pose ($R, t$).
   • Symmetry Handling: Pallets often have symmetric structures, leading to ambiguous insertion orientations. The system implements a geometric refinement step:
     • It evaluates the local x-axis direction relative to the camera.
     • It applies specific homogeneous transformations ($T_1$ or $T_2$) to resolve symmetry and align the reference frame with the optimal fork insertion point.
3. Temporal Pose Tracking:
   • To maintain robustness during movement, a factor-graph-based tracker (using iSAM2) fuses vehicle odometry, GNSS, and perception updates at 25 Hz. This smooths pose estimates between slower 6D updates.

B. Planning and Control Pipeline

• Motion Planning: Uses a hierarchical Hybrid A* algorithm adapted for articulated forklift kinematics (Reeds-Shepp curves) to navigate unstructured terrain.
• Control: A Lyapunov-based controller ensures stable path following, while a dedicated high-precision loop manages hydraulic fork positioning to achieve centimeter-level accuracy.
• Integration: The entire system runs on a ROS2 architecture, linking perception outputs directly to the control stack.

3. Key Contributions

1. End-to-End Language-Guided System: The first demonstration of a full-scale autonomous forklift that accepts natural language commands to select specific pallets in outdoor environments, bridging the gap between human intent and robotic action.
2. Training-Free Foundation Model Integration: Successfully integrates large-scale, pre-trained models (Florence-2, SAM-2, FoundationPose) into an industrial automation pipeline without task-specific fine-tuning, relying instead on prompt engineering.
3. Tolerance-Driven Evaluation: Introduces a quantitative evaluation framework that links perception accuracy directly to manipulation feasibility (fork insertion tolerances) rather than just abstract detection metrics.
4. Real-World Deployment Analysis: Provides a comprehensive analysis of system timing, failure modes, and engineering trade-offs (latency vs. accuracy) in real-world outdoor logistics scenarios.

4. Experimental Results

The system was evaluated on a dataset of 129 images with 387 prompt-image pairs across four conditions: Sunny, Snowy, Low-light, and Occlusion.

• Segmentation Performance:
  • The best configuration (Florence-2 open-vocab + SAM-2) achieved a mean IoU of 0.587 and a Success Rate (SR) of 60.47% at IoU $\ge$ 0.5.
  • Low-light performance was notably strong (mIoU 0.805), suggesting descriptive prompts help disambiguate targets when visual cues are weak.
  • Ablation Study: Removing SAM-2 caused a drastic drop in success rate (SR@0.75 dropped from 52.71% to 8.53%), proving that pixel-level mask refinement is critical for accurate pose estimation.
• Pose Estimation Accuracy:
  • The system met operational tolerances for fork insertion: $\pm$0.05m lateral and $\pm$0.04m vertical accuracy.
  • Accuracy degraded with distance and extreme angles, but dynamic approach maneuvers allowed the system to recover and meet tolerances.
• Timing Analysis:
  • Total perception-to-pose cycle: ~1.05 seconds.
  • Total planning-to-execution cycle: ~1.45 seconds.
  • While the 6D pose estimation is the bottleneck (0.83s), high-rate tracking (25 Hz) maintains stability during the approach.

5. Significance and Limitations

Significance:

• Operational Flexibility: Lang2Lift eliminates the need for reprogramming when new pallet types or layouts appear, allowing operators to use intuitive voice commands.
• Scalability: By using foundation models, the system avoids the data collection and labeling costs associated with training custom CNNs for every new environment.
• Engineering Roadmap: The paper shifts focus from algorithmic novelty to system integration, providing a practical blueprint for deploying AI-driven perception in safety-critical industrial robotics.

Limitations & Failure Modes:

• Linguistic Ambiguity: The system struggles with complex grammatical variations (singular vs. plural) or underspecified spatial relationships.
• Occlusion: Complete occlusion of the pallet or heavy clutter can lead to segmentation boundary errors.
• Latency: The current prototype relies on high-end workstations (RTX 4090); deployment on embedded edge hardware requires model compression and pipeline optimization.
• Environmental Constraints: Performance drops in severe occlusion or extremely low-light conditions where visual features are indistinguishable.

In conclusion, Lang2Lift demonstrates that language-guided perception is a viable and powerful approach for next-generation autonomous logistics, provided that robust geometric refinement and tolerance-aware control strategies are employed to handle the uncertainties of the real world.