Vision-Language Models for Ergonomic Assessment of Manual Lifting Tasks: Estimating Horizontal and Vertical Hand Distances from RGB Video

This study demonstrates the feasibility of using vision-language models to non-invasively estimate horizontal and vertical hand distances from RGB video for ergonomic risk assessment, showing that a multi-view pipeline incorporating pixel-level segmentation achieves mean absolute errors of approximately 5–8 cm.

The Gist

Imagine you are a safety inspector trying to figure out if a worker is lifting a heavy box in a way that might hurt their back. To do this, you need to measure two specific things:

1. How far the box is from their body (Horizontal distance).
2. How high off the ground the box is (Vertical distance).

In the old days, a human had to stand there with a tape measure, which is slow and annoying. If you tried to use a robot or a camera, the computer often got confused, thinking a hand was a shoe or getting lost in the background clutter.

This paper is about teaching a super-smart computer to do this measuring job automatically using just a regular video camera (like the one on your phone) and a new type of AI called a Vision-Language Model (VLM).

Here is the breakdown of how they did it, using some simple analogies:

1. The Problem: The "Blind" Computer

Traditional computer vision is like a robot that only sees stick figures. It knows where a "hand" is because it sees a joint, but it doesn't really understand what a hand is holding or how it relates to the box. If the worker bends over, the robot gets confused about where the hands actually are. It's like trying to navigate a maze while wearing a blindfold and only feeling the walls.

2. The Solution: The "Smart Detective" (Vision-Language Models)

The researchers used a new kind of AI that is like a detective who can read and see at the same time.

• The Text Prompt: Instead of just telling the computer "find the hand," they told it, "Find the person lifting, find the wooden box, find the shoes."
• The Magic: Because the AI understands language, it knows what a "box" looks like even if it's never seen that specific box before. It can point to the exact spot in the video.

3. The Two Methods: "The Box" vs. "The Laser Cutter"

The researchers tried two different ways to help the AI measure the distances:

• Method A: The "Cardboard Box" Approach (Detection Only)
  The AI draws a square box around the worker's hand or the load. It's like putting a cardboard frame around a painting. It's quick, but the frame includes some extra background (like the wall behind the hand), which makes the measurement a little fuzzy.

  • Result: It worked okay, but the measurements were a bit off (like guessing a distance and being off by 10 inches).

• Method B: The "Laser Cutter" Approach (Detection + Segmentation)
  After the AI finds the object, it uses a second tool (called SAM) to cut out the object pixel-by-pixel. It's like using a laser cutter to slice the hand or the box out of the background perfectly, leaving no extra wall or floor attached.

  • Result: This was much more accurate. The AI could see the exact edge of the hand, reducing the error significantly (like getting the distance right within 2–3 inches).

4. The Camera Angle: "One Eye" vs. "Three Eyes"

The researchers tested the AI with different camera setups:

• One Camera (One Eye): Like trying to judge how far away a car is with one eye closed. It's hard to tell depth. The AI struggled a lot here, especially with vertical height.
• Three Cameras (Three Eyes): They used three cameras at different angles (front, left, right) all watching the worker at the same time. This is like having three eyes working together. The AI could cross-reference the views to figure out exactly where the hand was in 3D space.
  • Result: The three-camera setup was the winner. It was the most accurate, especially for measuring how high the box was.

5. The Results: How Good Was It?

The goal was to measure distances in centimeters.

• The "Laser Cutter" + "Three Eyes" combo was the best.
  • For Horizontal distance (how far the box is from the body), it was off by only about 6 to 8 cm (roughly 2.5 to 3 inches).
  • For Vertical distance (how high the box is), it was off by about 5 to 8 cm (roughly 2 to 3 inches).
• This is accurate enough to be useful for safety assessments without needing the worker to wear sensors or a human to stand there with a tape measure.

Why Does This Matter?

Think of this as a safety net that watches itself.

• No Sensors Needed: Workers don't have to wear uncomfortable vests or straps.
• No Tape Measures: Safety managers don't have to stop work to measure things.
• Real-Time Safety: Imagine a camera in a warehouse that watches workers lift boxes. If the AI sees someone lifting a box too far away from their body (which is dangerous), it could instantly alert a supervisor to step in and fix the technique before an injury happens.

The Catch

The study was done in a clean, well-lit lab with young, healthy people. Real-world factories are messy, dark, and crowded. The AI might get confused if there are too many people blocking the view or if the lighting is bad. But this paper proves the idea works, and it's a huge step toward making workplaces safer using just a video camera and some smart software.

In short: They taught a computer to "see" and "understand" lifting tasks better than ever before, proving that a simple video camera can replace expensive sensors and manual measuring for checking worker safety.

Technical Summary

1. Problem Statement

Manual material handling (MMH) tasks, particularly lifting, are a leading cause of work-related musculoskeletal disorders (WMSDs). The Revised NIOSH Lifting Equation (RNLE) is the standard tool for assessing ergonomic risk, relying on six parameters, including Horizontal (H) and Vertical (V) hand distances.

• Current Limitations: Accurate measurement of H and V typically requires manual observation (prone to bias and time-consuming) or specialized sensing systems (wearable sensors, marker-based motion capture), which are intrusive, expensive, and difficult to deploy in real-world environments.
• Computer Vision Gaps: Existing markerless pose estimation methods (e.g., OpenPose, BlazePose) often struggle with occlusion, temporal instability, and a lack of explicit object-centric spatial reasoning (e.g., the relationship between the worker, the load, and the environment).
• Research Gap: The application of Vision-Language Models (VLMs) for the quantitative estimation of ergonomic exposure metrics like RNLE parameters has not been explored, nor has the impact of camera view conditions on such estimates.

2. Methodology

Dataset and Ground Truth

• Data Source: A subset of data from a prior study involving 32 healthy young adults performing 24 lifting trials each (varying lift origins, hand configurations, and box masses).
• Input: RGB video streams (1280 × 720) from three synchronized Azure Kinect cameras (V1, V2, V3) representing oblique and frontal views.
• Ground Truth: H and V distances were derived from a wearable IMU system (16 sensors) and temporally aligned with video frames.
  • H: Horizontal distance between the midpoint of the ankles and the midpoint of the hands (projected to the floor).
  • V: Vertical distance from the floor to the midpoint of the hands.

Proposed Pipelines

The authors developed and compared two multi-stage VLM-based pipelines:

1. GD–Dv2 (Detection-Only): Uses Grounding DINO for text-guided object detection (zero-shot) to locate the "lifter" and relevant ROIs (hands, shoes, box). It relies on bounding boxes for feature extraction.
2. GD–SAM–Dv2 (Detection + Segmentation): Uses Grounding DINO for initial detection, followed by the Segment Anything Model (SAM) to generate pixel-level segmentation masks for the detected ROIs. This refines the region of interest by excluding background pixels.

Feature Extraction & Regression

• Visual Features: Both pipelines extract features from the ROIs using DINOv2 (a self-supervised Vision Transformer) in a zero-shot configuration.
• Geometric Augmentation: DINOv2 features (768-dim) are augmented with geometric features derived from the detected box (normalized image dimensions and known physical dimensions) to provide scale, resulting in a 773-dimensional feature vector.
• Temporal Modeling: A Transformer-based regression model (bidirectional encoder with 6 layers and 8 attention heads) processes sequences of 100 frames to estimate H and V, leveraging temporal dependencies to smooth noise and handle occlusions.
• Validation Strategy: Leave-One-Subject-Out (LOSO) cross-validation was used across 32 folds to ensure the model generalizes to unseen individuals.

Experimental Conditions

The study evaluated performance across seven camera view conditions:

• Single-view: V1, V2, V3.
• Multi-view: V1+V2, V1+V3, V2+V3, V1+V2+V3.

3. Key Results

Overall Performance

• Segmentation Advantage: The GD–SAM–Dv2 pipeline (with segmentation) consistently outperformed the detection-only pipeline.
  • H Estimation: Reduced Mean Absolute Error (MAE) by 20–30%.
  • V Estimation: Reduced MAE by 35–40%.
• Multi-View Advantage: Multi-view conditions (especially V1+V2+V3) yielded significantly lower errors than single-view conditions.
  • Best Performance (Start of Lift):
    • H: MAE ≈ 6.2 cm (V1+V2+V3, GD–SAM–Dv2).
    • V: MAE ≈ 7.8 cm (V1+V2+V3, GD–SAM–Dv2).
  • Worst Performance: Single-view conditions (e.g., V1 or V2 alone) resulted in MAEs exceeding 10 cm for H and up to 27 cm for V.

Temporal Dynamics (Start vs. End of Lift)

• Vertical Distance (V): Errors were higher at the start of the lift (hands near floor, high occlusion by torso/load) and lower at the end (hands near hip, better visibility).
• Horizontal Distance (H): Errors were lower at the start and increased at the end. This is attributed to the difficulty in localizing ankle landmarks when the load is raised and potentially occludes the lower body, particularly in single-view setups.

Specific Findings on View Conditions

• Two-camera combinations including a frontal view (e.g., V1+V3 or V2+V3) often performed comparably to the three-camera setup, suggesting a favorable trade-off between system complexity and accuracy.
• The frontal view (V3) was critical for accurate V estimation, while oblique views helped with H estimation.

4. Key Contributions

1. Feasibility of VLMs for Ergonomics: Demonstrated that VLMs (Grounding DINO + SAM + DINOv2) can non-invasively estimate critical RNLE parameters (H and V) from RGB video with reasonable accuracy (MAE ~6–8 cm).
2. Segmentation Impact: Provided empirical evidence that pixel-level segmentation significantly improves distance estimation accuracy over bounding-box detection, particularly for vertical distances where perspective projection errors are magnified.
3. Camera View Analysis: Systematically quantified how camera viewpoint affects ergonomic estimation, showing that multi-view setups (especially those including a frontal view) drastically reduce error compared to single-view setups.
4. Zero-Shot Approach: Utilized pre-trained VLMs without task-specific fine-tuning, suggesting potential for generalizability to new environments without extensive retraining.

5. Significance and Implications

• Practical Application: This work supports the development of ambient, video-based ergonomic assessment systems that do not require workers to wear sensors or for safety officers to perform manual measurements.
• Risk Assessment Accuracy: By reducing errors in H and V estimation, these systems can improve the reliability of RNLE risk classifications (e.g., Recommended Weight Limits), leading to better-informed workplace interventions.
• Scalability: The use of standard RGB cameras and zero-shot models offers a scalable path for deploying ergonomic monitoring in diverse industrial settings (logistics, manufacturing, healthcare).

6. Limitations and Future Work

• Controlled Environment: The study was conducted in a lab with uniform lighting and backgrounds; performance in cluttered, real-world environments with variable lighting remains to be tested.
• Demographics: The dataset consisted of young, healthy adults; generalizability to older workers or those with different anthropometries is unknown.
• Scope: The study focused only on H and V. Future work needs to extend VLMs to estimate other RNLE parameters (asymmetry, coupling, frequency).
• Ground Truth: Reliance on IMU data for ground truth introduces its own measurement uncertainties compared to optical motion capture.