💻 computer science

Factorized Spatio-Temporal Convolutions for Human Pose Estimation from Planar Lidar

This paper presents a lightweight, factorized spatio-temporal convolution network that enables real-time human detection and 2D pose estimation on computationally constrained service robots using only omnidirectional planar LiDAR, achieving superior accuracy through cross-modal self-supervised training without manual LiDAR labels.

Original authors: Simone Arreghini, Mirko Nava, Nicholas Carlotti, Antonio Paolillo, Alessandro Giusti

Published 2026-07-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Simone Arreghini, Mirko Nava, Nicholas Carlotti, Antonio Paolillo, Alessandro Giusti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to see the world. Most robots today are like people with glasses: they use cameras to take pictures, looking for shapes and colors to figure out where a person is standing and which way they are facing. This works great in a sunny room, but if the lights go out, or if the robot is in a crowded hallway where the camera gets blocked, the robot goes blind. To solve this, many robots carry a "laser scanner" (called a LiDAR) that shoots out invisible beams of light in a full circle, like a lighthouse. It doesn't see colors or faces; it just sees how far away things are. The problem is, a single snapshot from this laser scanner is like looking at a person through a keyhole: you might see a leg, but you can't tell if they are walking toward you or away from you, or if they are even a person at all.

To make sense of this blurry, one-dimensional view, scientists need to use a trick called "spatio-temporal" processing. Think of "spatial" as looking at the shape of the object right now, and "temporal" as watching how that shape changes over time. If you watch a person walk past a keyhole, you don't just see a leg; you see a leg appear, move, and disappear. By stitching these moments together, the robot can guess the person's full pose. This paper tackles the challenge of teaching a robot to do exactly that: to take a stream of these laser scans and figure out not just where a human is, but which way they are facing, all while running on a robot's small, weak computer brain without needing a supercomputer.

The Laser Lighthouse and the Time-Traveling Detective

Meet the robot. It's a service bot, the kind you might see in a hotel or a hospital, rolling around on wheels. It has a 360-degree laser scanner spinning around its waist, shooting out 360 invisible beams every second. But there's a catch: a single beam is just a dot of distance. If a person stands in front of the robot, the laser sees a "blob" of points. Is it a person? A chair? A trash can? And if it is a person, are they looking at the robot to say hello, or looking away to ignore it?

The researchers realized that looking at just one snapshot is like trying to guess a movie plot from a single frozen frame. You need the whole scene to play out. So, they built a special "Time-Traveling Detective" network. Instead of just looking at the current laser scan, this network looks at a short movie of the last few seconds of scans. It watches how the "blobs" move and change shape over time.

The Magic of "Factorized" Thinking

Here is the clever part. Most computer brains try to do everything at once: they look at the shape and the movement in one big, messy brain cell. The authors of this paper said, "Let's split the job." They created a new type of brain cell called a Space-Time Block.

Imagine you are trying to describe a dance.

The Spatial Step: First, you look at the dancer's pose right now. Are their arms up? Are their legs wide? This is the "spatial" part. The robot's network does this by looking at the laser beams in a circle, respecting the fact that the first beam and the last beam are actually next to each other (like a ring).
The Temporal Step: Next, you watch how the dancer moves from one second to the next. Did they spin? Did they step forward? This is the "temporal" part.

The paper's big discovery is that if you separate these two steps—first analyzing the shape, then analyzing the movement—the robot gets much better at guessing. It's like having a specialist for "shapes" and a specialist for "movies" working together, rather than one generalist trying to do both at once. This method, called factorized spatio-temporal convolution, allows the robot to spot a person and guess their facing direction much more accurately than previous methods that mashed everything together.

The "Shadow Teacher" Trick

Now, here is the biggest hurdle: How do you teach a robot to recognize humans from laser beams if you don't have a million hours of humans labeled in laser scans? Usually, you'd need a human to sit there and draw boxes around every person in every laser scan, which is boring and expensive.

The authors came up with a brilliant workaround: Self-Supervision. They used a robot that had both a laser scanner and a regular camera (like a webcam with depth sensing). The camera is good at seeing people and knows exactly where they are and which way they are facing. The laser scanner is bad at this.

So, they set up a "Shadow Teacher" system. The camera looks at the person and says, "Hey, there's a person at 2 meters, facing North!" The robot then checks the laser scanner. If the laser beam hits that same spot, the robot says, "Okay, I'll learn that this specific pattern of laser dots means 'person facing North'." But here's the kicker: the robot only learns this lesson where the camera can see. For the rest of the 360-degree circle (behind the robot, where the camera can't see), the robot has to use what it learned in the front to guess what's happening in the back. It's like learning to recognize a friend's voice in a quiet room, and then using that skill to recognize them in a noisy crowd where you can't see their face.

The Results: Faster, Smarter, and Real-World Ready

The team tested their "Space-Time Block" robot brain against older, simpler brains. The results were impressive.

Distance: The new robot guessed how far away a person was with 38% less error than the old way.
Position: It knew where the person was standing with 28% less error.
Facing Direction: It guessed which way the person was looking with 15% less error.

Even more exciting, this smart brain didn't need a supercomputer to run. The authors tested it on a standard service robot with a regular laptop processor (a CPU), and it worked in real-time. It could see people, guess where they were, and even guess if they were looking at the robot, all while the robot was moving around a busy office or a hallway.

They even tested it on a public dataset called FROG (which is like a standardized test for robot vision). Their model performed just as well as the heavy-duty models that usually need powerful graphics cards (GPUs) to run, but theirs ran smoothly on the robot's own small computer.

The Real-World Test: The Waiter Robot

To prove it wasn't just a lab trick, they put the robot in a real-world scenario. They programmed it to act like a waiter or a receptionist. The robot would scan the room, find a person, and if that person was facing the robot and close enough, the robot would roll over, turn to face them, and extend its arm in a "offering" gesture.

In one test, the robot successfully spotted a person standing in front of it, even though the person was partially hidden behind another object. The robot correctly guessed the person was there and facing it. However, the paper also notes the limits: if two people are standing exactly on top of each other from the robot's point of view (on the same laser beam), the robot gets confused and can only see one. Also, if the room is extremely cluttered, it's harder to guess. But overall, the experiment showed that this lightweight, time-aware approach is a solid step toward robots that can safely and socially navigate our world without needing expensive, heavy equipment.

In short, by teaching robots to watch the "movie" of laser scans instead of just the "frozen frame," and by letting a camera teach the laser how to see, the authors created a robot vision system that is smarter, faster, and ready for the real world.

Technical Summary: Factorized Spatio-Temporal Convolutions for Human Pose Estimation from Planar LiDAR

Problem Statement
Service robots operating in shared human spaces require reliable perception of nearby people to ensure safe navigation and facilitate socially aware Human-Robot Interaction (HRI). While vision-based pipelines and 3D LiDAR approaches are well-established, many mobile service robots are constrained to omnidirectional planar (2D) LiDARs and modest onboard processors. Planar LiDAR scans are inherently sparse and ambiguous compared to 3D point clouds; individual scans often suffer from clutter and the front-back symmetry of the human body, making it difficult to infer facing direction from a single snapshot. Existing solutions for 2D LiDAR often rely on hand-crafted features or fail to address the specific challenge of estimating relative human orientation (facing direction) without manual annotation or heavy computational resources.

Methodology
The authors propose a lightweight, fully convolutional network based on Space-Time Blocks (ST-Block) to perform omnidirectional human detection and relative 2D pose estimation (presence, distance, and orientation) from sequences of planar LiDAR scans.

Architecture (ST-Block): The core innovation is the explicit factorization of spatial and temporal processing. Unlike monolithic 3D convolutions that entangle space and time, the ST-Block separates these dimensions:
- Spatial Processing: Circular 1D convolutions are applied along the ray dimension of each scan independently. This respects the 360° angular topology of the LiDAR (where the first and last rays are adjacent) and allows the network to learn patterns invariant to the ray's position.
- Temporal Processing: A shallow Temporal Convolutional Network (TCN) aggregates features across a fixed history of scans (temporal window). This allows the model to capture motion cues and resolve ambiguities inherent in single static scans.
- Design: The network stacks multiple ST-Blocks with residual connections and skip connections to expand the receptive field in both space and time, outputting per-ray predictions for human presence ( $\hat{p}$ ), distance ( $\hat{d}$ ), and relative orientation encoded as sine and cosine components ( $\sin \hat{o}, \cos \hat{o}$ ).
Training Strategy (Masked Cross-Modal Self-Supervision): To eliminate the need for manual LiDAR labeling, the system employs a self-supervised approach. A narrow-Field of View (FOV) RGB-D camera (Azure Kinect) mounted on the robot provides a reliable 3D body tracker.
- The tracker generates pseudo-labels (skeleton pelvis position and facing direction) only within the camera's FOV.
- These labels are projected onto the 2D ground plane to create ray-wise supervision for the LiDAR model.
- A masked loss function is applied, enforcing regression errors only in the region where the camera and LiDAR FOVs overlap. The network is then tasked with generalizing these learned patterns to the entire 360° field of view, including areas behind the robot where no direct supervision exists.

Key Contributions

ST-Block Architecture: The adaptation of space-time factorization to circular 2D LiDAR sequences, separating circular spatial convolutions from lightweight temporal convolutions to efficiently aggregate motion cues.
Masked Cross-Modal Self-Supervision: A training pipeline that leverages a narrow-FOV RGB-D tracker to generate labels, enabling on-site adaptation to new environments without manual annotation.
Onboard-Effective Pose Perception: A system that predicts per-ray human presence, distance, and facing direction from planar LiDAR, validated on a service robot with real-time CPU inference.

Experimental Results
The approach was evaluated on a custom dataset collected across four environments (Corridor, Break Area, Office, Lab) and a public benchmark (FROG).

Performance vs. Baseline: Compared to a parameter-matched 1D fully convolutional baseline that implicitly mixes space and time, the ST-Block approach consistently outperformed the baseline. At an operating point targeting 80% recall, the best configuration achieved:
- Distance Error: Reduced by 38% (from 10.7 cm to 6.6 cm in specific configurations).
- Position Error: Reduced by 28% (from 13.2 cm to 9.5 cm).
- Orientation Error: Reduced by 15% (from 37.5° to 32.0°).
- Precision: Improved detection precision ( $P_{80}$ ) from 73.8% to 79.4%.
Ablation Study: The study confirmed that while skip connections improved position accuracy, residual connections were crucial for refining orientation estimation.
FROG Benchmark: On the public FROG dataset, the model demonstrated competitive detection accuracy (AP ~66%) while running in real-time on a standard service robot CPU (Intel Core i7), whereas competing high-performance methods required powerful GPU systems.
Self-Supervised Adaptation: Experiments showed that a model pre-trained on FROG could rapidly adapt to a new environment and robot platform via self-supervised learning, significantly improving metrics (AP, position, and orientation errors) over just three days of operation.
In-Field Demonstration: A closed-loop experiment on a TIAGo robot demonstrated the system's ability to detect, track, and approach a human target to perform an "offering gesture," validating the utility of the perception module for HRI behaviors.

Significance and Claims
The paper claims that the proposed approach offers a practical solution for spatial perception on computationally constrained service robots. By explicitly factorizing spatio-temporal processing and utilizing cross-modal self-supervision, the system overcomes the limitations of sparse 2D LiDAR data without requiring manual labeling or heavy compute resources. The authors emphasize that the method enables continuous, omnidirectional human awareness and relative pose estimation, which are critical for safe navigation and social interaction in dynamic environments. The work highlights that while facing direction estimation from 2D geometry remains inherently ambiguous, the proposed factorized architecture significantly reduces this error compared to implicit space-time models, making it suitable for deployment on resource-limited platforms.