⚡ electrical engineering

Can we stabilize an inverted pendulum with feedback from a time-of-flight camera?

This paper demonstrates that an inexpensive, low-resolution time-of-flight camera provides sufficient feedback to reliably and precisely stabilize an inverted pendulum on a cart, challenging the belief that such sensors are too noisy for precise control of fast, unstable dynamics.

Original authors: Anthony Czubarow, Antonio Terpin, Raffaello D'Andrea

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anthony Czubarow, Antonio Terpin, Raffaello D'Andrea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to balance a broomstick on the palm of your hand. It's a classic trick: the stick wants to fall over, and you have to move your hand quickly and precisely to keep it upright. Now, imagine doing this while blindfolded, but instead of feeling the stick, you have to guess its angle by looking at a blurry, noisy photo of it taken from underneath your hand.

That is essentially what this paper does. The researchers asked a simple question: Can a cheap, low-quality camera be good enough to balance a broomstick on a moving cart?

Here is the breakdown of their experiment and findings, using everyday analogies:

The Challenge: The "Blurry" Eye

Usually, to balance a broomstick (or an "inverted pendulum") on a cart, engineers use high-end, expensive sensors like laser encoders or motion-capture cameras. These are like having a pair of perfect, high-definition eyes that never miss a detail.

The researchers wanted to see if they could get away with a "budget" sensor: a Time-of-Flight (ToF) camera.

What is it? Think of it as a camera that measures distance by bouncing light off objects, similar to how a bat uses sonar. They are cheap, small, and work in the dark.
The Problem: These cameras are known for being "noisy" and having low resolution. It's like trying to balance the broomstick while looking at it through a foggy, grainy window. Most people thought this "fuzziness" would make precise balancing impossible because the camera's errors would trick the computer into making the wrong moves.

The Setup: The "Upward" View

To make the test even harder, the researchers didn't just put the camera in front of the cart. They mounted it underneath the cart, looking straight up at the bottom of the broomstick.

Why? This is the "worst-case scenario." Looking up at a moving object creates weird distortions and shadows (like trying to see a car's license plate from directly underneath it).
The Goal: If they can balance the stick with the camera in this difficult, upside-down position, it proves the technology is robust enough to work even in perfect conditions.

The Brain: The "Smart Filter"

The camera sends a stream of messy, jumpy data to a computer (a Raspberry Pi). If the computer reacted to every single pixel of noise, the cart would shake violently and the stick would fall.

Instead, they used a mathematical "smart filter" (called a Kalman Filter).

The Analogy: Imagine you are trying to guess the speed of a car through a foggy window. You see the car jump around a bit because of the fog. Your brain (the filter) knows the car can't teleport or stop instantly. So, it ignores the tiny, impossible jumps and calculates where the car probably is.
The researchers combined the noisy camera data with data from the cart's motor (which knows how fast the cart is moving) to create a smooth, accurate picture of where the stick is.

The Result: Success with a "Budget" Sensor

The result was a resounding yes.

Using only the cheap camera and the smart filter, they successfully balanced the stick for 10 seconds straight, 30 times in a row.
The stick wobbled only a tiny amount (less than the width of a human hair in terms of angle).
They proved that you don't need a $10,000 motion-capture system to do this; a $50 camera is sufficient if the software is smart enough to clean up the noise.

The Takeaway

The paper concludes that low-cost sensors are more powerful than we thought.

They aren't saying you should replace all high-end sensors with cheap ones immediately. Instead, they are proving that with the right "brain" (software), even a "blurry" eye can perform high-precision tasks. They have also made their entire robot design and code open-source, inviting others to try it themselves and see how far this "budget" technology can go.

In short: You don't need perfect eyes to balance a broomstick; you just need a smart enough brain to interpret what the blurry eyes are seeing.

Technical Summary: Stabilizing an Inverted Pendulum with Time-of-Flight Camera Feedback

Problem Statement
Time-of-Flight (ToF) cameras are widely adopted in robotics for their compactness, low cost, and robustness to lighting conditions, yet they are generally considered unsuitable for precise, high-bandwidth feedback control due to low spatial resolution, depth noise, and systematic errors (e.g., multipath interference, motion blur). The central question addressed in this work is whether a low-cost, low-resolution ToF sensor can provide measurements of sufficient quality to stabilize a fast, unstable dynamic system. The authors select the inverted pendulum on a cart (cart-pole) as the testbed. This canonical benchmark is notoriously difficult to control due to its non-minimum phase nature, light damping, and sensitivity to millisecond-level latency and jitter. The challenge is exacerbated by the specific experimental geometry: the ToF camera is mounted beneath the pendulum, looking upward. This "adverse" viewpoint induces large incidence angles that worsen depth bias and multipath propagation, intentionally stress-testing the sensing pipeline.

Methodology
The authors constructed a compact, reproducible experimental platform consisting of a 3D-printed cart driven by a Maxon BLDC motor via a rack-and-pinion mechanism, and a pendulum (a gold pen) balanced on the cart. The control and sensing stack runs on a Raspberry Pi 5 using ROS2, introducing non-trivial processing jitter, which serves as a realistic stress test for the control loop.

Sensing Pipeline: The system relies solely on a low-end ArduCam ToF module (240 × 180 resolution, 30 fps) for angle feedback. The cart's linear position and velocity are measured via the motor's Hall-effect sensor.
Vision Processing: Raw depth images are preprocessed by filtering out-of-range pixels, smoothing, and binary thresholding to isolate the pendulum. The system fits two circles within the pendulum's contour (one for the base, one for the top) using Euclidean distance transforms and sub-pixel refinement. The angle is derived from the orientation of the segment connecting these circle centers.
State Estimation and Control: The authors employ a standard Linear Quadratic Gaussian (LQG) regulator. A Kalman filter fuses the noisy ToF-derived angle measurements with the motor encoder data to estimate the full state vector (cart position, cart velocity, pendulum angle, and pendulum angular velocity). The noise covariances for the filter are tuned via Bayesian optimization to minimize the Root-Mean-Squared Error (RMSE) against ground-truth encoder data.
System Identification: Key dynamic parameters (masses, moments of inertia, friction coefficients) were identified using a batch least-squares approach comparing collected trajectories to a nonlinear model integrated via Runge–Kutta 4.

Key Results
The system successfully achieved reliable and precise stabilization of the inverted pendulum using only the ToF camera for angular feedback, without high-end encoders or external motion-capture systems.

Stabilization Performance: Across 30 trials of 10 seconds each, the controller maintained the pendulum in an upright position. The time-averaged RMSE between the controlled pendulum angle and the upright equilibrium was $1.0 \times 10^{-3}$ rad·s $^{-1}$ .
Sensor Accuracy: The raw angle estimates derived from the ToF camera showed an RMSE of $3.0 \times 10^{-3}$ rad when compared to the ground-truth encoder during controlled trajectories. The stationary noise of the angle estimate was characterized as approximately zero-mean Gaussian with a standard deviation of $1.7 \times 10^{-3}$ rad.
Robustness: Despite the adverse camera placement (inducing multipath and motion blur) and the processing jitter inherent to the ROS2/Raspberry Pi stack, the LQG controller maintained stability, demonstrating that the ToF sensor provided sufficient signal fidelity for closed-loop regulation.

Significance and Claims
The paper presents a "proof-of-existence" study demonstrating that inexpensive, low-resolution ToF sensors can support precise feedback control for fast, unstable systems. The authors explicitly frame their results as a conservative lower bound on the potential of ToF-based feedback. By choosing a difficult geometry and a standard, non-optimized controller (LQG), they show that even under suboptimal conditions, ToF sensors are viable for high-performance tasks.

The significance lies in challenging the prevailing assumption that ToF cameras are limited to low-to-moderate control demands (e.g., obstacle avoidance or coarse servoing). The authors argue that with thoughtful system design, ToF sensors can replace expensive motion-capture infrastructure or high-end encoders in specific applications. They contribute an open-source platform and code to facilitate further benchmarking and research, suggesting that future work could explore advanced control strategies (such as Model Predictive Control or robust controllers) and alternative sensor configurations to further push the performance boundaries.