🔢 mathematics

HOT-POT: Optimal Transport for Sparse Stereo Matching

This paper proposes HOT-POT, an unsupervised sparse stereo matching framework that leverages optimal transport with epipolar and 3D ray distances to overcome geometric challenges and enable efficient feature and object matching, particularly for aligning distinct facial landmark conventions.

Original authors: Antonin Clerc, Michael Quellmalz, Moritz Piening, Philipp Flotho, Gregor Kornhardt, Gabriele Steidl

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Antonin Clerc, Michael Quellmalz, Moritz Piening, Philipp Flotho, Gregor Kornhardt, Gabriele Steidl

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer vision, giving machines the ability to see depth is a fundamental challenge. Just as human eyes work together to judge distance by comparing two slightly different views of the same scene, cameras can be paired to reconstruct a three-dimensional world from flat images. This process, known as stereo matching, is the engine behind obstacle detection in self-driving cars and the 3D scanning of faces for security or medical analysis. However, the real world is messy. Light changes, objects block each other, and cameras often capture different types of information, such as visible light versus heat. When these systems rely on sparse data—meaning they only track a few key points like the corners of an eye or the tip of a nose rather than every single pixel—the task becomes incredibly difficult. A tiny error in locating a point can throw off the entire calculation, leading to a distorted or broken 3D model.

A team of researchers from institutions in Germany, France, Taiwan, and Japan has developed a new mathematical approach to solve this specific problem. They focused on a scenario where two cameras, one capturing standard visible light and the other capturing thermal heat, are looking at the same faces. The goal is to match the specific points on the face seen by the first camera with the corresponding points seen by the second, even when the two cameras detect different numbers of points or when some parts of the face are hidden. To do this, the researchers turned to a concept called optimal transport. Imagine trying to move a pile of sand from one location to another with the least amount of effort; optimal transport is the mathematical framework for finding that most efficient path. In this context, the "sand" is the collection of points on a face, and the "effort" is the distance the computer thinks it needs to move a point to find its match.

The researchers realized that the standard way of measuring distance between these points was often too loose. Traditional methods rely on a geometric rule called the epipolar constraint, which essentially says that a point seen by one camera must lie on a specific line when viewed from the other camera. While this rule is mathematically sound, the researchers found that in noisy, real-world conditions, many different points could sit on that same line, making it impossible to know which one was the true match. To fix this, they proposed a new way of measuring distance based on the actual 3D rays of light traveling from the camera lenses into the scene. Instead of just checking if a point lies on a line, their method calculates the shortest distance between the two invisible lines of sight extending from each camera to the point in space. If the lines cross or come very close in front of the cameras, it is a good match. If they miss each other or cross behind the cameras where the scene cannot be seen, the match is rejected.

To make this even more robust, the team added a layer of depth regulation. They knew that in a typical room, objects cannot be infinitely far away or impossibly close. By setting reasonable limits on how far a matched point could be, they prevented the computer from making wild guesses that would result in a face floating in deep space or collapsing into the camera lens. They tested this new "ray distance" against the traditional method using both computer-generated 3D faces and real footage of people moving in a room. In simulations where they added random noise to the data to mimic real-world imperfections, the new method consistently outperformed the old one. It reduced the number of incorrect matches significantly, keeping the reconstructed 3D shapes much closer to the truth.

The researchers also tackled the problem of matching entire objects, such as whole faces, rather than just individual points. Sometimes one camera sees a face clearly while the other sees it partially hidden, or the software detects a different number of landmarks on each side. To handle this, they built a two-step system. First, the computer matches the points within each pair of faces to see how similar they are. Then, it looks at the collection of all faces and decides which face in the first image corresponds to which face in the second image. This hierarchical approach proved to be remarkably stable. In tests with real video footage of three people moving around a room, the new system correctly matched every single face in every frame, whereas the traditional method failed to match the correct faces in nearly a quarter of the cases.

The study demonstrates that by combining the geometry of light rays with a smart way of organizing the matching process, computers can become much more reliable at seeing in 3D, even when the data is sparse or comes from different types of cameras. This has practical implications for fields like public health screening, where identifying individuals with elevated body temperatures requires accurately matching faces across different camera types to prevent the spread of infectious diseases. The work suggests that with the right mathematical tools, machines can learn to see the world with a clarity that rivals human perception, even when the view is imperfect.

=== SUMMARY ===

Technical Summary: HOT-POT: Optimal Transport for Sparse Stereo Matching

Problem Statement

Stereo matching between images faces significant challenges in real-world applications such as autonomous driving, robotics, and facial analysis. These challenges include occlusions, motion, camera distortions, and radiometric differences (e.g., between RGB and thermal modalities). While dense stereo matching methods exist, they often rely on pixel-wise comparisons or learned feature representations that may not be applicable or robust when only sparse features (such as facial landmarks) are available.

Matching sparse features is inherently ill-posed due to:

Detector Noise and Occlusions: Sparse points are highly sensitive to noise and partial occlusions.
Modality Differences: In cross-modal scenarios (e.g., RGB vs. Thermal), feature trackers may produce sets of points with different cardinalities and distinct landmarking conventions.
Instability: Traditional nearest-neighbor matching is unstable when the number and location of points differ significantly between views.

The paper addresses the need for a training-free, unsupervised method to perform robust sparse stereo matching and object matching (e.g., face-to-face) under these conditions, leveraging camera calibration and geometric constraints.

Methodology

The proposed approach, HOT-POT, formulates stereo matching as an Optimal Transport (OT) problem. The core innovation lies in defining cost functions derived from epipolar geometry that quantify the matching quality between projected points.

1. Geometric Cost Functions

The authors propose two distance metrics to serve as the cost function $c(x, y)$ in the OT formulation:

Epipolar Distance ( $d_{epi}$ ): Based on the classical epipolar constraint. It measures the distance of a point in one image to the epipolar line of the corresponding point in the other image. While standard, the authors note it can be ambiguous when points from different objects lie close to the same epipolar line.
3D Ray Distance ( $d_{ray}$ ): A novel cost function derived from the distance between 3D half-lines (rays) originating from the camera centers and passing through the projected 2D points.
- The distance is calculated as the minimal distance between the two rays in 3D space.
- Depth Regularization: To address instability when rays are nearly parallel (leading to infinite depth) or when points are behind the cameras, the authors introduce a depth-regularized ray distance ( $d_{reg}$ ). This incorporates soft thresholds ( $\gamma_1, \gamma_2$ ) on the expected depth and a regularization parameter ( $\beta$ ) to penalize unrealistic depth values, ensuring matches occur within a physically plausible range.

2. Optimal Transport Formulation

Partial Optimal Transport (POT): To handle cases where the number of points differs between views ( $N \neq M$ ) due to occlusions or modality-specific detection limits, the authors utilize Partial OT. This allows for a "partial" assignment where excess mass is discarded, rather than forcing a one-to-one mapping for all points.
Hierarchical Optimal Transport (HOT): To match entire objects (e.g., faces) represented as unordered sets of points, the authors propose a two-step hierarchical strategy:
1. Local Step: Compute the POT cost between every pair of source and target objects (e.g., face $i$ and face $j$ ) based on their constituent points.
2. Global Step: Solve a second POT problem at the object level to determine the optimal assignment of objects to objects.
- This framework is termed HOT-POT when using partial transport.

3. Algorithmic Implementation

The method utilizes efficient solvers (e.g., Sinkhorn algorithm or linear programming via the PythonOT library) to solve the convex optimization problems. For binary matching, soft transport plans are projected to binary matrices based on mass concentration.

Key Contributions

OT Formulation for Sparse Stereo: The paper formulates the stereo matching problem as an instance of an Optimal Transport problem with cost functions specifically tailored to the geometric setting of camera projections.
Novel 3D Ray Distance: The authors introduce the 3D ray distance as a cost function. Numerical experiments demonstrate its superiority over the classical epipolar distance, particularly in reducing ambiguity and handling occlusions.
Hierarchical OT for Object Matching: The paper extends the approach to object matching via a Hierarchical Optimal Transport (HOT) formulation. This allows for stable correspondences between entire objects (like faces) even when the underlying point sets are unordered and have varying cardinalities.
Depth-Regularized Cost: The introduction of depth regularization stabilizes the ray distance, preventing degenerate matches at infinite depth and improving robustness in noisy or cross-modal scenarios.

Experimental Results

The authors evaluated their method on both synthetic and real-world datasets:

Synthetic Faces Dataset:
- Point Matching: The ray distance ( $d_{ray}$ ) significantly outperformed the epipolar distance ( $d_{epi}$ ). For a full dataset of 1872 points, OT with $d_{ray}$ reduced errors from 809 (standard) to 264, whereas $d_{epi}$ only reduced them to 681.
- Object Matching: When using HOT, the object-wise mismatch rate dropped to 0% for both distances, demonstrating the efficacy of the hierarchical approach.
- 3D Reconstruction: The depth-regularized ray distance yielded the lowest Wasserstein-2 distance for reconstructed 3D point clouds, indicating superior geometric accuracy.
Synthetic Spheres Simulation:
- Under varying levels of Gaussian noise, HOT-POT with the ray distance maintained stable object-level matching (low mismatch rates) even when point-wise matching degraded.
- The depth-regularized variant ( $d_{reg}$ ) further improved reconstruction quality under noise.
Real-World RGB-Thermal Data:
- Point Matching: In matching RGB and thermal landmarks (where no ground truth correspondence exists), the unregularized ray distance failed to reconstruct the face shape correctly due to infinite depth matches. The depth-regularized ray distance ( $d_{reg}$ ) successfully reconstructed the facial geometry.
- Face Matching: In a video sequence with multiple moving faces, HOT-POT using $d_{reg}$ achieved a 0% mismatch rate (20/20 frames correctly matched). In contrast, the epipolar distance resulted in a 24% mismatch rate, and the unregularized ray distance had a 5% mismatch rate.

Significance and Claims

The paper claims that the proposed HOT-POT framework addresses classical challenges in stereo vision, specifically:

Robustness: The proposed cost functions (especially the depth-regularized ray distance) enhance robustness against noise, ambiguity, and occlusions by limiting the set of admissible matches.
Efficiency: The OT formulation provides an efficient, mathematically grounded approach to matching that does not require training data.
Scalability to Objects: The HOT framework successfully extends sparse point matching to object-level matching, which is crucial for applications like identifying specific individuals in multi-person scenes.

The authors position their method as particularly effective for cross-modal scenarios (e.g., RGB-thermal) and applications in facial analysis, where matching distinct landmarking conventions is required. They suggest potential utility in public health screening for identifying individuals with elevated temperatures, though they do not claim to have implemented a full screening system, only the matching algorithm.

The paper concludes that while point-wise matching is sensitive to noise, the hierarchical approach (HOT) provides stable object-level correspondences even with large deviations and real-world measurement uncertainties. Future work is suggested to include three-way matching via multimarginal OT and integration into dense stereo networks.