A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction

In the world of computer vision, machines are learning to see the world in three dimensions from flat photographs. They take a series of two-dimensional images and, in a single pass, reconstruct a 3D map of the scene, placing points in space to represent walls, floors, and objects. For these systems to be useful in real life, such as guiding a robot or helping a self-driving car, they need more than just a guess about where things are; they need to know how sure they are about that guess. This is where a "confidence signal" comes in. Every time the model predicts a point in 3D space, it also emits a number representing its confidence. Downstream systems, which are the programs that actually use this 3D data, read this number as a reliability signal. If the confidence is high, the system trusts the point and uses it; if it is low, the system ignores it or treats it with caution. For years, this confidence signal has been treated as a simple ranking tool: a way to sort pixels from most reliable to least reliable. But a critical question remained unanswered: does the number itself actually tell the truth about how large the error might be?

A team of researchers at the Technical University of Munich and other institutions decided to put this confidence signal under a microscope. They did not build a new model or try to teach an old one to be better. Instead, they performed a rigorous audit of seven existing, publicly released models that are already being used by the scientific community. They tested these models on thirteen different datasets, ranging from indoor rooms and city streets to synthetic video games, covering billions of individual pixels. Their goal was to see if the confidence numbers these models produced were accurate measurements of uncertainty, or if they were merely good at sorting errors without telling the truth about their size.

The researchers found that the models are excellent at ranking. If a model says one pixel is more confident than another, it is almost always correct that the first pixel is indeed more accurate. However, when they looked at the actual size of the error, a different story emerged. The models were consistently overconfident. They claimed to be much more certain than they actually were. On average, across all the models tested, the predicted uncertainty was too low by a factor of 2.4. This means that if a model predicted an error of one centimeter, the actual error was often closer to 2.4 centimeters. The problem got worse the more confident the model felt. When the model was very sure, it was actually the most wrong, underestimating its error by a factor of up to 15 in some cases.

This overconfidence was not a sign that the models were poorly trained or had stopped learning too early. The researchers proved that even if they took a released model and continued training it on its own data, it would quickly reach its mathematical limit and then stay overconfident on new, unseen images. The issue was not that the models hadn't learned enough; it was that the way they were taught to produce confidence numbers was fundamentally different from how those numbers are used in the real world. During training, the models learned to minimize a specific mathematical loss function, which optimized the ranking of errors but did not guarantee that the confidence numbers matched the actual physical size of the errors in a new environment.

To fix this, the researchers developed a simple post-hoc correction. They discovered that by applying a specific mathematical adjustment—a power law that scales the confidence numbers based on the model and the dataset—they could bring the predicted uncertainty much closer to the truth. This correction did not change the ranking; the models still sorted pixels correctly. But it fixed the scale. With this adjustment, the average error in the prediction dropped from being off by 2.4 times to being off by only 1.35 times. If the researchers had a small amount of labeled data from the specific dataset they were using, they could refine this correction even further, bringing the error down to just 1.12 times the prediction.

Despite this success, the researchers found a hard limit to what a simple number can fix. While the correction worked well on average, it could not perfectly predict the error for every single scene. A correction that works for a whole dataset often fails for a specific room or street because the error in a scene depends on the unique shape and structure of that scene, not just a global shift in numbers. The models lack a deep understanding of the specific geometry of the scene they are looking at, and no amount of rescaling can fully compensate for that missing knowledge.

The study concludes that while these feed-forward 3D reconstruction models are powerful tools that provide excellent rankings of reliability, their raw confidence numbers cannot be trusted as direct measurements of error. They are reliable guides for sorting, but not for measuring. The researchers have released their audit protocol and the specific correction constants for every model and dataset they tested, allowing other scientists and engineers to apply these fixes immediately. This work provides a clear path forward: users can now take these powerful, pre-trained models, apply a simple correction to their confidence signals, and use them with a much clearer understanding of their actual limitations.