💻 computer science

Gravity-aware partially calibrated absolute pose estimation from affine- or rotation-covariant features

This paper introduces two efficient solvers, UP1PfAC and UP2PfORI, that leverage IMU-derived gravity vectors and local geometric information from affine or rotation-covariant features to achieve fast and accurate absolute pose and focal length estimation with fewer correspondences than traditional methods.

Original authors: Marcus Valtonen Örnhag, Alberto Jaenal, Stefan Adalbjörnsson

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcus Valtonen Örnhag, Alberto Jaenal, Stefan Adalbjörnsson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to find your way through a vast, unfamiliar city using only a photograph of a landmark and a map. For a computer, this task is known as absolute pose estimation: determining exactly where a camera is and which way it is pointing based on a single image. This capability is the backbone of modern augmented reality, allowing digital overlays to stick firmly to the real world, and it is essential for drones and robots to navigate without getting lost. For years, these systems have relied on matching specific points in an image to a pre-built 3D model of the world. However, this process often requires many matching points to work reliably, and it struggles when the camera's internal settings, such as its zoom level or focal length, are unknown. Furthermore, traditional methods often ignore the subtle geometric clues hidden within the way features are described in the image, treating every point as a simple dot rather than a shape with direction and scale.

A team of researchers from Ericsson Research and the University of Zaragoza has developed a new approach that solves this problem more efficiently by combining visual data with information from a standard motion sensor found in most smartphones. By fusing the camera's view with the sensor's knowledge of gravity, they created a system that can determine a camera's position and its unknown zoom level using far fewer visual clues than before. Their work introduces two new methods that can calculate this location using either a single detailed image feature or just two features that maintain their orientation. This breakthrough means that devices can localize themselves faster and with greater accuracy, even when the camera's internal settings have changed or are not known in advance.

The core of the problem lies in how computers understand the relationship between a 3D world and a 2D photograph. Traditionally, to figure out where a camera is, a system needs to match at least three or four points between the image and the 3D map. If the camera's focal length is also unknown, the number of required points increases, making the calculation slower and more prone to errors when the data is noisy. The researchers realized that they could bypass this heavy requirement by using two powerful sources of information that were previously underutilized. First, they used the gravity vector provided by the device's inertial measurement unit, a small chip that senses acceleration and orientation. This sensor tells the computer which way is down, effectively removing two degrees of uncertainty about the camera's tilt. Second, they tapped into the rich geometric data embedded in modern feature descriptors. Instead of just seeing a point, these descriptors describe how a small patch of the image stretches or rotates, providing extra constraints that help solve the puzzle with fewer samples.

To test their ideas, the team derived new mathematical rules that link the camera's position, its unknown focal length, and the gravity direction. They built two specific solvers to handle these rules. The first solver, which they call UP1PfAC, can determine the entire camera pose and focal length using just a single affine correspondence. In simple terms, this means it needs only one feature that describes how a small area of the image is transformed, including its scale and rotation. The second solver, UP2PfORI, requires two features that are sensitive to orientation but not necessarily to scale. In their experiments, the researchers generated thousands of synthetic scenes to test the stability of these new tools. They found that the solvers were incredibly stable, producing errors so small they were nearly invisible on a standard scale, even without any added noise. When they introduced realistic errors, such as slight inaccuracies in matching points or small jitters in the gravity sensor, the new methods outperformed existing state-of-the-art techniques, maintaining high accuracy where older methods began to falter.

The true test came when the researchers applied their solvers to real-world data from two famous datasets: the Cambridge Landmarks and the Aachen Day-Night collection. These datasets contain images of historic buildings and city centers taken with various cameras and under different lighting conditions. The team integrated their new solvers into a standard localization pipeline that uses a robust method to filter out bad data. The results showed that their approach not only matched the accuracy of the best existing methods but did so significantly faster. In the Cambridge dataset, their solvers reduced the time required to find a location while also improving the precision of the rotation and focal length estimates. In the more challenging Aachen dataset, which includes images taken at night and with different devices, their methods consistently found the correct location more often than the previous best semi-calibrated approaches. The speed advantage was particularly notable, with the new solvers completing their calculations in a fraction of the time required by older algorithms that needed more data points.

This work demonstrates that by listening to the subtle geometric whispers in feature descriptors and combining them with the steady hand of a gravity sensor, computers can navigate the world with far less effort. The researchers found that their single-feature solver was particularly effective, leveraging the rich information in affine features to act as a strong guide, while the two-feature solver offered a balanced alternative for devices that might not have access to such detailed descriptors. The study confirms that these new tools are not just theoretical improvements but practical solutions that can handle the messy reality of the real world, from noisy sensors to imperfect image matches. By reducing the number of samples needed and speeding up the calculation, this research paves the way for a new generation of low-power, high-precision localization systems. This is crucial for the future of extended reality headsets, drones, and robots, which need to know exactly where they are in a split second to function seamlessly in our shared physical spaces.

Technical Summary: Gravity-aware Partially Calibrated Absolute Pose Estimation

Problem Statement

The paper addresses the absolute pose estimation problem, specifically the Perspective-n-Point with unknown focal length (PnPf) scenario. In this setting, a camera's position and orientation (pose) and its focal length must be estimated relative to a pre-built 3D map using 2D-to-3D correspondences.

Traditional approaches to PnPf typically require a minimum of 3.5 point correspondences (e.g., the P3.5Pf or P4Pf solvers) to solve for the seven degrees of freedom (6 for pose + 1 for focal length). While point-based methods are standard, they ignore the rich local geometric information embedded in modern feature descriptors (such as SIFT or ORB), which provide affine or orientation-covariant properties. Furthermore, many consumer devices (smartphones, drones, XR headsets) are equipped with Inertial Measurement Units (IMUs) that provide a gravity vector, yet traditional solvers often fail to fully exploit this constraint to reduce the number of required correspondences.

Methodology

The authors propose a framework that fuses IMU data (gravity vector) with affine- or orientation-covariant feature descriptors to derive minimal solvers that require fewer correspondences than traditional point-based methods.

Core Assumptions and Setup

Gravity Alignment: The camera rotation is decomposed such that the pitch and roll are known (aligned with the gravity vector), leaving only the yaw angle ( $\theta$ ) as an unknown rotational parameter.
Feature Utilization: The method leverages local affine transformations ( $A$ ) derived from feature descriptors. If only orientation-covariant features are available, the scale component is eliminated.
Unknown Focal Length: The query camera's focal length ( $f$ ) is treated as an unknown parameter.

Proposed Solvers

The paper derives two novel polynomial solvers:

UP1PfAC (Upright 1-Point with Affine Correspondence):
- Input: A single affine correspondence between a query image and a reference image, combined with the gravity vector.
- Mechanism: The authors formulate constraints based on the relationship between the local affine transformation and the camera pose. By substituting the unknown focal length into the affine constraint equations and utilizing the tangent half-angle parameterization for the yaw angle, they derive a system of polynomial equations.
- Solution: The system is reduced to a quartic equation in the yaw parameter, allowing for an analytical solution. The focal length is then recovered via back-substitution. An unused constraint is employed to select the correct solution among the four candidates.
UP2PfORI (Upright 2-Point with Orientation-Covariant Features):
- Input: Two orientation-covariant features (e.g., ORB) and the gravity vector.
- Mechanism: Similar to UP1PfAC, but utilizes constraints derived from two orientation-covariant features. The scale ambiguity is resolved by subtracting the orientation constraints.
- Solution: The system is reduced to a quartic equation in the yaw parameter, with the focal length recovered subsequently.

Mathematical Derivation

The derivation involves:

Parameterizing the unknown yaw rotation using $r = \tan(\theta/2)$ to convert trigonometric terms into polynomials.
Formulating a linear system $M(r, f)[t, 1]^T = 0$ where $t$ is the translation vector.
Enforcing the condition that the matrix $M$ has a non-trivial nullspace (i.e., its subdeterminants vanish), leading to polynomial constraints in $r$ and $f$ .
Reducing the system to a single quartic equation solvable analytically.

Key Contributions

Novel Constraints: Derivation of new polynomial constraints for joint estimation of absolute pose and unknown focal length using affine or orientation-covariant features.
Minimal Solvers: Introduction of two efficient solvers:
- UP1PfAC: Solves the problem with a single affine correspondence.
- UP2PfORI: Solves the problem with two orientation-covariant features.
- Both require fewer samples than the state-of-the-art P3.5Pf or P4Pf solvers (which need 3.5 or 4 points).
Efficiency: The solvers are designed for low computational cost, facilitating faster hypothesis generation in RANSAC-like frameworks.
Comprehensive Evaluation: A rigorous comparison against state-of-the-art methods on both synthetic and large-scale real-world datasets.

Experimental Results

The authors evaluated their solvers against existing methods (P4Pf, P3.5Pf, UP2P, UP1SIFT, etc.) on synthetic data and real-world datasets (Cambridge Landmarks and Aachen Day-Night).

Numerical Stability: On noise-free synthetic data, the proposed solvers demonstrated high numerical stability, with errors comparable to or better than existing methods.
Noise Sensitivity: In the presence of Gaussian noise (image points, IMU gravity vector, and feature orientations), the proposed methods showed superior performance, particularly in focal length estimation, which is a novel capability for affine-based solvers.
Execution Time: The proposed solvers were significantly faster than the P3.5Pf and P4Pf solvers. In RANSAC frameworks with high outlier ratios, the reduced minimal sample size (1 or 2 vs. 3.5 or 4) led to a substantial reduction in total execution time to reach a consensus.
Real-World Localization:
- On the Cambridge Landmarks dataset, the proposed solvers improved estimation errors (position and rotation) compared to other semi-calibrated methods and reduced execution time.
- On the Aachen Day-Night dataset, the methods achieved competitive localization accuracy (percentage of images localized within thresholds) while maintaining faster execution times compared to point-based semi-calibrated solvers.
- The UP1PfAC solver generally outperformed UP2PfORI, attributed to the richer geometric information encoded in affine constraints.

Significance and Claims

The paper claims that by leveraging local geometric information from feature descriptors and IMU data, it is possible to solve the partially calibrated absolute pose problem with fewer correspondences than traditional point-based methods.

Efficiency: The reduction in required correspondences (from 3.5/4 to 1/2) significantly lowers the computational cost of robust estimation (RANSAC), which is critical for real-time applications.
Accuracy: The methods achieve fast and accurate localization and focal length estimation, outperforming existing semi-calibrated approaches in both synthetic and real-world scenarios.
Application: The authors position these solvers as enabling technologies for future low-energy localization on constrained devices such as Extended Reality (XR) headsets, UAVs, and embedded robotics, where speed and robustness are paramount.

The authors conclude that while modern robust estimation frameworks rely on many factors (nonlinear refinement, sampling strategies), the speed of the minimal solver remains critical. Their work demonstrates that integrating diverse information sources (deep learning descriptors, geometric priors, and sensor data) into minimal solvers can push the boundaries of robust estimation.