🤖 AI

InCaRPose: In-Cabin Relative Camera Pose Estimation Model and Dataset

The paper presents InCaRPose, a Transformer-based model trained on synthetic data that achieves robust, real-time, metric-scale relative camera pose estimation for highly distorted in-cabin automotive environments, addressing critical calibration challenges for driver monitoring systems.

Original authors: Felix Stillger, Lukas Hahn, Frederik Hasecke, Tobias Meisen

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Felix Stillger, Lukas Hahn, Frederik Hasecke, Tobias Meisen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in the driver's seat of a car. There is a camera mounted on your rearview mirror, watching you and the passengers. This camera is like a pair of eyes for the car's "brain," helping it know if you are looking at the road or if a child is unbuckling their seatbelt.

But here's the problem: Rearview mirrors move. You tilt them up, down, left, or right to see better. Every time you do that, the camera's view changes. For the car's computer to understand what it's seeing, it needs to know exactly where the camera is pointing at that exact moment. If it guesses wrong, it might think you are looking at the road when you are actually looking at your phone, or it might deploy an airbag at the wrong time.

This is where InCaRPose comes in. Think of it as a super-smart, instant "camera GPS" that works even when the camera lens is warped like a fishbowl.

Here is a simple breakdown of how it works and why it's special:

1. The "Fishbowl" Problem

Most car cameras use wide-angle lenses (fisheye) to see the whole cabin. But these lenses make straight lines look curved, like looking through a funhouse mirror.

Old Way: Traditional computers try to "un-curve" the image first, like trying to flatten a crumpled piece of paper before reading it. This is slow and often loses important details at the edges.
InCaRPose Way: This model is like a wizard who can read the crumpled paper without flattening it first. It understands the "fishbowl" distortion naturally and still figures out exactly where the camera is pointing.

2. The "Virtual Training" Trick

Training a robot to do this usually requires thousands of real photos of real car interiors. But taking photos of every possible car, with every possible mirror angle, in every lighting condition is a nightmare.

The Solution: The creators built a virtual video game world (using software called Blender) with fake cars and fake passengers. They trained the AI entirely inside this game.
The Magic: Usually, a robot trained in a video game fails when it sees the real world. But InCaRPose is like a student who studied in a simulation but walked into a real exam and aced it. It learned the geometry of the car interior so well that it didn't need to see a single real car photo during training.

3. The "Reference Frame" Shortcut

Imagine you are trying to tell a friend how to move a chair.

The Hard Way: You say, "Move the chair 5 meters North, 2 meters East, and 1 meter up from the Earth's center." (This is hard because every car is in a different spot).
The InCaRPose Way: You say, "Move the chair 10 inches to the right and 5 inches up from where it was a second ago."
Why it matters: The model doesn't care about the car's location in the world. It only cares about the change between the "standard" view and the "current" view. This makes it work in any car, from a tiny hatchback to a massive truck, without needing to be retrained.

4. Speed and Safety

In a crash, an airbag has to deploy in about 15 to 50 milliseconds. That is faster than a human blink.

InCaRPose is incredibly fast. It can figure out the camera's position in about 15 milliseconds (roughly 60 times a second).
It gives the answer in real-world units (meters), not just "it moved a little bit." This is crucial. If the system knows a passenger is sitting 20cm closer to the airbag, it can adjust the airbag's force to prevent injury.

5. The "New Dataset" Gift

The researchers didn't just build the brain; they also built a test. They created a new, open-source dataset called In-Cabin-Pose.

Think of this as a "driving test" for other AI researchers. It contains real photos of car interiors with the exact camera positions marked (using special invisible markers).
They are sharing this with the world so everyone can build better safety systems.

Summary Analogy

Imagine you are trying to navigate a maze while wearing glasses that distort your vision.

Old AI: Tries to take off the glasses, fix the distortion, and then navigate. It's slow and sometimes the glasses are too warped to fix.
InCaRPose: Wears the glasses, learns exactly how the distortion works, and navigates the maze instantly. It learned the maze in a video game, but it can navigate the real maze perfectly.

The Bottom Line: InCaRPose is a fast, smart, and flexible tool that helps cars "see" their passengers clearly, even when the camera moves or the lens is weird. This means safer airbags, better driver monitoring, and a future where cars truly understand the people inside them.

1. Problem Statement

The paper addresses the critical challenge of camera extrinsic calibration within automotive interiors (In-Cabin Monitoring or ICAM).

Context: Modern vehicles use multiple interior cameras (often fisheye/NIR) for driver monitoring, occupant detection, and airbag deployment. These cameras are frequently mounted on dynamic components like rear-view mirrors, which are manually or automatically adjusted, causing frequent changes in extrinsic parameters.
Challenges:
- High Distortion: Interior cameras often use wide-angle or fisheye lenses, causing severe geometric distortion that degrades traditional feature-matching methods.
- Metric Scale Ambiguity: Most deep learning relative pose methods predict translation only up to an unknown scale. However, safety-critical applications (e.g., airbag deployment) require absolute metric translation (in meters) to determine occupant distance within milliseconds.
- Data Scarcity & Domain Shift: Collecting real-world ground truth in confined car interiors is difficult and invasive. Existing models often require large datasets or specific camera intrinsics, failing to generalize across different vehicle models or from synthetic to real data.
- Real-Time Constraints: Safety systems require inference latencies under 15–50ms, necessitating lightweight, efficient models.

2. Methodology: InCaRPose

The authors propose InCaRPose, a Transformer-based architecture designed for robust, metric-scale relative pose estimation between image pairs.

A. Problem Formulation

Instead of predicting absolute poses in a vehicle-specific coordinate frame (which hinders generalization across car models), the task is reformulated as Reference-Relative Pose Estimation.

Given a calibrated reference view ( $T_{v1}$ ) and a target view ( $T_{v2}$ ), the model predicts the relative transformation $T_{rel}$ such that $T_{v2} = T_{v1} \cdot T_{rel}$ .
This approach is vehicle-agnostic, allowing deployment across different cabin configurations without retraining.

B. Architecture

The model utilizes a frozen Vision Transformer (ViT) backbone (specifically DINOv3) and a learnable decoder:

Frozen Backbone: Uses DINOv3 (or DINOv2/DUNE) to extract patch-level features. The backbone is frozen to leverage pre-trained, domain-invariant geometric features and prevent overfitting on the limited synthetic dataset.
Cross-Attention Decoder: A Transformer decoder processes features from both the reference and target images. It employs:
- Self-attention to refine features within each view.
- Cross-attention to model the geometric relationship between the two views.
- 2D Rotary Positional Embeddings (RoPE) to handle spatial sensitivity without learned positional tokens.
Prediction Head: A lightweight MLP head regresses the relative pose (rotation and translation).
- Output: The model predicts the pose in both directions (View 1 $\to$ 2 and 2 $\to$ 1) to enforce geometric consistency, though only one direction is used at inference for speed.
- Representation: The model supports various rotation representations (Euler, Quaternions, Rotation Matrices), with Quaternions + a specific Quaternion-based Pose Loss yielding the best results.

C. Training Strategy

Synthetic-to-Real: The model is trained exclusively on synthetic data generated in Blender.
- The synthetic dataset includes 8 vehicle models with randomized occupants and objects.
- It simulates fisheye distortion and NIR lighting conditions.
Data Efficiency: Despite training on only ~6,500 synthetic image pairs, the model generalizes effectively to real-world data without fine-tuning.
Preprocessing: Images are zero-padded (not center-cropped) to preserve the full field of view (FOV) of fisheye lenses, which is critical for edge geometric cues.

3. Key Contributions

In-Cabin-Pose Dataset: The release of a real-world test dataset containing highly distorted, wide-FOV NIR fisheye images with metric ground truth (derived from ArUco markers).
Vehicle-Agnostic Formulation: A novel problem definition that predicts relative pose with respect to a reference view, eliminating the need for vehicle-specific coordinate frames.
End-to-End Fisheye Processing: The model processes distorted fisheye images directly without undistortion, preserving corner information and achieving metric-scale translation.
Synthetic-to-Real Generalization: Demonstrates that training on limited synthetic data is sufficient to recover large relative pose changes and generalize to unseen real-world cabins and camera intrinsics.
Real-Time Performance: Achieves high precision (median rotation error ~2.75°) while running at >45 FPS (up to ~70 FPS for smaller backbones) on consumer GPUs, meeting safety-critical latency requirements.

4. Experimental Results

The model was evaluated on the proprietary In-Cabin-Pose dataset, the public 7-Scenes (indoor), and Cambridge Landmarks (outdoor).

In-Cabin Performance:
- InCaRPoseLarge224 achieved a median translation error of 0.07m and a median rotation error of 2.75°.
- It significantly outperformed classical SIFT matching (which fails to provide metric scale) and large-scale baselines like Reloc3r (which only predicts direction, not metric scale).
- It successfully handled real-world sensor noise and dynamic lighting without retraining.
Generalization (7-Scenes & Cambridge):
- Achieved competitive or superior results compared to state-of-the-art methods (Reloc3r, RelPoseNet) on 7-Scenes, with a mean rotation error of 2.21° (65% improvement over Reloc3r).
- Performance on outdoor Cambridge Landmarks was lower than indoor results due to scale variations and complex scenes, but still competitive.
Ablation Studies:
- Backbone: DINOv3 outperformed DINOv2 and DUNE. Larger backbones (ViT-Large) improved accuracy but at a higher computational cost.
- Loss Function: The Quaternion-based Pose Loss combined with Euclidean translation loss yielded the lowest rotation errors.
- Resolution: Increasing input resolution from 224 to 512 improved rotation accuracy but had diminishing returns on translation error.

5. Significance

Safety-Critical Application: By providing absolute metric translation in real-time, InCaRPose enables reliable occupant positioning for safety systems like adaptive airbag deployment, which must decide within 15–50ms.
Cost-Effective Deployment: The ability to train on synthetic data and generalize to real-world, vehicle-agnostic scenarios drastically reduces the cost and complexity of deploying in-cabin monitoring systems across diverse vehicle fleets.
Robustness: The end-to-end handling of fisheye distortion without undistortion simplifies the pipeline and preserves critical geometric information often lost in traditional preprocessing.

In summary, InCaRPose bridges the gap between synthetic training and real-world automotive safety applications, offering a data-efficient, real-time, and metric-accurate solution for relative camera pose estimation in challenging, distorted environments.