💻 computer science

Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks

The paper proposes DenseMarks, a novel learned representation that utilizes a Vision Transformer to predict 3D canonical embeddings for human head pixels, enabling robust, high-quality dense correspondences and tracking across diverse poses and individuals through training on point tracks from in-the-wild videos.

Original authors: Dmitrii Pozdeev, Alexey Artemov, Ananta R. Bhattarai, Artem Sevastopolsky

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dmitrii Pozdeev, Alexey Artemov, Ananta R. Bhattarai, Artem Sevastopolsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, magical library of human heads. In this library, every single person's head is organized not by their name or photo, but by a unique 3D coordinate system inside a tiny, invisible cube.

This is the core idea behind DenseMarks, a new technology described in this paper. Here is a simple breakdown of how it works, using everyday analogies.

1. The Problem: The "Landmark" Limitation

Current computer programs that track faces are like stick-figure artists. They are great at finding specific, easy-to-spot dots: the tip of the nose, the corner of the mouth, or the center of an eye.

The Issue: If you turn your head, wear a hat, or have long hair covering your ear, the "stick-figure" gets confused. It loses track because it can't see those specific dots anymore. It's like trying to follow a car in a race by only watching its license plate; if the plate gets covered by mud, you lose the car.

2. The Solution: The "Universal Head Map"

DenseMarks changes the game. Instead of looking for a few dots, it looks at every single pixel of the head image (including hair, ears, and accessories) and assigns it a specific address in that invisible 3D cube.

The Analogy: Imagine every human head is a city.
- Old Way: We only have a map of the post office and the library (the nose and eyes).
- DenseMarks Way: We have a GPS for every single street, house, and tree in the city, even the ones hidden behind a wall (like hair or a hat).
- The Magic: No matter who the person is (a child, an adult, a person with red hair or black hair), their "Left Ear" always lives at the exact same coordinate in this invisible 3D cube. Their "Forehead" is always at another specific coordinate.

3. How They Taught the Computer (The "Twin" Trick)

To teach the computer this map, the researchers didn't use perfect 3D scans (which are hard to get). Instead, they used a clever trick with video.

The Setup: They took thousands of videos of people talking (like interviews).
The Tool: They used a super-smart "point tracker" (like a digital highlighter) to follow thousands of tiny dots on a person's face as they moved their head.
The Lesson: The computer was shown two frames from the same video (Frame A and Frame B). It was told: "Hey, the dot on the left eyebrow in Frame A is the same dot as the one on the left eyebrow in Frame B. Even though the head moved, they are the same 'address' in our 3D cube."
The Result: The computer learned to ignore the messy details (lighting, hair style) and focus on the structure. It learned that "Left Ear" = "Coordinate (0.2, 0.5, 0.1)" for everyone.

4. Why This is a Big Deal

Because the computer understands the "address" of every part of the head, it can do things that were previously impossible:

The "Find My Ear" Trick: You can click on a person's left ear in a photo, and the computer can instantly find the left ear in any other photo, even if that person is wearing a scarf or has their head turned sideways.
The "Morphing" Trick: If you want to swap the hair of Person A onto the head of Person B, the computer knows exactly which pixels belong to the hair and which belong to the skin, making the swap look natural rather than like a bad Photoshop job.
The "3D Rebuild": By looking at two photos of a person from different angles, the computer can use these "addresses" to figure out exactly how deep the nose is or how round the head is, creating a 3D model from just 2D pictures.

5. The "Secret Sauce"

The paper mentions a few special ingredients that make this work:

The Cube: They use a 3D cube (like a Rubik's cube) as the map. This is better than a flat map (like a world map) because it handles 3D objects (like the back of the head or hair sticking out) without getting twisted or torn.
The "Smoothie" Filter: They made sure the map is "smooth." This means if you move your finger slightly from the nose to the cheek, the coordinates change gradually, not in a jagged jump. This prevents the computer from getting confused.
Robustness: Even if the video is blurry, the lighting is bad, or the person is wearing sunglasses, the system is surprisingly good at guessing the right "address" because it learned the structure of the head, not just the look.

Summary

DenseMarks is like giving every human head a unique, universal 3D GPS system. It allows computers to understand that a nose is a nose, and an ear is an ear, regardless of who they belong to or what they are wearing. This makes it much easier to track people in videos, create realistic 3D avatars for movies and games, and fix messy photos.

It's a step forward from "recognizing a face" to truly understanding the geometry of a head in 3D space.

1. Problem Statement

Current human head modeling and tracking methods face significant limitations:

Sparse vs. Dense: Traditional methods rely on sparse facial landmarks (eyes, nose, mouth) or parametric 3D models (3DMMs). These struggle with occlusions, extreme poses, and non-facial features like hair, accessories, and clothing, which are often omitted or poorly tracked.
Robustness: Existing dense feature extractors (e.g., Vision Foundation Models like DINOv3) often match based on visual texture or color rather than semantic geometry, leading to artifacts in correspondence tasks.
Canonical Space: While canonical spaces (like NOCS) exist for general objects, they are difficult to apply to human heads due to the lack of large-scale 3D annotated datasets and the complexity of modeling the entire head (including hair) in a UV space.

The authors propose a need for a dense, semantic, and view-invariant representation that covers the entire head (including hair and accessories) and enables robust correspondence search across different individuals and poses.

2. Methodology: DenseMarks

The paper introduces DenseMarks, a learned representation that maps 2D human head images into a 3D canonical volumetric space (a unit cube).

Core Architecture

Embedder Network ( $\psi_\theta$ ): A Vision Transformer (ViT) backbone (initialized with pre-trained DINOv3) predicts a dense 3D embedding vector for every pixel in the input image.
Canonical Space (The Unit Cube): Instead of a 2D UV map, the method uses a 3D unit cube. This space is discretized into a voxel grid ( $N_d \times N_d \times N_d$ ).
Latent Feature Matrix ( $E$ ): A learnable matrix stores $D$ -dimensional latent features for each voxel in the grid.
Feature Extraction: To get the semantic feature for a specific pixel, the network predicts a 3D coordinate in the cube. It then queries the latent matrix $E$ at this coordinate using trilinear interpolation to retrieve the semantic feature.

Training Strategy

The model is trained in a Siamese fashion using pairs of frames from the same talking-head video. Since ground-truth dense correspondences are unavailable, the authors use pseudo-ground truth generated by state-of-the-art point trackers (CoTracker3).

The training objective combines four loss functions:

Contrastive Loss: Encourages semantic features of matched points (from the point tracker) in different frames to be close in the embedding space, while pushing unmatched points apart. This enforces cross-subject consistency.
Landmark Loss: Anchors standard facial landmarks (e.g., 300W format) to predefined, fixed locations within the 3D unit cube. This ensures the canonical space is interpretable and structured.
Segmentation Loss: A segmentation head predicts face parsing regions (skin, hair, eyes, etc.) from the extracted features, ensuring the latent space aligns with semantic regions.
Spatial Smoothness: A 3D Gaussian filter is applied to the latent feature matrix $E$ during training. This ensures that nearby voxels in the canonical space have similar features, creating a smooth, continuous manifold.

3. Key Contributions

Dense Canonical Representation: A novel 3D volumetric embedding space that covers the entire human head, including difficult-to-annotate regions like hair and accessories, unlike traditional 3DMMs.
Robust Training via Point Tracks: A method to train dense correspondences without manual 3D annotations by leveraging off-the-shelf point trackers on in-the-wild video datasets (CelebV-HQ).
Interpretability and Queryability: The canonical space is structured such that specific semantic parts (e.g., "left ear," "forehead center") map to fixed, queryable locations in the 3D cube, enabling intuitive interactions (e.g., click-based retrieval).
Low-Dimensional Efficiency: Despite using a very low-dimensional embedding (3D coordinates) to store semantics, the method outperforms high-dimensional foundation model features in geometry-aware tasks.

4. Experimental Results

The authors evaluated DenseMarks on dense warping, monocular head tracking, and stereo reconstruction.

Point Querying & Region Selection: The method successfully identifies specific semantic points (e.g., ear centers, eyebrow corners) across different identities and poses with high accuracy, outperforming baselines like Sapiens, CSE, and DINOv3.
Dense Warping: When warping pixels from a source to a target image based on nearest-neighbor search in the embedding space, DenseMarks produces semantically consistent results. In contrast, baselines often match based on color or produce artifacts.
- Metrics: On the Nersemble dataset, DenseMarks achieved a MAE of 3.68 (vs. 7.60 for DINOv3) and PCK@0.05 of 0.90 (vs. 0.72 for DINOv3).
Monocular Head Tracking: Integrated with a standard 3DMM tracker (VHAP), DenseMarks significantly improved robustness against extreme poses, occlusions (hair/glasses), and rapid motion, reducing tracking failures.
Stereo Reconstruction: The embeddings enabled accurate triangulation of 3D points from 2-view correspondences without explicit 3D supervision.
Ablation Studies:
- Removing the canonical space (training without the 3D grid) degraded performance.
- Removing landmark or segmentation losses made region finding unreliable.
- The method showed high robustness to noise in the pseudo-ground truth (point tracks), even with significant Gaussian noise or missing tracks.

5. Significance and Impact

Bridging 2D and 3D: DenseMarks provides a bridge between 2D video data and 3D understanding without requiring expensive 3D annotations.
Generalization: The method generalizes well to challenging conditions (extreme poses, occlusions) where traditional landmark-based trackers fail.
Foundation for Future Applications: The structured canonical space opens doors for applications in AR/VR (realistic avatars), video editing (semantic manipulation), and 3D reconstruction.
Efficiency: By learning a compact 3D representation, it offers a more efficient alternative to high-dimensional feature matching for human head tasks.

In conclusion, DenseMarks establishes a new state-of-the-art for human head correspondence by combining the power of foundation models with a structured, learnable 3D canonical space, guided by robust point tracking supervision.