💻 computer science

NBAvatar: Neural Billboards Avatars with Realistic Hand-Face Interaction

NBAvatar is a novel avatar rendering method that combines oriented planar primitives with neural rendering to achieve realistic, temporally consistent, and high-fidelity head avatars capable of handling complex non-rigid hand-face interactions, outperforming existing approaches in both novel-view and novel-pose rendering quality.

Original authors: David Svitov, Mahtab Dahaghin

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: David Svitov, Mahtab Dahaghin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to create a perfect digital twin (an avatar) of yourself for a video call or a video game. You want it to look so real that people can't tell the difference. But there's one tricky situation that breaks most digital avatars: when you touch your face.

Think about it. When you rub your eye, scratch your cheek, or rest your chin on your hand, your skin stretches, squishes, and changes color because of the pressure and the shadow your hand casts. Most current avatars just "glitch" here. They might make your face look blurry, or they might show your hand floating weirdly above your skin, or they might fail to show the shadow your hand casts on your cheek.

The paper "NBAvatar" introduces a new way to build these avatars that solves this problem. Here is how it works, explained with some simple analogies:

1. The Old Way: The "Fuzzy Balloon" Problem

Previous methods used something called 3D Gaussian Splatting. Imagine trying to paint a realistic portrait using thousands of tiny, fuzzy, floating balloons.

The Good: It's fast and looks okay from a distance.
The Bad: When your hand touches your face, those fuzzy balloons get confused. They might pop out of your face like a pimple, or the texture of your skin might turn into a blurry mess. It's like trying to mold clay with a balloon; it just doesn't hold the shape well when you press on it.

2. The New Way: The "Smart Post-it Notes" (Neural Billboards)

The authors of NBAvatar came up with a hybrid solution they call Neural Billboards.

Imagine your avatar's face isn't made of balloons, but is covered in thousands of tiny, flat Post-it notes (or billboards) that are stuck perfectly flat against your skin.

The "Billboard" part: These notes are physical (explicit) shapes. They know exactly where they are and how they are oriented. This keeps the shape of your face and hand crisp and sharp, even when they touch.
The "Neural" part: Instead of just having a static picture printed on the Post-it note, these notes have a "smart brain" (a neural texture). This brain can change the color and shading of the note depending on what's happening.

3. How They Handle the "Touch"

Here is the magic trick: The Smart Post-it notes talk to each other.

When your hand moves close to your face, the "smart brain" on the Post-it notes on your cheek sees the hand nearby. It doesn't need a special instruction manual saying, "If hand is here, make shadow." Instead, because the notes are all packed together in a shared space, the system naturally figures out: "Oh, a hand is right next to this note, so I should make this note darker to look like a shadow."

It's like a crowd of people at a concert. If someone stands up, the people around them naturally shift and react. The system learns these reactions automatically without needing to be explicitly told every single rule.

4. The "Training Camp" (Disentanglement)

There was a big problem with mixing these two ideas. Sometimes, the system would get lazy. Instead of moving the Post-it note to the right spot (geometry), it would just change the color of the note to fake the movement (appearance). This leads to a messy, glitchy look.

The authors created a special training rule (a "segmentation loss"). Think of this as a strict coach during practice. The coach says: "Stop cheating! If the hand moves, the Post-it note must physically move with the skin. Don't just change the color to fake it." This ensures the shape stays perfect while the colors change realistically.

Why This Matters

Realism: When you touch your face, the avatar doesn't glitch. The skin squishes realistically, and the shadows look natural.
Clarity: No more blurry faces or floating "fuzzy balloons" sticking out of your chin.
Versatility: It works even if you move your hand to a position the computer has never seen before (like scratching your ear instead of your cheek).

In a nutshell: NBAvatar builds avatars using a mix of rigid, flat shapes (to keep the geometry sharp) and smart, changing colors (to handle shadows and skin stretching). By teaching these two parts to work together without stepping on each other's toes, they created the most realistic hand-face interactions we've seen in digital avatars so far.

1. Problem Statement

The paper addresses the challenge of photo-realistic rendering of head avatars that involve non-rigid hand-face interactions (e.g., touching the face, scratching the nose).

Current Limitations: Existing methods typically handle heads or hands separately. Methods that attempt to render interactions (like InteractAvatar) often rely on 3D Gaussian Splatting (3DGS) combined with MLPs. However, these approaches suffer from:
- Artifacts: "Frosty" patterns along body boundaries, protruding Gaussians, and blurry facial textures.
- Lack of Detail: Difficulty in capturing high-frequency appearance changes (shadows, skin deformation, color shifts) caused by contact.
- Overfitting: View-dependent overfitting in screen-space methods or slow inference in volumetric NeRF methods.
Goal: To create an avatar system that handles dynamic geometry (non-rigid deformation) and complex appearance changes (shadows, occlusions) under novel views and novel poses in real-time.

2. Methodology: NBAvatar

The core innovation is the Neural Billboard primitive, a hybrid representation combining explicit geometry with implicit neural rendering.

A. Neural Billboards (Hybrid Representation)

Instead of using standard 3D Gaussians or fixed meshes, NBAvatar represents the scene using oriented planar primitives (billboards) anchored to a parametric mesh (FLAME for the face, MANO for the hand).

Explicit Geometry: The position ( $\mu$ ), scale ( $s$ ), and rotation ( $r$ ) of each billboard are derived from the underlying mesh polygons, ensuring geometric consistency.
Implicit Appearance: Instead of storing static RGB textures, each billboard carries a learnable Neural Texture ( $T^{NT}$ ) with 6 channels (feature vectors) and an Alpha map ( $T^\alpha$ ).
Rendering Pipeline:
1. Rasterization: Billboards are projected to screen space, accumulating the 6-channel feature map ( $I_f^{NB}$ ) and opacity map ( $I_\alpha^{NB}$ ).
2. Deferred Neural Rendering (DNR): A U-Net based renderer ( $R$ ) processes the 6-channel feature map to decode the final RGB image. This allows the model to learn view-dependent effects and complex interactions (like shadows) through spatial feature aggregation rather than explicit physics simulation.

B. Animation and Driving

Parametric Fitting: The system fits FLAME (face) and MANO (hand) models to multi-view input videos.
Deformation: Coarse deformations caused by hand contact are estimated using Position-Based Dynamics (PBD) simulation.
Anchoring: Neural Billboards are anchored to the mesh surface. Their global orientation is updated based on the transformation of the underlying polygon, ensuring they move and rotate with the avatar's animation.

C. Training Strategy: Geometry-Appearance Disentanglement

A major challenge is that the optimization can become ill-posed: the system might try to explain shading changes by moving the geometry (billboards) rather than updating the neural features. To solve this, the authors introduce:

Intermediate Segmentation Loss ( $\mathcal{L}_{NB}$ ): A Dice loss is applied directly to the rasterized alpha map ( $I_\alpha^{NB}$ ) against the ground-truth silhouette. This forces the billboards to strictly align with the object's geometry, preventing them from drifting to fake shadows or appearance changes.
Regularization: KNN regularization is applied to neighboring billboards to ensure smooth transitions and prevent noise.
No Explicit Interaction Modules: The system does not use specific "interaction-conditioned" modules. Instead, the U-Net renderer implicitly learns contact dynamics because hand and face features are aggregated in the shared screen space, allowing the decoder to modulate responses based on spatial proximity.

3. Key Contributions

Neural Billboards: A novel hybrid representation that reconciles surface-aligned planar primitives (explicit) with deferred neural rendering (implicit), enabling high-fidelity detail and real-time performance.
Stabilized Training Scheme: A geometry-aware training technique using intermediate silhouette supervision to disentangle rigid geometry from view/pose-dependent appearance, ensuring stable joint optimization.
Implicit Interaction Modeling: Demonstrates that screen-space spatial feature aggregation is sufficient to model complex hand-face interactions (shadows, deformations) without needing explicit interaction-conditioned modules.

4. Experimental Results

The method was evaluated on the Decaf dataset (multi-view hand-face interaction videos) against state-of-the-art baselines: GaussianAvatars, SplattingAvatar, and InteractAvatar.

Quantitative Performance (Megapixel Resolution):
- LPIPS (Perceptual Quality): NBAvatar achieved a 30% reduction in LPIPS compared to SplattingAvatar and a 26.3% reduction compared to GaussianAvatars. Lower LPIPS indicates significantly better perceptual quality.
- PSNR & SSIM: The method achieved competitive or superior scores in structural similarity and peak signal-to-noise ratio.
- Comparison with InteractAvatar: While InteractAvatar reported higher PSNR in their specific protocol (likely due to different preprocessing/cropping), NBAvatar achieved higher SSIM (structural similarity) and produced visually sharper, more realistic results with fewer boundary artifacts.
Qualitative Results:
- Artifacts: NBAvatar eliminates the "frosty" boundary artifacts and protruding Gaussians seen in 3DGS-based methods.
- Details: It produces sharp facial textures and realistic contact shadows/deformations that Gaussian-based methods blur out.
- Generalization: The method successfully generalizes to novel poses (self-reenactment) and novel views, maintaining realism even for unseen hand-face configurations.

5. Significance and Limitations

Significance: NBAvatar bridges the gap between the speed of Gaussian Splatting and the visual fidelity of neural rendering. It is the first method to effectively handle the non-rigid deformations and color changes of hand-face interactions in a photo-realistic manner, which is critical for telepresence and VR applications.
Limitations:
- Dependency on 3DMM: The method's performance is sensitive to the quality of the initial FLAME/MANO fitting. Inaccuracies in the parametric model fitting can affect objective metrics.
- Future Work: The authors plan to investigate pose-dependent warping mechanisms to better compensate for fitting errors.

In summary, NBAvatar represents a significant step forward in avatar rendering by introducing a hybrid primitive that leverages the stability of explicit geometry and the expressiveness of neural textures to solve the difficult problem of realistic hand-face interaction.