💻 computer science

Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction

This paper introduces Hestia, a voxel-face-aware hierarchical next-best-view planner that overcomes the limitations of existing reinforcement learning-based approaches by employing a diverse dataset, hierarchical search, a close-greedy strategy, and face-aware design to achieve significantly improved 3D reconstruction coverage and accuracy while maintaining real-time inference.

Original authors: Cheng-You Lu, Zhuoli Zhuang, Nguyen Thanh Trung Le, Da Xiao, Yu-Cheng Chang, Thomas Do, Srinath Sridhar, Chin-teng Lin

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cheng-You Lu, Zhuoli Zhuang, Nguyen Thanh Trung Le, Da Xiao, Yu-Cheng Chang, Thomas Do, Srinath Sridhar, Chin-teng Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D digital twin of a mysterious object, like a statue or a piece of furniture, but you can only take photos with a camera mounted on a drone. The problem is: where should the drone fly next to get the best picture?

If you just fly in a random circle, you might miss the back of the statue's head or the underside of a table leg. If you rely on a pre-planned path, you might get stuck in a spot where the object blocks your view of itself. This is the "Next-Best-View" problem.

The paper introduces Hestia, a smart AI system that acts like a highly skilled, curious photographer drone. Instead of guessing or following a rigid script, Hestia decides exactly where to fly next to build the most complete 3D model possible, using the fewest photos.

Here is how Hestia works, explained through simple analogies:

1. The "Cube vs. Dot" Insight

Most previous AI systems treated the 3D world like a collection of tiny dots. If you look at a dot, you only see one side. But in reality, objects have volume.

The Old Way: Imagine trying to paint a basketball by only seeing the tiny dot where your brush touches it. You might miss the curves on the sides.
Hestia's Way: Hestia treats every tiny piece of the object as a small cube with six faces (top, bottom, front, back, left, right). It asks, "Have I seen the top of this cube? What about the bottom?"
The Result: By checking all six sides of every tiny cube, Hestia ensures it doesn't miss hidden details, like the underside of a chair or the back of a teddy bear's ear. The paper claims this "cube" approach captures about 22% more hidden surface area than the old "dot" method.

2. The "Look First, Fly Second" Strategy

Predicting exactly where to fly in 3D space (up/down, left/right, forward/back) and which way to tilt the camera all at once is incredibly hard for a computer. It's like trying to solve a Rubik's cube while juggling.

The Hierarchy: Hestia breaks this big problem into two smaller steps, like a human pilot:
1. Step 1 (The Gaze): "Where should I look?" It picks a specific point on the object that needs attention (e.g., "Look at that broken window").
2. Step 2 (The Flight): "Now, where should I stand to get the best view of that point?"
The Benefit: This "look-then-fly" approach makes the math much easier and faster, allowing the drone to make decisions in real-time (about 25 times per second).

3. The "Greedy" Photographer

In many AI systems, the computer tries to plan the entire future mission at once. It might take a "bad" photo now because it thinks it will lead to a "great" photo later. This often leads to confusion and wasted time.

Hestia's Approach: Hestia uses a "close-greedy" strategy. It asks, "What is the single best photo I can take right now to reveal the most new information?"
The Analogy: Imagine cleaning a messy room. A "long-term planner" might move a chair to get to a box, but then realize the chair was blocking a light switch they needed. A "greedy" cleaner just picks up the biggest pile of clothes right in front of them. Hestia focuses on immediate, visible improvements, which surprisingly leads to a cleaner room (better 3D model) faster.

4. Training on a "Universe" of Objects

To learn how to be a good photographer, Hestia didn't just practice on a few toy boxes. It was trained on Objaverse, a massive dataset containing over 800,000 different 3D shapes—from animals and furniture to vehicles and plants.

The Result: Because it saw so many different shapes, Hestia is robust. If you put a chair in a corner, or a statue in the middle of the room, Hestia knows how to handle it. It doesn't get confused by the object's position.

The Results: Faster, Better, and Real

The paper tested Hestia against other top methods and found:

More Complete Models: It covered at least 4% more of the object's surface area.
Fewer Mistakes: The 3D models were 50% more accurate (less "fuzzy" or distorted).
Efficiency: With a limit of only 5 photos, Hestia achieved a 92% complete model, whereas other methods needed 15 photos to reach a similar level.
Real-World Proof: The team didn't just simulate this on a computer. They put Hestia on a real drone (a DJI Mini 3 Pro) flying indoors. Even without a special depth camera, using a standard camera and a smart software trick to guess depth, the drone successfully flew around objects and built 3D models in the real world.

In summary: Hestia is a smarter, faster, and more thorough way for robots to "see" the world. By treating objects as 3D cubes, breaking the flight path into simple steps, and focusing on immediate progress, it builds better 3D maps with fewer photos than ever before.

Technical Summary: Hestia – Voxel-Face-Aware Hierarchical Next-Best-View Acquisition

Problem Statement
While advances in 3D reconstruction and novel view synthesis have enabled photorealistic rendering, the acquisition of images for reconstruction remains a bottleneck. Current data collection is largely manual, time-consuming, or constrained by simple preplanned camera trajectories that often fail to capture occluded regions. Existing learning-based Next-Best-View (NBV) planners, particularly those using reinforcement learning (RL) to generalize across scenes, face limitations in robustness and performance across diverse object geometries. Specifically, prior generalizable methods often treat voxels as points, leading to information loss, and struggle with high-dimensional continuous action spaces (5-DoF: x, y, z, yaw, pitch) and spurious correlations between current actions and distant future rewards.

Methodology
The paper introduces Hestia (Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction), a generalizable, RL-based NBV planner designed for object-centric scenes. Hestia addresses the shortcomings of prior approaches through four core components:

Voxel-Face-Aware Representation:
Unlike previous works that treat voxels as points (rays), Hestia treats each voxel as a cube, explicitly modeling the visibility of its six faces. This is grounded in the "coupon collector's problem," where sampling rays against point-voxels leaves approximately $k^{-1/6}$ of faces unseen in a scene of $k$ cubes. By incorporating face visibility into both the observation state and the reward function, Hestia ensures more comprehensive geometric capture with minimal computational overhead.
Hierarchical Action Space:
To manage the high-dimensional continuous 5-DoF action space, Hestia employs a hierarchical network structure. Instead of predicting the full viewpoint in one step, the policy first predicts a look-at point ( $L_t$ ) using a proposal network (3D CNN with self-attention). Subsequently, a grid encoder and trilinear interpolation extract features based on $L_t$ to predict the 3-DoF camera position ( $a_t$ ). This decomposition simplifies the search space and aligns with the intuition of focusing on a target before planning the flight path.
Close-Greedy Optimization Strategy:
Hestia formulates the NBV task as a close-greedy optimization problem. The reward function prioritizes immediate improvements in coverage ratio rather than relying on large terminal rewards. By using a small reward discount factor ( $\gamma = 0.1$ ), the method mitigates spurious correlations, where suboptimal current actions are incorrectly associated with large future rewards. The reward is calculated based on the increment of newly visible voxel faces, scaled to balance positive gains against penalties for collisions or invalid actions.
Diverse Dataset and Training:
The model is trained on a significantly larger and more diverse dataset processed from Objaverse (120,000 shapes across 18+ categories), moving beyond the limited cubic shapes found in previous benchmarks like Houses3K. The training setup uses a Markov Decision Process (MDP) where the state includes the grayscale image, camera pose, look-at point, and an aggregated occupancy grid containing cumulative face visibility.

Key Contributions

Voxel-as-Cube Paradigm: A novel observation and reward design that treats voxels as cubes rather than points, reducing information loss and improving geometric coverage.
Hierarchical Planning: A two-stage prediction network (look-at point followed by position) that effectively handles the 5-DoF continuous action space in RL-based generalizable planning.
Robust Learning Scheme: The integration of a close-greedy strategy to reduce spurious correlations and the use of a large-scale, diverse dataset to ensure robustness across varying object shapes and placements.
Real-World Feasibility: A demonstration of Hestia operating on a drone with an RGB camera (using a depth predictor for state estimation) in real-world indoor environments, validating its applicability beyond simulation.

Experimental Results
Hestia was evaluated on three datasets: OmniObject3D, Objaverse, and Houses3K, comparing against state-of-the-art generalizable and online-learning baselines (e.g., GenNBV, ScanRL, NeU-NBV).

Performance Gains: Hestia achieved at least a 4% gain in coverage ratio (CR) and a 6% gain in Area Under the Curve (AUC) compared to prior methods, while reducing the Chamfer Distance (CD) by 50%.
Efficiency: With a limited budget of 5 images, Hestia achieved a 92% coverage ratio, outperforming the next-best method (GenNBV) by at least 12%.
Robustness: The method maintained consistent performance across different object placement positions (center vs. corners) and diverse object categories, showing less fluctuation than baselines.
Speed: Hestia operates at 25 FPS, enabling real-time inference without the need for candidate viewpoint sampling or online optimization.
Ablation Studies: Removing any of the core components (face-aware design, greedy strategy, or hierarchical structure) resulted in performance degradation, confirming the necessity of the full system.

Significance
The paper claims that Hestia provides a non-marginal improvement over existing NBV planners, addressing critical limitations in robustness, geometric detail capture, and action space complexity. By treating voxels as cubes and employing a hierarchical, greedy-optimized learning scheme, Hestia enables efficient 3D reconstruction under strict acquisition budgets. The successful deployment on a real-world drone system demonstrates that generalizable, RL-based NBV planning is feasible for practical applications, particularly in power-constrained scenarios where rapid, autonomous data collection is required. The work establishes a foundation for future multi-agent extensions and large-scale outdoor scanning.