💻 computer science

VisTa3D: A Dataset and Benchmark for Thin Object Reconstruction from Vision, Tactile, and 3D Point Clouds

This paper introduces VisTa3D, the first dataset and benchmark comprising synchronized visual, depth, and tactile data for 70 thin objects, demonstrating that current 3D reconstruction models struggle with thin structures and proposing a novel visual-range-tactile baseline to improve reconstruction fidelity using tactile feedback.

Original authors: Shania Guo, Yeongsik Seo, Andrew Fu, Mei Hao, Iris Xia, Jiwon Jenny Lee, Xinyi Mary Xie, Hyoungseob Park, Aaron Dollar, Alex Wong

Published 2026-08-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shania Guo, Yeongsik Seo, Andrew Fu, Mei Hao, Iris Xia, Jiwon Jenny Lee, Xinyi Mary Xie, Hyoungseob Park, Aaron Dollar, Alex Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to build a perfect three-dimensional map of the world using only a camera and a distance sensor. For most objects, like a chair or a table, this works well. The camera sees the shape, and the sensor measures how far away it is. But there is a class of objects that breaks these tools: things that are incredibly thin, like a single wire, a delicate branch, or the spoke of a bicycle wheel. To a camera, these objects occupy so few pixels that they often disappear into the background. To a distance sensor, they are so narrow that the beam might slip right past them, or the reflection might be too weak to register. As a result, even the most advanced computer programs designed to reconstruct 3D scenes often fail completely when faced with these slender structures, leaving gaps or blurring them into nothingness. This is a significant hurdle for robotics and virtual reality, where understanding the fine details of the physical world is essential for a machine to interact with it safely and effectively.

To solve this puzzle, a team of researchers at Yale University set out to understand exactly how and why these systems fail, and whether adding a sense of touch could help. They created a new collection of data called VisTa3D, which is essentially a library of real-world scenes featuring thin objects, captured with multiple types of sensors working in perfect sync. The team gathered 387 different scenes, featuring 70 distinct thin objects such as wires, cables, and wooden figurines, placed in 17 different environments ranging from office hallways to outdoor staircases. For every single moment they recorded, they captured a standard color photo, a depth map showing distance, and a unique tactile response map. This last piece is the key innovation: they used a specialized sensor that acts like a digital skin, pressing against the object to record exactly how the surface deforms under pressure. This gives the computer a direct, local measurement of the object's shape, independent of how much space it takes up in a photograph.

The researchers first tested the limits of current technology by feeding this new data into 11 of the best existing 3D reconstruction models available. The results were stark. Even the most sophisticated systems, which can handle complex rooms and large furniture with ease, struggled profoundly with the thin objects. When the researchers focused their evaluation specifically on the thin parts of the scene, the accuracy of these models dropped dramatically. The computers simply could not figure out where the wires were or how thick they were, often treating them as if they did not exist at all. The study confirmed that the problem is not just a lack of data, but a fundamental limitation in how these models interpret visual information when an object is too small to be seen clearly from a distance.

To address this, the team introduced a new approach that combines sight, distance, and touch. They built a baseline model that takes the visual image, the sparse distance data, and the tactile pressure maps all at once. By feeding the computer the tactile information, they gave it a way to "feel" the object's shape even when the camera couldn't see it clearly. The results showed a clear improvement. The new model, which they named Tactile-DC, was able to reconstruct the thin structures with much higher fidelity than any of the vision-only systems. It successfully recovered details that the other methods missed, proving that the local information provided by touch can fill in the gaps left by sight. While the system is not yet perfect and still faces challenges with certain materials and lighting conditions, the experiment demonstrates a viable path forward. By adding the sense of touch to the visual toolkit, machines may finally be able to see the world in its full, delicate detail, from the thickest wall to the thinnest wire.

Technical Summary: VisTa3D

Problem Statement
State-of-the-art 3D reconstruction models, including monocular depth estimation (MDE), multi-view stereo (MVS), depth completion (MDC), and neural rendering methods, consistently underperform when reconstructing thin objects. The primary challenges stem from the limited spatial occupancy of thin structures in RGB images and 3D point clouds, leading to sparse sensor returns and correspondence ambiguities. Furthermore, small camera motions often cause occlusion or disocclusion of these structures, degrading reconstruction quality. While high-resolution sensors exist, they are often cost-prohibitive or computationally demanding. Existing datasets and benchmarks predominantly focus on general 3D scenes or 2D segmentation, lacking a dedicated resource to evaluate and improve reconstruction specifically for thin objects.

Methodology
The authors introduce VisTa3D, a multimodal dataset and benchmark designed to address these gaps.

Data Collection Pipeline:
- Sensors: The system utilizes an Intel RealSense D455 for synchronized RGB images, depth maps, and inertial measurements (IMU), and a DIGIT sensor for vision-based tactile response maps. High-fidelity ground truth is acquired using a 3DMakerPro Moose structured-light 3D scanner.
- Ground Truth Generation: To overcome the noise and incompleteness of raw depth sensors, the authors propose a processing pipeline:
  1. High-Fidelity Scanning: Each thin object is individually scanned to create a dense, high-fidelity point cloud ( $P_{scan}$ ).
  2. Registration: $P_{scan}$ is registered to the raw, noisy point cloud of the scene ( $P_{obj}$ ) derived from the RealSense data. This involves segmenting the object in RGB frames (using SAM2), backprojecting depth pixels, and performing a two-stage rigid registration (manual coarse alignment followed by Iterative Closest Point).
  3. Object Insertion: The noisy object points in the raw point cloud are removed and replaced with the high-fidelity $P_{scan}$ .
  4. Reprojection: The refined point cloud is reprojected onto the image plane for each camera pose to generate ground-truth depth maps and segmentation masks. Special handling (z-buffer for object regions, k-median buffer for background) ensures thin structures are preserved without background overwriting.
- Tactile Data: For each object, eight contact locations (edges, corners, textures) are manually selected. The DIGIT sensor records tactile videos (approx. 200 frames) capturing surface deformation under varying force levels.
Dataset Composition:
- Real-World: 225 scenes covering 52 thin objects (wires, figurines, plants, household items) across 8 different environments (6 indoor and 2 outdoor).
- Synthetic: 162 scenes generated in NVIDIA Isaac Sim 5.0, featuring 18 thin objects on 9 tabletop configurations, providing 500 frames per scene with perfect ground truth.
- Total: The full dataset comprises 387 scenes covering 70 thin objects (52 real-world + 18 synthetic).
- Modalities: Synchronized RGB, depth, tactile response maps, IMU data, camera poses/calibration, and ground-truth depth/segmentation.
Baseline Model (Tactile-DC):
To test the hypothesis that tactile data aids reconstruction, the authors propose Tactile-DC, the first visual-range-tactile 3D reconstruction baseline.
- Architecture: Inspired by depth completion networks, it employs shallow encoders for RGB and sparse depth. A dedicated tactile encoder processes each tactile response map.
- Fusion: 3D contact locations are projected onto the image plane using camera intrinsics and poses. Tactile features are compressed into latent vectors and injected into the fused RGB-depth feature map at the corresponding projected locations to modulate the features.
- Decoding: A backbone network followed by a Spatial Propagation Network (SPN) decodes the modulated features into a dense depth map.

Key Contributions

VisTa3D Dataset: The first multimodal dataset for thin object reconstruction, comprising synchronized RGB, depth, and tactile data with associated inertial measurements, camera poses, and high-fidelity ground-truth depth and segmentation masks derived from laser scanning.
Data Curation Pipeline: A robust methodology for generating ground truth by fusing high-fidelity 3D scans with noisy sensor data, including object insertion and outlier filtering.
Benchmarking: An extensive evaluation of 11 existing state-of-the-art methods (spanning MDE, MVS, MDC, and NVS) across 6 evaluation modes.
Baseline Model: The introduction of Tactile-DC, a novel architecture integrating visual, range, and tactile modalities for thin object reconstruction.

Results

Benchmark Performance: Existing methods (MDE, MVS, MDC, NVS) consistently struggle with thin objects. In "thin-object focused" evaluations (masked metrics), performance drops significantly compared to general evaluations. For instance, MDE models like UniDepth and DepthAnything show near-zero A1-A3 scores in object-focused settings under default protocols, indicating severe failure in recovering thin structures.
Tactile-DC Performance: Tactile-DC achieves the best performance among all tested methods.
- General Evaluation: A1/A2/A3 scores of 0.844/0.900/0.920.
- Object Evaluation: A1/A2/A3 scores of 0.719/0.811/0.852.
- It consistently outperforms OmniDC (the strongest non-tactile baseline) across all protocols and evaluation modes.
Qualitative Analysis: Visual comparisons show that while OmniDC produces globally accurate depth, it fails to resolve fine details and boundaries of thin objects. Tactile-DC significantly reduces these errors, particularly in regions where visual and sparse depth cues are weak or ambiguous.
NVS Results: Among Novel View Synthesis methods, Nerfacto performed best, while Gaussian Splatting methods showed limitations in rendering thin structures without consistent constraints.

Significance and Claims
The paper claims that VisTa3D is the first resource to systematically benchmark the limitations of current 3D reconstruction models on thin objects. The results demonstrate that even advanced foundation models and depth completion techniques exhibit low fidelity on thin structures due to insufficient representation in training data and sensor limitations.

The authors posit that tactile sensing offers a complementary modality that provides local shape and deformation information largely independent of object thickness. The success of the Tactile-DC baseline validates the hypothesis that fusing tactile data with visual and range inputs can effectively aid in the reconstruction of thin objects, offering a pathway to overcome the correspondence and sampling issues inherent to purely visual or range-based approaches. The dataset and benchmark are intended to drive future research in multimodal 3D reconstruction, specifically targeting the recovery of fine geometric details.

More like this