Field imaging framework for morphological characterization of aggregates with computer vision: Algorithms and applications

This dissertation presents a comprehensive field imaging framework that leverages advanced computer vision algorithms, including 2D instance segmentation and an integrated 3D reconstruction-segmentation-completion approach, to overcome the limitations of traditional methods and enable accurate morphological characterization of construction aggregates across diverse field scenarios.

The Gist

Imagine you are a construction engineer. Your job is to build roads, bridges, and dams. To do this, you need aggregates—basically, rocks, gravel, and sand. These aren't just any rocks; they are the "skeleton" of our infrastructure.

For decades, checking if these rocks are the right size and shape has been a nightmare.

• The Old Way: Engineers would stand in a dusty quarry, squinting at a pile of rocks the size of small cars. They'd guess the size, maybe use a giant ruler, or weigh them one by one with heavy machinery. It was slow, subjective (based on who was looking), and often inaccurate.
• The Problem: You can't put a giant rock on a kitchen scale, and you can't easily measure a rock buried under a pile of other rocks.

The Solution: This dissertation introduces a "Digital Eye" for construction. It's a smart system that uses cameras, smartphones, and artificial intelligence (AI) to take pictures of rock piles and instantly tell you exactly what's inside them—down to the volume of every single rock.

Here is how the system works, broken down into three simple levels, like a video game getting harder:

Level 1: The Solo Rock (The "Individual" Mode)

Imagine you have one giant rock sitting alone.

• The Trick: You take three photos of it from different angles (top, front, side) using a smartphone. You also snap a photo of a known object, like a basketball, to act as a ruler.
• The Magic: The computer looks at the photos, cuts the rock out of the background (even if the sun is glaring or there are shadows), and "stitches" the three photos together to build a 3D model.
• The Result: It calculates the rock's volume and weight with amazing accuracy. It's like turning a flat photo into a 3D hologram that you can measure.
• Why it matters: It's much faster and more accurate than an engineer trying to guess the size with a tape measure.

Level 2: The Rock Pile (The "2D Crowd" Mode)

Now, imagine a massive pile of thousands of rocks. You can't measure them one by one.

• The Trick: You take a photo of the whole pile.
• The Magic: This is where the AI shines. Think of the AI as a super-organized librarian. It looks at the messy photo and says, "Okay, that's Rock #1, that's Rock #2, and that's Rock #3." It separates them automatically, even if they are touching or overlapping.
• The Result: It counts them all, measures their shapes, and tells you the "grade" of the pile (e.g., "This pile has 60% big rocks and 40% medium rocks").
• Why it matters: It turns a chaotic pile into a neat spreadsheet in seconds.

Level 3: The Invisible Rock (The "3D Detective" Mode)

This is the hardest part. In a real pile, rocks are stacked on top of each other. If you take a photo, you only see the top half of the rocks. The bottom half is hidden. How do you know the full size?

• The Trick: The system uses a "Virtual Reality" lab.
  1. The Library: First, the researchers built a digital library of perfect 3D models of real rocks.
  2. The Simulator: They used a computer game engine to drop these digital rocks into a virtual pile, just like gravity works in real life. Because it's a simulation, the computer knows the exact shape of every rock, even the parts that are hidden.
  3. The Training: They taught the AI to look at the "fake" pile and guess the hidden parts. It's like showing a student a picture of a half-eaten apple and asking them to draw the whole apple.
  4. The Real World: Finally, they took this trained AI to a real quarry. It scanned a real pile, guessed the hidden parts of the rocks, and reconstructed the full 3D shape.
• The Result: It can estimate the weight and volume of rocks that are partially buried, which was previously impossible.

The "Secret Sauce": Shape Percentage

The researchers realized that sometimes the AI guesses wrong because it can't see enough of the rock. So, they invented a "Confidence Score" called Shape Percentage (SP).

• If the AI can only see 50% of a rock, it says, "I'm not sure about this one."
• If it can see 80%, it says, "I'm pretty confident."
• By filtering out the low-confidence guesses, the final results become incredibly accurate.

Why Should You Care?

This isn't just about rocks; it's about building a better world.

• Safety: If a bridge is built with rocks that are too small or the wrong shape, it could collapse. This system ensures the rocks are perfect.
• Money: Construction companies waste millions of dollars guessing wrong. This system saves time, fuel, and money.
• Speed: What used to take a team of engineers days to measure can now be done in minutes with a smartphone and some software.

In a nutshell: This research took the messy, dusty, hard-to-measure world of construction rocks and gave it a "digital twin." It uses cameras to see, AI to think, and math to measure, turning a pile of chaos into a precise, data-driven blueprint for building our future.

Technical Summary

Here is a detailed technical summary of the dissertation "Field Imaging Framework for Morphological Characterization of Aggregates with Computer Vision: Algorithms and Applications" by Haohang Huang.

1. Problem Statement

Construction aggregates (sand, gravel, crushed stone, and riprap) are fundamental to civil infrastructure. While small-to-medium aggregates are routinely characterized using sieving and calipers in laboratories, large-sized aggregates (e.g., riprap used for erosion control, weighing up to 1,150 lbs) present significant challenges for Quality Assurance/Quality Control (QA/QC).

• Current Limitations: State-of-the-practice methods rely on subjective visual inspection and labor-intensive manual measurements (weighing or estimating dimensions), which are time-consuming, unsafe, and lack morphological detail (shape, volume, 3D structure).
• State-of-the-Art Gaps: Existing computer vision systems are mostly confined to controlled laboratory environments, require manual separation of particles, and are limited to small aggregates. They struggle with densely stacked stockpiles, overlapping particles, and field lighting conditions (shadows, varying sun angles).
• Core Challenge: There is a lack of a robust, automated, and scalable framework capable of characterizing the 2D and 3D morphological properties (size, shape, volume/weight) of large-sized aggregates directly in the field, particularly within stockpiles.

2. Methodology

The dissertation proposes a multi-scenario Field Imaging Framework that addresses aggregates at three levels of complexity: individual particles, 2D stockpile images, and 3D stockpile point clouds.

A. Individual Aggregate Volumetric Reconstruction (2D to 3D)

• System Design: A portable field imaging system using three smartphones arranged in orthogonal views (top, front, side) with a blue background and a calibration ball.
• Algorithms:
  • Color-Based Segmentation: Utilizes the CIE Lab* color space to separate rock particles from backgrounds under uncontrolled lighting. It employs adaptive thresholding and morphological operations to handle shadows and reflections.
  • Volumetric Reconstruction: Uses orthogonal intersection of silhouettes from the three views.
  • Correction Mechanisms: Applies a systematic correction factor (empirically derived) and a resolution-based correction factor (accounting for pixel discretization) to mitigate volume overestimation inherent in silhouette-based reconstruction.

B. Automated 2D Instance Segmentation for Stockpiles

• Approach: Deep Learning-based instance segmentation for densely stacked aggregates.
• Dataset: A custom dataset of 164 stockpile images containing 11,795 manually labeled aggregate particles from various Illinois quarries.
• Model: Implements Mask R-CNN (Region-based Convolutional Neural Network).
  • R-CNN: Detects object regions.
  • FCN (Fully Convolutional Network): Performs pixel-level semantic segmentation to extract particle boundaries.
• Morphological Analysis: Calculates Equivalent Spherical Diameter (ESD) and Flat and Elongated Ratios (FER) from segmented 2D silhouettes, converting them to 3D volume estimates using assumed 3D FER values.

C. 3D Reconstruction, Segmentation, and Completion (RSC-3D)

This is the core advanced component, designed to handle partial visibility in stockpiles.

• 3D Particle Library:
  • Developed a marker-based photogrammetry approach using Structure-from-Motion (SfM) to create high-fidelity 3D mesh models of individual aggregates.
  • Features include background suppression (using U-Net for saliency detection), object markers for robust point cloud stitching, and scale markers for physical dimensioning.
  • Resulted in a library of 82 high-fidelity 3D models (RR3 and RR4 categories).
• Synthetic Data Generation:
  • Since manual 3D labeling of stockpiles is impossible, a synthetic data pipeline was built using the Unity game engine.
  • Process: Physics-based gravity simulation drops aggregate instances into stockpiles. Multi-view cameras and LiDAR sensors simulate raycasting to generate 3D point clouds with ground-truth instance labels.
  • Dataset: 300 synthetic stockpile scenes with 105,054 aggregate instances.
• 3D Instance Segmentation:
  • Implements PointGroup, a detection-free deep learning network.
  • Uses Submanifold Sparse Convolution (SSC) and learns per-point offset vectors to shift points toward their instance centroids, enabling robust clustering in dense point clouds.
• 3D Shape Completion:
  • Addresses the "missing side" problem where only the top surface of stockpiles is visible.
  • Implements SnowflakeNet, which models shape completion as a multi-stage "snowflake-like" growth process (Encoder-Decoder with Point Splitting).
  • Training Data: Generated 9,184 partial-complete shape pairs using varying-visibility and varying-view raycasting schemes on the 3D particle library.

D. Integrated Framework & Field Validation

• Workflow: Multi-view images $\rightarrow$ SfM 3D Reconstruction $\rightarrow$ 3D Instance Segmentation $\rightarrow$ 3D Shape Completion $\rightarrow$ Morphological Analysis.
• Validation: Tested on 12 re-engineered stockpiles (using known library particles) and 9 field stockpiles (unknown particles).
• Refinement: Introduced a Shape Percentage (SP) metric to quantify the visibility of a segmented particle. An SP threshold (e.g., 75%) is applied to filter out low-confidence predictions, significantly improving accuracy.

3. Key Contributions

1. First Comprehensive Field Imaging Framework for Large Aggregates: Bridges the gap between laboratory imaging and challenging field conditions, specifically targeting large-sized riprap and stockpiles.
2. Robust 2D/3D Segmentation for Dense Stockpiles: Successfully adapted deep learning (Mask R-CNN and PointGroup) to segment densely packed, overlapping aggregates where traditional methods fail.
3. Synthetic Data Pipeline for 3D Learning: Developed a novel pipeline to generate massive, labeled 3D point cloud datasets with ground truth, overcoming the bottleneck of manual 3D labeling.
4. 3D Shape Completion for Partial Observations: Implemented a learning-based approach to predict the unseen geometry of aggregates in stockpiles, enabling full 3D morphological characterization from surface-only data.
5. Quantitative Error Correction: Identified and quantified systematic volume underestimation (due to occlusion and conservative segmentation) and proposed the Shape Percentage (SP) thresholding method to improve reliability and enable uniform correction factors.

4. Key Results

• Individual Aggregates:
  • Volumetric reconstruction achieved a Mean Absolute Percentage Error (MAPE) of 3.6% (Source 1) and 7.9% (Source 2) against ground truth.
  • Outperformed manual field measurements significantly (Manual MAPE: 68.3% vs. Imaging MAPE: 8.2%).
• 2D Stockpile Segmentation:
  • Achieved 88.0% completeness and 86.7% precision (IoU) on validation images, demonstrating robustness in separating overlapping particles.
• 3D Stockpile Analysis:
  • Segmentation: 78.4% completeness and 82.2% IoU precision on synthetic test sets.
  • Shape Completion: Achieved low Chamfer Distances (0.00483 mm on validation, 0.00559 mm on test sets).
  • Field Validation:
    • Size dimensions (length, width, height) showed MAPE errors between 8% and 18%.
    • Volume/Weight predictions initially showed systematic underestimation (~30-35%).
    • After SP Thresholding (75%): Volume/Weight MAPE improved to 15–20%, and the discrepancy between re-engineered and field stockpiles was minimized, allowing for a unified correction approach.

5. Significance

• Engineering Impact: Provides a non-destructive, rapid, and objective tool for QA/QC of large aggregates, replacing subjective visual inspections and dangerous manual weighing.
• Cost & Safety: Reduces labor costs, minimizes heavy machinery usage for individual rock handling, and improves worker safety.
• Scientific Advancement: Advances the state-of-the-art in computer vision by solving the "dense packing" and "partial observation" problems in 3D point cloud analysis, with applications extending beyond aggregates to other granular materials (e.g., railway ballast).
• Standardization Potential: The ability to accurately measure 3D dimensions and volumes could lead to updated industry standards that rely on dimensional metrics rather than just weight, improving material specification and design.