Technical Summary: Cracks in the Foundation (CiF)

Problem Statement

Automated structural health monitoring is critical for preventing catastrophic infrastructure failures, yet progress in this domain is severely hindered by two primary factors: extreme data scarcity and algorithmic limitations specific to civil engineering contexts.

1. Data Scarcity: High-quality, pixel-level defect segmentation requires expert annotation to distinguish microscopic deterioration from harmless shadows, making crowdsourcing impractical. Consequently, existing datasets are small, low-resolution, or restricted to classification and semantic segmentation rather than instance segmentation.
2. Algorithmic Hurdles: Civil infrastructure materials (e.g., concrete) are often monochromatic and flat, forcing models to rely on shape rather than texture—a known weakness of deep learning. Furthermore, defects like cracks are thin, highly non-convex structures that conflict with the "center-bias" inherent in many modern architectures.
3. The Zero-Shot Illusion: Despite the advent of promptable Foundation Models (FMs) and Vision-Language Models (VLMs) trained on massive internet-scraped datasets (e.g., COCO, LAION), there is an unproven assumption that dense image understanding is solved. The paper posits that these models fail to generalize to the specific scale and feature variations required for infrastructure inspection.

Methodology and Dataset Construction

The authors introduce Cracks in the Foundation (CiF), a dataset curated over five years through collaboration between AI researchers and civil engineering experts (including partners like Sund & Bælt and Trafikverket).

Data Acquisition

The dataset comprises approximately 150,000 high-resolution images (native resolution mean: 16.66 MPx) captured from European civil infrastructure, primarily bridges. The acquisition strategy employs a hybrid approach:

• In-the-Wild Data (26%): Collected during routine inspections using consumer-grade and industrial cameras, capturing high variability in lighting, weather, and viewpoint.
• Controlled Inspection Setups: Drone-assisted acquisitions using two modes:
  • Scan Setup: Drones fly parallel to surfaces for large-scale coverage with GPS/RTK registration.
  • High-Resolution Setup: Drones hover stationary to capture structured grids of zoomed-in images, enabling mosaic reconstruction of localized areas.

Annotation and Classes

• Instance-Level Annotation: All defects are annotated as polygon masks at native resolution, preserving fine-grained details (e.g., hairline cracks a few pixels wide) alongside large defects.
• Classes: Six categories were defined with domain experts:
  1. Crack: Linear discontinuities.
  2. Net-Crack: Interconnected polygonal patterns (alligator/map cracking).
  3. Crack with Precipitation: Cracks with mineral deposits (efflorescence).
  4. Spalling: Detachment of surface material.
  5. Rust: Oxidation of steel reinforcement.
  6. Algae: Biological surface growth.
• Quality Control: A rigorous pipeline involving pilot phases, batch processing, and a two-pass QA system (vendor internal + in-house spot-checking) ensures label accuracy. Ambiguous samples are triaged with experts to refine guidelines.

Dataset Variants

• Full Variant: 12,896 native high-resolution images (16.66 MPx mean).
• Tiled Variant: 148,642 patches (1024×1024) derived from the Full variant, suitable for standard vision backbones.

Evaluation and Results

The authors evaluated both supervised baselines and zero-shot Foundation Models on the CiF dataset.

Supervised Baselines

State-of-the-art models (YOLOv11/v26, Mask DINO, RF-DETR, EoMT) were fully fine-tuned on the training split.

• Object Detection: Models achieved moderate success, with bounding box mAP@50:95 reaching approximately 35%. Performance was strong for high-contrast defects like Algae (~59% mAP) but dropped significantly for complex structural defects like Net-Cracks and Cracks with Precipitation (20–30% mAP).
• Instance Segmentation: Performance degraded drastically. Mask mAP for fine-grained defects (e.g., cracks) fell to single digits. The best-performing model (EoMT with a DINOv3 backbone) plateaued at roughly 25% mAP for segmentation tasks.

Foundation Model Evaluation (Zero-Shot)

The paper tested promptable models (SAM3) and Vision-Language Models (Qwen3-VL) without fine-tuning.

• Zero-Shot Failure: Qwen3-VL (up to 235B parameters) achieved a mean bounding box mAP of only 2.4%, failing to identify structural anomalies.
• Promptable Limitations: Even SAM3, when provided with "oracle" ground-truth bounding boxes or point prompts, struggled with fine-grained defects. For the "Crack" class, SAM3 with 5 oracle points achieved only 33.9% mAP, and with text prompts, performance was negligible.
• Conclusion: Current FMs lack the specific priors required for civil infrastructure defect segmentation.

Key Contributions

1. CiF Dataset: The largest and most detailed civil infrastructure instance segmentation dataset to date, featuring ~150k high-resolution images and ~250k fine-grained masks.
2. Benchmarking: A rigorous analysis exposing the limitations of current visual AI. The dataset serves as a benchmark where even specialized models with domain supervision plateau at ~25% mAP, and zero-shot FMs fail almost entirely.
3. Identification of a Blind Spot: The paper demonstrates that dense image understanding in the built environment is an unsolved problem, challenging the paradigm that internet-scale pretraining is sufficient for specialized, safety-critical domains.

Significance and Claims

The paper claims that CiF establishes the inspection of civil infrastructure as an open, fundamental challenge for present-day visual AI. The authors argue that the current "foundation model paradigm," trained predominantly on internet images, contains "cracks" when applied to real-world infrastructure.

The significance lies not in the immediate deployment of a perfect solution, but in the exposure of a critical gap:

• Automated structural health monitoring is not yet solved by current SOTA models.
• The domain requires a shift from relying solely on massive-scale pretraining to developing architectures capable of handling shape-based reasoning, extreme scale variations, and low-texture environments.
• By open-sourcing this dataset, the authors aim to foster the development of visual AI capable of genuinely safeguarding the physical environment, moving beyond the "illusion" of zero-shot generalization.

The paper remains modest regarding future solutions, noting that while scaling datasets further might help, the root causes of these failures (architectural vs. data-driven) remain an open question, and the current economic cost of acquiring such massive, expert-annotated datasets is prohibitive.