A graph generation pipeline for critical infrastructures based on heuristics, images and depth data

1. Problem Statement

Critical infrastructures (e.g., water and energy plants) require digital twins and simulations to ensure resilience and continuity. Creating these virtual representations typically relies on expensive 3D laser scanners and specialized point cloud processing, which are costly and require expert knowledge. Furthermore, existing deep learning approaches for "scene graph generation" (predicting object relations) often function as "black boxes," lacking the explainability and interpretability required for high-stakes decision-making in critical infrastructure. There is a need for a cost-effective, accessible, and transparent method to automatically generate relational graphs of infrastructure objects from standard imaging data.

2. Methodology

The authors propose a prototypical pipeline that combines photogrammetry (RGB images and depth data from stereo cameras) with deep learning and user-defined heuristics. The system is designed to be modular and explainable.

A. Data Acquisition

• Source: Due to security restrictions on real critical infrastructure, the study uses realistic synthetic environments created in Unreal Engine 5 (simulating hydraulic systems).
• Input: RGB images and depth maps captured from multiple viewpoints using a simulated stereo camera setup.
• Objects: The test environments contain pipes, pumps, tanks, valves, and sprinklers.

B. The Pipeline Architecture

The pipeline processes "Non-pipe objects" and "Pipe objects" through distinct branches before merging them into a final graph.

1. Non-Pipe Objects (Pumps, Tanks, Valves)

• Detection: Uses a fine-tuned YOLOv8 pose model to detect objects and predict specific keypoints representing connection points (e.g., where a pipe attaches to a pump).
• Location Approximation: Instead of time-consuming instance segmentation, the system approximates object locations within bounding boxes using:
  • Depth filtering (removing background pixels deeper than the median).
  • Point cloud projection and statistical outlier removal.
  • DBSCAN clustering to identify the main object cluster.
• Matching: Identical objects across different images are merged based on voxel distance thresholds.

2. Pipe Objects

• Detection: Uses a fine-tuned YOLOv8 instance segmentation model to generate masks for pipe elements.
• Cleanup: Masks are cleaned by removing background pixels via depth projection and DBSCAN. Overlapping masks belonging to non-pipe objects (e.g., pipes inside a pump housing) are removed.
• Matching: Masks from different images are matched by projecting them into 3D space and calculating intersection overlaps.
• Endpoint Approximation: Since the segmentation model does not predict keypoints, endpoints are estimated:
  • Straight pipes: Endpoints are derived from the mean of voxels at the ends of a rotated bounding box.
  • Bent/Ambiguous pipes: Endpoints are approximated by pulling the object's centroid toward the nearest neighboring object.

3. Graph Generation & Rule Enforcement

• Initial Graph: Created by connecting object endpoints iteratively based on the shortest pairwise distances, up to a maximum threshold (graph_max_distance).
• Heuristic Refinement: The initial graph is refined by enforcing a set of user-defined rules (functions) to remove invalid edges. Examples include:
  • Rule 1: Pumps/Valves max 2 connections; Tanks max 1.
  • Rule 2: Pipes max 3 connections.
  • Rule 3: No connections between neighbors of the same non-pipe object.
  • Rule 4 & 5: No single nodes or cycles allowed.
  • Rule 6 & 7: Classification of pipes as "Reducer/Expander" or "PipeCrossing" based on topology.

3. Key Contributions

• Cost-Effective Alternative: Demonstrates that stereo cameras and photogrammetry can replace expensive laser scanners for generating infrastructure graphs.
• Explainability: Unlike end-to-end deep learning graph generators, this pipeline uses explicit, user-defined rules. This transparency allows operators to understand why a connection was made or rejected, a critical requirement for safety-critical systems.
• Modular Design: The pipeline separates detection, matching, and rule enforcement. Detection models (e.g., YOLO) can be swapped without altering the logic of the graph generation, and rules can be tailored to specific infrastructure types.
• Hybrid Approach: Successfully combines deep learning (for detection) with classical computer vision and heuristics (for relation inference) to balance accuracy with interpretability.

4. Results

The pipeline was tested on two synthetic hydraulic systems:

• System 1 (Simple): 1 pump, 1 tank, 1 valve, 1 sprinkler, 3 T-fittings.
  • Performance: The predicted graph was nearly identical to the ground truth.
  • Metric: Graph Edit Distance (GED) was 4.
  • Limitation: The sprinkler was missed due to being too small for the detection model.
• System 2 (Complex): 2 pumps, 2 tanks, 3 valves, 4 sprinklers, X-shaped layout.
  • Performance: Captured the main structure but missed the sprinkler system and had minor pipe matching errors.
  • Metric: GED was 22 (including sprinkler errors). When excluding the undetected sprinklers, the GED dropped to 10.
• Runtime: The total pipeline runtime averaged ~785 seconds (approx. 13 minutes) for 29 image pairs on a standard CPU, with the pipe matching module being the most computationally intensive.

5. Significance and Future Work

• Significance: This work provides a viable pathway for creating digital twins of critical infrastructure without prohibitive costs. The emphasis on rule-based explainability addresses a major barrier to adopting AI in high-stakes industrial decision-making.
• Limitations:
  • Reliance on the quality of the object detection model (small objects like sprinklers were missed).
  • Heuristics for endpoint approximation struggle with ambiguous pipe shapes.
  • Currently validated only on synthetic data; real-world validation is pending.
• Future Directions:
  • Improving detection models for small objects (e.g., using Vision Transformers).
  • Replacing crude heuristics with skeletonization techniques for better pipe endpoint estimation.
  • Validating the pipeline on real-world critical infrastructure sites.

In conclusion, the paper presents a robust, transparent, and cost-effective framework for converting 2D/3D image data into relational graphs, offering a practical solution for the digitalization of critical infrastructure assets.