Image-Based Structural Analysis Using Computer Vision and LLMs: PhotoBeamSolver

1. Problem Statement

The paper addresses the challenge of automating structural analysis for idealized beam models (common in textbooks and academic exercises) using only a single photograph or hand-drawn sketch.

• Current Limitations: Existing engineering software (e.g., SAP2000, ETABS) requires manual data entry and is often expensive or complex. Free online calculators exist but often lack versatility, require programming knowledge, or have significant usage restrictions.
• The Gap: There is a need for a tool that can bridge the gap between visual input (a photo of a beam diagram) and analytical output (reaction forces, shear, moment, and deflection) without requiring the user to manually transcribe geometric and loading data.
• Core Challenge: Translating unstructured visual data (hand-drawn diagrams with varying styles) into a structured mechanical representation suitable for symbolic computation, while managing uncertainty and avoiding hallucinations common in AI models.

2. Methodology

The PhotoBeamSolver system is a hybrid image-to-analysis pipeline that integrates Computer Vision (CV) and Large Language Models (LLMs). The architecture consists of two primary stages:

A. Perception Stage (Computer Vision)

• Model Architecture: Utilizes a custom-trained YOLO (You Only Look Once) based Convolutional Neural Network (CNN). YOLO was selected for its computational efficiency and ability to perform single-stage inference for multi-object detection.
• Training Data: The model was trained on a dataset of 532 annotated beam diagrams. The annotations include bounding boxes and categorical labels for:
  • Supports: Pinned (simple), Roller, and Fixed (clamped).
  • Loads: Point loads (concentrated) and Distributed loads.
  • Geometric References: Span markers and dimension lines.
• Detection Process: The model performs object detection to localize structural elements. It outputs bounding box coordinates, class probabilities, and confidence scores. Non-maximum suppression (NMS) is used to remove redundant detections.
• Geometric Post-Processing: Raw detection outputs are converted into a structured mechanical representation. This involves:
  • Projecting support and load positions onto the beam axis.
  • Normalizing load locations relative to the beam span.
  • Associating load magnitudes with their corresponding text annotations based on positional proximity.
  • Generating a symbolic description of boundary conditions, load types, and geometric parameters.

B. Reasoning Stage (Analytical Solver)

The system offers two modes for the analytical solver, depending on configuration:

1. Deterministic Pipeline: Uses the structured output from the CV stage to feed a custom solver (based on the IndeterminateBeam Python package) to calculate equilibrium equations, reaction forces, and internal forces.
2. LLM-Based Reasoning: When available, the structured textual representation is passed to a Large Language Model (via GPT API). The LLM acts as a symbolic interpreter rather than a numerical regressor. It generates analytical formulations consistent with classical structural mechanics.
   • Constraint: The LLM is strictly constrained to structured inputs to mitigate "hallucination" risks and ensure analytical outputs are grounded in detected parameters.

3. Key Contributions

• Hybrid Architecture: The paper proposes a novel pipeline that decouples visual perception from structural reasoning. By using CV for detection and LLMs (or deterministic solvers) for symbolic derivation, it separates the ambiguity of image interpretation from the rigor of engineering calculation.
• End-to-End Automation: It demonstrates the feasibility of taking a raw image of a beam diagram and outputting complete structural analysis results (Shear Force, Bending Moment, and Deflection diagrams) without manual data entry.
• Uncertainty Analysis: The paper critically discusses epistemic uncertainty in deep learning. It highlights that standard softmax confidence scores are heuristic and not rigorous measures of certainty, advocating for the separation of visual detection and analytical solving to limit error propagation.
• Open-Source Implementation: The authors provide the source code and training scripts via a GitHub repository, contributing to the accessibility of AI-driven civil engineering tools.

4. Results

• Training Performance: The YOLO-based detector achieved a Mean Average Precision (mAP) > 0.93 on the validation set after 10,000 epochs. Per-class precision remained consistently high across all support types and load categories, indicating balanced learning with no significant class bias.
• Analytical Accuracy: When compared to reference solutions generated by the IndeterminateBeam package, the system's computed reactions and internal force diagrams showed discrepancies within machine precision for statically determinate cases.
• Visual Output: The system successfully reconstructed classical structural response curves (Shear, Moment, Deflection) from input images (see Figures 6 and 7 in the paper).
• Failure Modes: The system struggles with ambiguous handwriting, overlapping annotations, or incomplete geometric references. In these cases, detection confidence drops, and manual correction may be required.

5. Significance and Limitations

Significance:

• Accessibility: Provides a low-barrier tool for students and professionals to quickly verify structural calculations from sketches.
• Educational Utility: Bridges the gap between theoretical diagramming and computational analysis, serving as a powerful aid in academic settings.
• Future Direction: Establishes a framework for integrating AI into civil engineering decision-support systems, moving beyond simple calculation aids to intelligent interpretation tools.

Limitations & Future Work:

• Scope Restriction: The current system is limited to planar, idealized beam configurations. It does not yet handle inclined loads (which must be decomposed manually), truss systems, or inclined beams (treated as horizontal equivalents).
• Data Scarcity: Unlike generic object detection, high-quality structural datasets require expert annotation and engineering metadata, making data acquisition a bottleneck for scaling.
• Parameter Inference: The system cannot reliably infer material properties (e.g., Young's Modulus), cross-sectional characteristics, or design loads from a single image. These parameters must be provided or assumed, limiting the system's application to idealized academic problems rather than complex real-world infrastructure inspection.
• Uncertainty Quantification: While the paper discusses epistemic uncertainty, a robust probabilistic treatment of uncertainty for real-world structural safety is identified as a necessary future step.

In conclusion, PhotoBeamSolver successfully validates the concept of an image-to-analysis pipeline for structural engineering, demonstrating that combining computer vision for perception with LLMs or symbolic solvers for reasoning can automate the analysis of idealized beam models with high accuracy.