💻 computer science

Visual Localization via Semantic Structures in Autonomous Photovoltaic Power Plant Inspection

This paper presents a novel visual localization pipeline for autonomous UAV inspection of photovoltaic power plants that integrates module detection, semantic structure association with a plant model, and object tracking to enable precise navigation and positioning.

Original authors: Viktor Kozák, Karel Košnar, Jan Chudoba, Miroslav Kulich, Libor Přeučil

Published 2026-01-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Viktor Kozák, Karel Košnar, Jan Chudoba, Miroslav Kulich, Libor Přeučil

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a solar farm as a giant, perfectly organized library, where every book is a solar panel. The job of a drone is to fly through the aisles, get close enough to read the "spine" of every single book, and check if any of them are damaged (like having a hot spot or a crack).

The problem is that solar farms are huge, and the "books" (panels) all look exactly the same. If you just tell a drone, "Fly to coordinates X, Y, and Z," it might miss the mark by a few feet. In a library, that's fine. But for a drone taking a high-resolution photo to find a tiny defect, being off by a few feet means the photo is blurry or misses the book entirely.

This paper presents a new way to guide the drone, acting like a super-precise GPS that uses the library shelves themselves as landmarks.

Here is how the system works, broken down into simple steps:

1. The "Eyes" (Seeing the Panels)

First, the drone needs to know exactly what it is looking at. The researchers taught the drone's computer vision system to recognize individual solar panels. They tried three different "eyes" to do this:

The "Smart Learner" (YOLO): A deep learning model that looks at an image and draws boxes around every panel it sees, kind of like a student who has studied thousands of photos of solar farms.
The "Pixel Painter" (U-Net): Another AI model that paints a mask over the panels, separating them from the background.
The "Line Tracer" (Edge Detection): A more traditional method that looks for the straight lines and corners of the panels, connecting them like a dot-to-dot puzzle to figure out where the panels are.

2. The "Map" (The Library Blueprint)

The drone doesn't just fly blindly; it has a digital blueprint of the solar farm. This blueprint knows exactly where every single panel is supposed to be, how big it is, and how it's oriented. Think of this as the library's master catalog.

3. The "Matchmaker" (Connecting the Dots)

This is the magic part. As the drone flies, its "eyes" spot a row of panels. It then asks the "Matchmaker" (the localization pipeline): "I see a row of panels here. Does this match the blueprint?"

Finding the Start: To get started, the drone looks for specific "anchor points," like the very beginning or end of a row, or a gap between sections. It's like finding the "Aisle A" sign in a library to know where you are.
Tracking: Once it knows where it is, it keeps track of the panels as it flies. If it sees a panel, it knows exactly which "book" in the blueprint it is looking at.

4. The "Steering Wheel" (Real-Time Correction)

Instead of just following a pre-set path, the drone constantly corrects its course.

If the drone's standard GPS says it's at point A, but its "eyes" see the panels are actually at point B, the drone trusts its eyes.
It calculates exactly how far off it is and steers itself to the perfect distance and angle to take a crystal-clear photo.
It uses a mathematical trick (called PnP) to solve the puzzle: "I see these specific points on the image; where must my camera be in 3D space to see them exactly like this?"

5. The "Safety Net" (Kalman Filter)

Sometimes, the camera might get a little blurry, or the sun might reflect off a panel, confusing the "Matchmaker." To prevent the drone from getting lost, the system uses a "Safety Net" (a Kalman filter). This is like a wise navigator who says, "The camera thinks we are here, but based on how fast we were moving a second ago, we are probably actually there. Let's smooth it out." This ensures the drone stays on a steady path even if the camera gets momentarily confused.

What Did They Prove?

The researchers tested this system on two real solar farms with different layouts. They found that:

The drone could fly autonomously and stay perfectly aligned with the panels.
It could take photos from the optimal distance and angle, which is crucial for finding defects.
It worked even if the digital blueprint wasn't 100% perfect or if the standard GPS was slightly off. The visual system corrected these errors.
The system was fast enough to run in real-time, meaning the drone didn't have to stop and think; it just flew and adjusted.

In short, this paper describes a drone that doesn't just follow a line; it reads the room. It looks at the solar panels, recognizes them, and uses them as a guide to fly itself to the perfect spot for inspection, ensuring no defect is missed due to a blurry photo or a bad angle.

Technical Summary: Visual Localization via Semantic Structures in Autonomous Photovoltaic Power Plant Inspection

1. Problem Statement

The maintenance of photovoltaic (PV) power plants increasingly relies on Unmanned Aerial Vehicles (UAVs) equipped with thermal cameras to detect defects such as hot spots and circuit anomalies. However, fully autonomous inspection remains challenging. Current systems often rely on Global Navigation Satellite Systems (GNSS) and Inertial Measurement Units (IMU) for navigation, which lack the precision required to capture thermal images from optimal distances and viewing angles. Furthermore, standard mission planning based on pre-defined waypoints is prone to errors due to GNSS drift and inaccuracies in the power plant model. To acquire detailed defectoscopy data, the UAV must maintain a precise 6-DoF (six degrees of freedom) position relative to individual PV modules, a task difficult to achieve without visual feedback.

2. Methodology

The paper proposes a novel localization pipeline that directly integrates the detection of PV modules into the UAV navigation loop. The system operates on RGB camera images, chosen for their richer feature space and higher resolution compared to thermal images, though the authors note the methods can be adapted for thermal data.

2.1. Localization Pipeline Overview

The pipeline consists of three primary stages:

Module Detection: Visual detection and segmentation of individual PV modules.
Data Association: Tracking detections across image sequences to preserve unique IDs and associating them with a georeferenced power plant model. This involves identifying "visual anchor points" (bench ends and gaps) to initialize the association and using object tracking to maintain global consistency.
Camera Pose Estimation: Solving the Perspective-n-Point (PnP) problem to estimate the camera's relative position based on the association between detected image points and their known real-world coordinates.

2.2. PV Module Segmentation Approaches

The authors evaluate three distinct segmentation methods to determine their efficacy within the localization pipeline:

YOLOv8 (Instance Segmentation): A deep learning model that outputs oriented bounding boxes and segmentation masks.
U-Net (Fully Convolutional Network): A deep learning model producing binary segmentation masks, which are post-processed (erosion/dilation) to extract individual bounding boxes.
Edge-Based Detection (Traditional CV): A method using Canny edge detection and Hough transforms to identify line intersections, forming a graph structure representing module corners. This approach includes a novel detection mechanism for "bench gaps" (transitions between rows) using histogram similarity, requiring no training data.

2.3. Pose Estimation and Filtering

PnP Solver: The system uses the EPnP algorithm to calculate the transformation between the world coordinate system (power plant model) and the camera frame.
Kalman Filtering: To ensure robustness against noise and occasional PnP errors, a simplified Kalman filter integrates PnP pose estimates, GNSS data (for velocity), and a UAV motion model. The filter weights the PnP measurements dynamically based on reprojection error ( $\epsilon_r$ ) and distance deviation thresholds.

2.4. Power Plant Modeling

The system requires a compact, georeferenced model containing the positions, orientations, and dimensions of PV modules. The authors tested two model generation approaches:

High-Altitude (H-Alt) Model: Derived from public aerial imagery and topographic maps.
Structure-from-Motion (SfM) Model: Generated from the inspection flight data itself using OpenSfM, serving as a high-precision ground truth for evaluation.

3. Key Contributions

Direct Module-to-Drone Localization: A method for estimating UAV pose relative to individual PV modules, enabling precise navigation and data acquisition from optimal viewing angles.
Comparative Segmentation Analysis: The proposal and evaluation of three fundamentally different segmentation methods (two CNN-based, one traditional CV) within a unified localization framework.
Visual Anchor Points: The introduction of visually recognizable navigation points (bench ends and gaps) to initialize the localization pipeline and associate detections with the power plant model without prior GNSS alignment.
Public Dataset and Models: The release of a custom annotated dataset for PV module segmentation and the trained segmentation models.

4. Experimental Results

The system was evaluated on a custom dataset comprising manual inspection flights over two distinct PV power plants (Power Plant A and B) with varying module layouts (vertical vs. horizontal rows).

Detection Performance:
- U-Net demonstrated the highest robustness and success rates across both power plants and for bench gap detection.
- YOLO performed well on Power Plant A but showed reduced generalization on Power Plant B (horizontal modules), attributed to underrepresentation in training data.
- Edge-based methods were effective but suffered performance drops on darker modules in Power Plant A due to fixed parameter tuning.
Localization Accuracy:
- Using the SfM model (ground truth), the system achieved mean position errors ranging from 0.65m to 1.29m depending on the segmentation method and dataset.
- Using the H-Alt model, the system successfully compensated for model inaccuracies and GNSS drift, maintaining stable trajectories relative to the detected structures.
- The Kalman filter significantly improved trajectory smoothness and stability compared to raw PnP estimates.
Real-Time Capability:
- Computation times were measured on an Intel Core i7-7700 CPU and Nvidia GeForce 2080Ti GPU.
- The Edge-based method was the fastest (Total ~~58ms on GPU), followed by U-Net (~~38ms on GPU) and YOLO (~45ms on GPU). All methods supported real-time processing frequencies (3–15 fps) required for the flight speeds tested.

5. Significance and Claims

The paper claims that the proposed method represents a significant step toward fully autonomous PV inspection by shifting from waypoint-based navigation to object-oriented localization. By directly utilizing PV module detections, the system allows the UAV to maintain optimal viewing distances and angles, which is critical for high-quality thermal defectoscopy.

The authors emphasize that the method is robust to inaccuracies in the power plant model and GNSS offsets, provided that the visual structures are correctly identified and associated. They note that the approach is versatile, supporting different segmentation algorithms and allowing for the simultaneous extraction of image patches for defect analysis. While the current implementation focuses on row-based layouts, the authors suggest the methodology could be extended to other structures, such as rooftop installations. The work concludes that the integration of visual localization with defectoscopy pipelines offers a practical path to improving the efficiency and reliability of autonomous PV plant maintenance.