💻 computer science

Physical Annotation for Automated Optical Inspection: A Concept for In-Situ, Pointer-Based Training Data Generation

This paper presents a novel physical annotation system that utilizes pointer-based in-situ interaction and projector guidance to efficiently convert expert human inspection knowledge into standardized training data for automated optical inspection, thereby bridging the gap between non-IT experts and machine learning pipelines.

Original authors: Oliver Krumpek, Oliver Heimann, Jörg Krüger

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Oliver Krumpek, Oliver Heimann, Jörg Krüger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to spot defects on a mechanical part, like a scratch on a gear. Usually, to teach the robot, a human has to sit at a computer, look at a photo of the gear, and draw a box around the scratch using a mouse. This is slow, boring, and requires the human to stop their actual work to "annotate" the data.

This paper introduces a much more natural way to do this: Physical Annotation.

Here is how the system works, broken down into simple concepts:

1. The "Magic Wand" Approach

Instead of using a mouse on a screen, the inspector holds a special pointer (like a high-tech pen) and points directly at the actual object sitting on the table.

The Analogy: Think of it like a teacher pointing at a map in a classroom to show a student where a city is, rather than asking the student to type the city's coordinates into a computer. The inspector points at the real scratch, and the system understands exactly where they are pointing in 3D space.

2. The "Ghostly Guide" (Visual Feedback)

To make sure the inspector is pointing accurately, the system uses a projector.

The Analogy: Imagine a "ghost" hand or a laser line appearing on the object that follows the inspector's pointer in real-time. If the inspector moves their hand, the ghost line moves with it. This gives them instant visual confirmation that the computer is tracking them correctly, making the process feel intuitive and confident.

3. The "Two-Step Dance" (Solving the Blocking Problem)

There is a small problem: If you point at an object, your hand (and the pointer) blocks the camera's view of the object. You can't take a photo of the scratch if your hand is covering it.

The Solution: The system splits the job into two steps:
1. The Trace: The inspector points at the object to draw the shape of the defect. The system records the path of the pointer.
2. The Capture: The inspector moves their hand away. The system then takes a clear photo of the object.
The Magic Trick: The computer uses a technique called optical flow (which is like a smart video tracker) to "glue" the path the inspector drew onto the clear photo. It essentially says, "Even though the hand isn't in this photo, we know exactly where that line was relative to the object, so let's draw it there." This allows one single pointing action to generate many different training photos from different angles.

4. The "Translator" (Calibration)

For this to work, the computer needs to know exactly where the pointer is in relation to the camera and the projector.

The Analogy: Before the system starts, it goes through a "calibration dance." The inspector touches specific points on a special board with the pointer. This teaches the computer the exact geometry of the room, the camera, and the pointer, so it can translate a physical movement into a digital coordinate perfectly.

5. The Result

The system takes these physical movements and turns them into standard digital labels (like bounding boxes or outlines) that machine learning software (specifically a tool called CVAT) can read.

The Benefit: This allows experts who are great at spotting defects but bad at coding or using complex software to easily create training data. It bridges the gap between human expertise and the robot's need for data, making the whole process faster and more integrated into the actual factory workflow.

In summary: The paper proposes a system where you teach a robot to see defects by simply pointing at them with a special wand, while a projector shows you a "ghost" of your pointing path, and the computer cleverly combines your pointing path with clear photos to create perfect training data.

Technical Summary: Physical Annotation for Automated Optical Inspection

Problem Statement
The implementation of AI-driven Automated Optical Inspection (AOI) in industrial settings is frequently hindered by the scarcity of high-quality, manually labeled training data. Current workflows face several bottlenecks:

Data Scarcity: In safety-critical quality control, defective parts are rare compared to the total volume of inspected objects, making it difficult to collect sufficient positive samples.
Workflow Disruption: The process of digitizing and annotating defects is often separate from the actual inspection activity, requiring external devices, supplementary software, and distinct personnel.
Skill Barriers: Effective use of existing annotation tools (e.g., CVAT, LabelImg) typically requires IT expertise, creating a gap between domain experts (inspection personnel) and the machine learning pipeline.
Resistance to Change: Established inspection processes are difficult to modify due to financial constraints and the need for management approval, especially when new workflows are not intuitive.

Methodology
The paper proposes a novel Physical Annotation System designed to generate training data in-situ through direct, pointer-based interaction with physical objects. The system architecture and workflow are defined as follows:

Hardware Setup: The system comprises a 5 MP industrial camera, a motion tracking system with a tracked pointer, and a projector for visual feedback. All components are mounted on a rigid structure 90 cm above the workspace.
Calibration and Transformation:
- Pivot Calibration: Determines the fixed position of the pointer tip relative to the device's dynamic reference frame (DRF) by solving a system of linear equations based on multiple poses.
- System Calibration: Uses an asymmetric circle pattern with a 0.5 mm central notch. The pointer tip is placed in these notches to collect 3D reference points, while the camera captures 2D images. A Perspective-n-Point (PnP) algorithm solves for the camera-to-world transformation ($CTW$) and pointer-to-world transformation ($WTP$).
- Projector Calibration: Utilizes an extended pinhole model and a method by Moreno et al. involving a checkerboard and projected grey patterns to establish the projector-to-camera transformation ($BTC$).
- Mathematical Formulation: The system maps the 3D pointer tip position ( $p^P_{tip}$ ) to 2D pixel coordinates ( $u_C, v_C$ ) on the camera image and ( $u_B, v_B$ ) on the projector plane using a chain of homogeneous transformations and intrinsic matrices ( $K_C, K_B$ ).
Annotation Process:
- In-Situ Interaction: Inspectors use the tracked pointer to draw trajectories (lines, polygons, bounding boxes) directly on the object surface. Visual guidance is projected onto the object to ensure accuracy.
- Multistage Data Generation: To avoid occlusion (where the pointer blocks the view of the defect during recording), the system separates annotation from data capture.
  1. Trajectory Capture: The pointer traces the defect on a stationary object.
  2. Data Extraction: The object is moved within the camera's field of view. An optical flow-based point-tracking algorithm "glues" the recorded trajectory to the object's surface across multiple camera frames where the pointer is not visible.
- Output: The system exports standardized annotation formats (e.g., YOLO coordinates, XML, JSON) and integrates directly with the CVAT API for editing and consistency.

Key Contributions

In-Situ, Pointer-Based Interaction: A novel approach allowing non-IT experts to generate training data by pointing at objects, eliminating the need for screen-based annotation tools.
Workflow Integration: The system embeds the annotation process directly into existing inspection workflows, reducing the time and effort required to digitize expert knowledge.
Data Augmentation via Optical Flow: A method to generate multiple training samples from a single annotation event by tracking the object's movement and projecting the annotation path onto images taken from various angles.
System Architecture: A fully calibrated hardware and software stack that converts 3D spatial interactions into 2D image annotations compatible with open-source ML pipelines.

Results
The paper presents preliminary evaluation results focusing on the feasibility of the concept rather than a comprehensive performance benchmark:

Trajectory Accuracy: In preliminary experiments involving three individuals tracing various shapes (lines, hexagons, rectangles, sine curves, and irregular shapes) on a flat surface over 90 repetitions:
- The mean deviation for regular shapes was 0.72 mm.
- The mean deviation for irregular shapes was 1.62 mm.
Workflow Validation: The system successfully demonstrated the ability to capture detailed annotation trajectories and integrate them with CVAT, confirming the feasibility of the proposed data generation pipeline.

Significance and Claims
The authors position this work as a conceptual bridge between human expertise and automated data generation. The primary significance lies in:

Democratizing Data Generation: Enabling domain experts (inspectors) to contribute directly to ML training without requiring IT skills.
Enhancing Transparency: Making the data generation process more transparent and integrated, potentially increasing the acceptance of AI in industrial processes.
Efficiency: Preventing valuable training samples from being overlooked by capturing data at the moment of inspection.

The paper explicitly states that the focus is on the conceptual description and technical implementation. It acknowledges that further research is required to validate tracking accuracy, annotation efficiency, and overall system usability in practical, large-scale deployments. The authors do not claim the system is currently a finished commercial product but rather a proof-of-concept that addresses the critical bottleneck of training data availability in industrial AI.