SemiFA: An Agentic Multi-Modal Framework for Autonomous Semiconductor Failure Analysis Report Generation

This paper introduces SemiFA, an agentic multi-modal framework that autonomously generates semiconductor failure analysis reports in under one minute by integrating defect morphology analysis, SECS/GEM equipment telemetry, and historical data through a four-agent LangGraph pipeline, supported by a new dataset of 930 annotated defect images.

The Gist

Imagine a high-tech semiconductor factory as a massive, ultra-precise bakery. They are baking tiny, microscopic cakes (chips) that power our phones and computers. The problem? Sometimes, the cakes come out with a tiny scratch, a speck of dust, or a crack. If even one of these "defects" slips through, the whole batch is ruined, costing millions of dollars.

Traditionally, finding out why a cake is ruined is a slow, exhausting job. A human expert (the "Baker Detective") has to:

1. Look at high-magnification photos of the defect.
2. Check the oven's logbook to see if the temperature spiked.
3. Search through thousands of old case files to see if this happened before.
4. Write a long, detailed report explaining what went wrong and how to fix the oven.

This process takes days or even weeks.

Enter SEMIFA (SemiFA). Think of SEMIFA as a super-fast, AI-powered detective team that can solve these mysteries in under one minute.

Here is how it works, using a simple analogy:

The Team of AI Detectives

Instead of one person doing everything, SEMIFA uses a team of four specialized AI agents (robots) that pass the case file down a line, like an assembly line for solving crimes. They are orchestrated by a "Team Leader" (LangGraph) who makes sure everyone does their part.

1. The Eye (DefectDescriber):

   • What it does: This agent looks at the photo of the defective chip. It doesn't just see a "scratch"; it describes it like an artist. "It's a thin, linear scratch across 12 tiny squares, likely caused by a bump."
   • The Magic: It uses a super-smart visual brain (DINOv2) that has seen millions of patterns, so it knows exactly what a "scratch" looks like versus a "dust speck."

2. The Investigator (RootCauseAnalyzer):

   • What it does: This is the detective who asks, "Why did this happen?"
   • The Magic: It does two things simultaneously:
     • It checks the Oven's Black Box (Equipment Telemetry/SECS/GEM): It reads the machine's real-time logs to see if the pressure dropped or the temperature spiked 10 minutes ago.
     • It consults the Library of Past Cases (Qdrant Database): It instantly finds 5 similar cases from history. "Hey, last time we saw this ring pattern, it was because the cleaning nozzle was clogged."
   • Result: It combines the photo, the machine logs, and past history to guess the cause.

3. The Judge (SeverityClassifier):

   • What it does: This agent decides how bad the problem is. Is it a "Critical" disaster (throw the whole batch away), a "Major" issue (fix it and watch closely), or just a "Minor" scratch (ignore it)?
   • The Magic: It calculates the financial impact. "This defect will ruin 25% of our yield; we must stop the line immediately."

4. The Mechanic (RecipeAdvisor):

   • What it does: This agent tells the factory workers exactly how to fix the machine.
   • The Magic: Instead of just saying "fix it," it gives specific instructions: "Turn down the polishing speed by 5% and clean the nozzle." It writes these instructions in a format the factory machines can understand.

The Final Report

A fifth agent, the Scribe, takes all the notes from the Eye, Investigator, Judge, and Mechanic and instantly types up a professional PDF report.

The Result:

• Speed: What used to take a human expert several hours (or days for the whole process) now takes 48 seconds. That's a speed-up of about 300 times!
• Accuracy: The system is 92% accurate at identifying the defect type.
• Learning: Every time the system solves a case, it saves the story in its library. So, the more it works, the smarter it gets, just like a human detective who learns from every case.

Why This Matters

Before this, factories relied on the "Tacit Knowledge" of senior engineers—wisdom that lived only in their heads and was hard to pass down. If a senior engineer retired, that knowledge was lost.

SEMIFA captures that wisdom in code. It allows a junior engineer to get the same high-quality analysis as a 20-year veteran, instantly. It doesn't replace the human; it acts as a super-assistant, handling the boring data crunching so humans can focus on the big-picture decisions.

In short: SEMIFA is like giving a factory a team of AI detectives that never sleep, never forget a past case, and can solve a mystery in the time it takes to brew a cup of coffee.

Technical Summary

1. Problem Statement

Semiconductor Failure Analysis (FA) is a critical but labor-intensive process in manufacturing. When wafers exhibit defects, engineers must manually:

• Examine high-resolution inspection images (SEM, optical, or wafer maps).
• Correlate equipment telemetry (alarms, process parameters) from SECS/GEM protocols.
• Consult historical defect records.
• Synthesize a structured report detailing defect classification, root cause hypotheses, severity, and corrective actions.

Current Challenges:

• Time Consumption: The end-to-end process takes days to weeks; even the reporting phase alone consumes several hours per case.
• Scalability: As nodes shrink below 5nm, defect sensitivity increases, creating a bottleneck where inspection volume outpaces the FA engineering workforce.
• Knowledge Gap: Junior engineers often miss subtle correlations, while senior engineers' tacit knowledge remains uncodified.
• Limitations of Existing Tools: Commercial tools (e.g., KLA KLARITY) automate classification but do not generate natural language reports or fuse equipment telemetry into open-ended reasoning. Academic work focuses on pattern recognition without root cause analysis.

2. Methodology: The SemiFA Framework

The authors propose SemiFA, an agentic, multi-modal framework orchestrated by LangGraph. It decomposes the FA process into a five-node pipeline, utilizing a shared state (FAState) to pass data immutably between agents.

A. Architecture & Agents

The system employs four specialized agents (all utilizing LLaVA-1.6 as the language backbone) and one report generator:

1. DefectDescriber:
   • Input: Inspection images (Wafer Map, SEM, or Optical).
   • Process: Preprocesses images (denoising, normalization) and passes them through a frozen DINOv2-base encoder to generate 768-dimensional embeddings.
   • Output: A trained MLP head classifies the defect into one of 9 classes. The LLaVA model then generates a natural language narrative describing defect morphology and spatial distribution.
2. RootCauseAnalyzer:
   • Input: Defect description, classification, equipment telemetry, and historical data.
   • Process:
     • Telemetry: Queries TimescaleDB for real-time SECS/GEM HSMS messages (alarms, process variables like temperature, pressure, RF power).
     • Retrieval: Uses the DINOv2 embedding to query Qdrant (vector DB) for the 5 most similar historical defect cases.
     • Reasoning: LLaVA fuses these modalities to generate ranked root cause hypotheses, explicitly correlating equipment state transitions with defect patterns.
3. SeverityClassifier:
   • Process: LLaVA assesses severity based on industry standards (CRITICAL, MAJOR, MINOR, NONE) and estimates yield impact percentages.
4. RecipeAdvisor:
   • Process: Proposes actionable corrective actions and process parameter adjustments (e.g., reducing polish time, changing gas flow) based on the accumulated context.
5. ReportGenerator:
   • Process: Assembles all outputs into a structured PDF and JSON report, then upserts the new case embedding into Qdrant to create a self-improving knowledge base.

B. Infrastructure

• Orchestration: LangGraph (Stateful DAG).
• Models: LLaVA-1.6 (7B parameters, 4-bit NF4 quantization) and DINOv2-base.
• Data Stores: TimescaleDB (telemetry), Qdrant (vector retrieval), MinIO (image storage).
• Communication: MQTT for ingesting equipment telemetry.

3. Key Contributions

1. Agentic Architecture (C1): A novel four-agent pipeline that modularizes FA into description, analysis, severity assessment, and advisory, enabling auditable and extensible workflows.
2. Multi-Modal Fusion (C2): First system to integrate SECS/GEM equipment telemetry directly into a Vision-Language Model (VLM) pipeline for FA. It fuses visual embeddings, historical retrieval, and real-time machine data.
3. SemiFA-930 Dataset (C3): A new dataset of 930 annotated semiconductor defect images across 9 classes.
   • Unique Feature: Each image is paired with a structured, natural language FA narrative (morphology, root cause, severity, action).
   • Sources: Procedural synthesis (for scratch, particle, edge-crack), WM-811K, and MixedWM38.
   • Significance: Addresses the lack of public datasets pairing defect images with natural language reports.

4. Experimental Results

A. Defect Classification

• Model: Frozen DINOv2-base + Lightweight MLP head.
• Performance: Achieved 92.1% overall accuracy on 140 validation images (Macro F1 = 0.917).
• Comparison:
  • Outperformed ResNet-50 (fine-tuned, 23.5M params) which achieved 82.9% accuracy.
  • Significantly outperformed CLIP (zero-shot), which achieved only 20.7% accuracy, proving that natural image pre-training does not transfer well to wafer maps without domain adaptation.
  • DINOv2 achieved 92.1% accuracy with only 214K trainable parameters (vs. 23.5M for ResNet).

B. Pipeline Latency

• Total Time: 48.4 seconds to generate a complete report on an NVIDIA A100 GPU.
• Speedup: Represents a 150–300x speedup compared to manual expert review (hours to days).
• Bottleneck: The DefectDescriber node (45.5% of time) due to image processing and initial LLaVA inference.

C. Multi-Modal Ablation Study

• Method: Automated GPT-4o judge evaluated root cause quality (Specificity, Actionability, Grounding) under different modality conditions.
• Findings:
  • Full System (Image + Telemetry + Retrieval): Composite Score 3.60.
  • Image Only (Baseline): Score 2.74.
  • Impact: Multi-modal fusion improved reasoning quality by +0.86 points.
  • Key Insight: Equipment telemetry was identified as the most critical modality. Removing telemetry dropped the score significantly (3.60 → 3.33), whereas removing retrieval had a marginal effect in the small sample size.

D. Domain Adaptation Findings

• Fine-tuning LLaVA-1.6 on the 790-example training set via QLoRA resulted in severe overfitting (train loss collapsed, validation quality degraded).
• Conclusion: While 790 samples are sufficient for a lightweight classifier, a 7B-parameter VLM requires at least 5,000+ annotated pairs for stable domain adaptation.

5. Significance and Future Work

Significance:

• Automation: Demonstrates the feasibility of fully autonomous FA report generation, drastically reducing time-to-disposition for defect lots.
• Knowledge Preservation: Encodes senior engineer expertise into a retrievable, reusable system, democratizing FA capabilities in fabs with limited staffing.
• Data Contribution: SemiFA-930 fills a critical gap in the literature by providing the first public dataset linking semiconductor defect images to structured natural language reports.
• Telemetry Integration: Proves that SECS/GEM data is a "load-bearing" modality for accurate root cause reasoning, a capability previously absent in academic FA research.

Limitations & Future Directions:

• Dataset Size: The current dataset is too small for robust VLM fine-tuning; future work aims to expand to 5,000+ real-world pairs.
• Modalities: Currently tested primarily on wafer maps; performance on raw SEM/optical micrographs needs quantitative validation.
• Human-in-the-Loop: The system is designed to augment, not replace, engineers. Critical decisions (wafer scrap) require human review.
• Parallelization: Future iterations will explore running agents (e.g., Severity and Recipe) in parallel to further reduce latency.

In summary, SemiFA represents a significant step forward in industrial AI, successfully bridging the gap between visual defect detection, equipment telemetry, and natural language reasoning to automate a complex, high-stakes engineering workflow.