🤖 AI

POTracker: Optimizing Large Language Models for Standard-Compliant Power Outage Report Generation

The paper introduces POTracker, a fine-tuned LLM utilizing a novel loss function that balances textual and structural similarity, to achieve state-of-the-art accuracy in generating standard-compliant, machine-readable power outage reports for the US utility sector.

Original authors: Hung Phan, Aniroop Naladala, Dubey Avanindra, Supryia Chinthavali, Lunga Dalton, Ali Jannesari

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hung Phan, Aniroop Naladala, Dubey Avanindra, Supryia Chinthavali, Lunga Dalton, Ali Jannesari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a very important message to a giant, super-organized library called ODIN. This library manages information about power outages across the entire United States. But here's the catch: the library is extremely picky. It doesn't just want the story of what happened; it wants the story written in a very specific, rigid format (like a strict XML code) with exact tags, brackets, and rules.

If you send a messy, handwritten note or a loose email, the library's computer can't read it. It's like trying to fit a square peg into a round hole.

This is the problem the authors of this paper are trying to solve. They created a smart tool called POTracker to act as a translator.

The Problem: The "Messy Input" vs. The "Strict Output"

Think of power outage reports coming in from the real world like a chaotic pile of letters. Some are typed on computers, some are scribbled on napkins, some are perfect XML files, and some are just plain text.

The Goal: Turn all this chaos into a perfect, standardized "form" that the library's computer can instantly understand.
The Challenge: Regular AI (Large Language Models or LLMs) are great at writing stories or answering questions, but they are terrible at following strict formatting rules. If you ask a normal AI to write a power report, it might get the facts right but mess up the brackets, miss a required tag, or put the information in the wrong order. To the library computer, a missing bracket is the same as a missing fact.

The Solution: POTracker

The authors built POTracker, which is a specialized AI trained specifically to be a "format police officer" as well as a storyteller.

Here is how they made it work, using a simple analogy:

1. The Teacher (GPT-5.2)
First, they needed to know what the "perfect" answer looked like. Since no one had a database of perfect answers, they hired a very smart, expensive AI (GPT-5.2) to act as the Teacher. They fed the messy reports to this Teacher and asked it to rewrite them into the perfect, strict format. These rewritten reports became the "Gold Standard" or the answer key.

2. The Student (Qwen2.5-7B)
Next, they took a smaller, open-source AI (Qwen2.5) to act as the Student. They wanted to teach this Student to produce the same perfect answers as the Teacher.

3. The New Grading System (POTrackerLoss)
This is the paper's biggest innovation. Usually, when you train an AI, you grade it like a spelling test: "Did you write the right word?"

Old Way: If the AI wrote "Customer Count: 500" but the format required <count>500</count>, the old grading system might say, "Good job, you got the number right!" even though the format was wrong.
POTracker's Way: The authors invented a new grading system called POTrackerLoss. It grades the AI on two things at once:
- The Story (Text): Did you get the facts right? (e.g., 500 customers).
- The Structure (Tags): Did you use the right brackets and order? (e.g., <count>500</count>).

Think of it like grading a student's essay. The old way only checked if the sentences made sense. The new way checks if the sentences make sense AND if the student used the correct font, margins, and page numbers. If the margins are wrong, the grade goes down, even if the essay is brilliant.

The Results: Did it Work?

The authors tested their new "Student" (POTracker) against other methods:

The "Raw" AI: Without special training, the AI was terrible at the format (only about 16% accurate).
The "Rule-Based" Robot: They tried a simple computer program that just followed a list of rules. It was okay (about 61% accurate), but it couldn't handle messy or unexpected inputs well.
POTracker: The new AI, trained with the special "Text + Structure" grading system, crushed the competition. It achieved 86.47% structural accuracy.

The Human Check

Finally, to make sure the "Teacher" (GPT-5.2) wasn't just making up fake perfect answers, human experts looked at 50 of the generated reports. They gave them a score of 4.03 out of 5, confirming that the AI was actually producing high-quality, useful reports.

Summary

In short, the paper says: "We found that regular AI is too messy for strict power outage reports. So, we built a new training method that forces the AI to care about both the facts and the formatting rules. The result is an AI that can turn messy, real-world outage reports into perfect, computer-ready data much better than any previous method."

Technical Summary: POTracker

Problem Statement
While Large Language Models (LLMs) excel at general text generation, their application to domain-specific data generation remains challenging. Specifically, generating power outage reports for the utility sector requires outputs that are not only semantically accurate but also strictly adhere to rigid structural and format standards defined by regulatory bodies. The paper addresses the interoperability problem within the United States' power grid, where outage reports must be machine-readable (e.g., JSON or XML) and compliant with the Common Information Model (CIM) IEC 61968-3 standard, as defined by the Outage Data Initiative Nationwide (ODIN) benchmark. Current inputs are often unstructured or non-standard (free-text, malformed XML/JSON), and transforming these into canonical, standard-compliant reports typically requires manual intervention by domain experts.

Methodology
The authors propose POTracker, an optimized LLM pipeline designed to automate the generation of standard-compliant power outage reports. The methodology involves the following components:

Base Model: The system utilizes Qwen2.5-7B-Instruct, a compact open-weight LLM chosen for its balance of accuracy and efficiency in instruction following and structured generation.
Data Labeling: Due to the lack of existing datasets containing ground-truth standard reports, the authors employed GPT-5.2 (a closed LLM) to generate synthetic standard reports from non-standard inputs based on ODIN documentation. These synthetic reports serve as the "gold" labels for training.
POTrackerLoss (Custom Loss Function): The core innovation is a novel loss function that extends traditional cross-entropy. It incorporates a sequence-level quality signal derived from a custom metric, POTrackerMetric.
- POTrackerMetric: This metric evaluates the similarity between a generated report and a ground-truth report by combining two factors:
  1. Text Similarity: Measured via cosine similarity of textual values to ensure semantic accuracy.
  2. Tag Structure Similarity: Measured using difflib.SequenceMatcher to evaluate the alignment of XML hierarchical structures, attributes, and node ordering.
- The total loss is calculated as $L_{total} = L_{CE} + \lambda \cdot p$ , where $L_{CE}$ is the standard token-level cross-entropy, and $p$ is a penalty derived from the inverse of the POTrackerMetric score.
Baselines: The study compares POTracker against:
- Fine-tuning Variants: Token Weight Cross Entropy (TW-CE), Tag Masking Cross Entropy (TM-CE), Example Weight Cross Entropy (EW-CE-PO), and Direct Preference Optimization (DPO).
- Rule-Based Approach: A heuristic program that manually maps non-standard tags to standard tags based on rules defined by domain experts.
- Original Baseline: The unmodified Qwen2.5-7B-Instruct model.

Key Contributions

POTrackerMetric: A novel evaluation formula that quantifies the similarity between generated and expected power outage reports by jointly considering textual content and XML tag structure.
POTrackerLoss: A custom loss function that integrates the POTrackerMetric into the fine-tuning process, allowing the LLM to optimize for both semantic correctness and structural validity simultaneously.
Comprehensive Evaluation: A rigorous comparison of the proposed method against multiple fine-tuning strategies and a rule-based baseline on a dataset of 1,000 power outage reports.

Results
The experiments were conducted on a dataset of 1,000 entities (800 for training, 100 for validation, 100 for testing). The results indicate:

Superior Performance: POTracker significantly outperformed all other fine-tuning approaches. It improved overall accuracy by up to 51% compared to the baselines.
Structural Accuracy: POTracker achieved a structural accuracy (Tag Accuracy) of 86.47%, compared to only 3.56% for the original Qwen2.5-7B-Instruct and approximately 3.5% for other fine-tuning variants.
Baseline Comparison: While the rule-based approach performed well structurally (79.00% Tag Accuracy), POTracker surpassed it in overall accuracy (69.14% vs. 61.46%).
Limitations of Other Methods: Standard cross-entropy variants (TW-CE, TM-CE, EW-CE-PO) and DPO failed to provide meaningful gains over the unmodified baseline, suggesting that standard NLP fine-tuning pipelines are insufficient for tasks requiring strict schema compliance without structural-aware loss functions.
Human Evaluation: A study involving two domain experts rated the quality of the GPT-5.2 generated labels (used as ground truth) with an average score of 4.03 on a 0–5 scale, validating the reliability of the synthetic dataset.

Significance and Claims
The paper claims that POTracker effectively addresses the challenge of generating domain-specific, standard-compliant data where strict adherence to format is as critical as semantic meaning. By introducing a loss function that explicitly penalizes structural deviations, the authors demonstrate that LLMs can be optimized to produce machine-readable reports that meet industry regulatory standards (CIM IEC 61968-3). The work highlights that while rule-based systems are strong at enforcing structure, a custom-optimized LLM can achieve a better balance of semantic flexibility and structural correctness, potentially reducing the manual burden on utility providers and emergency responders in processing outage data.