💬 NLP

Information Extraction from Electricity Invoices with General-Purpose Large Language Models

This study demonstrates that for general-purpose Large Language Models extracting information from Spanish electricity invoices without fine-tuning, prompt engineering quality is the critical factor driving performance, with few-shot strategies significantly outperforming zero-shot baselines to achieve F1-scores exceeding 96%.

Original authors: Javier Gómez, Javier Sánchez

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Javier Gómez, Javier Sánchez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a huge stack of electricity bills from various companies. Some are neat and clean, others are messy, and all look slightly different. Your goal is to extract specific numbers and names (such as "Total Cost" or "Customer Name") from each individual bill and enter them into a neat table.

For years, computers struggled with this. Old programs were like rigid robots: if a bill looked even the tiniest bit different from what they had been trained on, they became confused and failed.

This article is about teaching a new type of "superintelligent robot" (a so-called Large Language Model, or LLM) to read these bills without needing to be retrained for every single new design. The researchers asked: Can we simply give these intelligent robots the right instructions, and will they figure it out themselves?

Here is a summary of their findings, using some everyday analogies:

1. The "Recipe" is more important than the "Chef's Hat"

The researchers tested two different "chefs" (AI models): Gemini 1.5 Pro (a huge, complex chef) and Mistral-small (a smaller, faster, and more efficient chef).

They also tried adjusting the "cooking settings" (technical parameters like temperature and randomness). They expected that perhaps the larger chef or the perfect cooking settings would make the biggest difference.

The Surprise: It made almost no difference which chef they used or how they adjusted the settings. The difference in results was tiny – like the difference between a cake that is 99% perfect and one that is 99.5% perfect.

The Real Winner: The Prompt (the instructions given to the AI).
Think of the prompt as the recipe.

Zero-Shot (No Recipe): Simply saying "Read this bill and give me the numbers" is like telling a chef to cook without a recipe. The results were okay (about 78% accuracy), but not great.
Few-Shot (With Examples): Giving the AI a few examples ("Here is a bill, and here are the data points I need") is like giving the chef an example dish to copy. Accuracy jumped to over 96%.

The Lesson: It's not about having the most expensive chef or the most modern oven; it's about writing a really good recipe.

2. The "Copy" Strategy Works Best

The researchers tried various ways of providing examples:

The "One-Shot" Method: Showing one example. Good, but not the best.
The "Cross-Validation" Method: This is the champion. They showed the AI three different types of bills (e.g., one tabular, one list-like, and one complex) and then asked it to read a fourth type it had never seen before.
- Result: The AI became a master copier. It learned the concept of an electricity bill, not just the appearance of a specific bill. It achieved 97.6% accuracy with Gemini and 96.1% with Mistral.

3. The "Translation" Problem

Before the AI could read the bills, the researchers had to convert the PDF files into text (Markdown).

The Analogy: Imagine trying to read a handwritten note that was copied through a dirty window. If the note is in a clean table, the text comes out clearly. If the note is squeezed into a messy, multi-column layout, the text gets jumbled.
The Insight: The AI was only as good as the text it received. If the bill had a clean, table-like layout, the AI was nearly perfect. If the bill had a messy, dense design, the AI struggled because the "translation" from PDF to text had lost important clues.

4. The "Hallucination" Trap

They identified a personality difference between the two chefs:

Gemini (The Cautious Chef): If it wasn't sure where a number was, it said, "I don't see it." It was very accurate but sometimes missed things.
Mistral (The Confident Chef): If it wasn't sure, it guessed a number anyway. It found almost everything (high "recall" rate) but also invented some numbers that didn't exist (low "precision" rate).
The Conclusion: If you need to be 100% sure the number is real, you want the cautious chef. If you want to find everything and can check the answers later, the confident chef is fine.

5. The Old Robots Fall Short

The article compares these new AI chefs with the old "robots" (traditional machine learning).

The Old Robot: If trained on Bill Type A, it could read Bill Type A perfectly (91% accuracy). But if given Bill Type B, it crashed and failed (accuracy dropped to 67%). It had memorized the appearance of the paper, not the meaning.
The New AI: It did not need to be trained on Bill Type B. Because it understood the meaning of an electricity bill, it read the new types with over 95% accuracy. It generalized the ability instead of memorizing the pattern.

Summary

The article concludes that to get a computer to read business documents perfectly, you don't need to build a unique brain for every single document type. Instead, you simply need to write better instructions and show it a few good examples.

The "secret recipe" is not the size of the AI model or the complex mathematical settings; it is the quality of the prompt (the instructions) and how well the document is converted into text so the AI can read it.

1. Problem Statement

The automated extraction of structured information from semi-structured business documents (e.g., electricity invoices) remains a critical bottleneck for enterprise management. Traditional approaches face significant limitations:

Rule-based/Template Systems: Fail when layout variations or different suppliers appear.
Classical Machine Learning (ML): Methods such as SVMs or Random Forests require extensive, manually annotated datasets for each new document type, leading to scalability issues and severe overfitting on specific training layouts.
Current Gap: While Large Language Models (LLMs) offer potential for flexible extraction without fine-tuning, there is a lack of rigorous empirical studies quantifying their performance on specific business documents compared to classical baselines. In particular, it is unclear whether Prompt Engineering or the tuning of inference parameters (e.g., Temperature, Top-p) is the primary driver for extraction quality.

2. Methodology

Dataset: IDSEM

The study utilizes a subset of the Invoices Database of the Spanish Electricity Market (IDSEM).

Scope: 75,000 synthetically generated PDF invoices across 6 training templates and 9 test templates.
Annotation: 107 semantic labels (e.g., customer name, total amount, energy prices), organized into 12 categories.
Challenge: The dataset contains unseen templates in the test set to evaluate generalization capability.
Preprocessing: PDFs were converted to Markdown to preserve semantic structure (tables, headings) for LLM input. A ground truth refinement process mapped the 107 possible labels to the specific subset present in each template to avoid penalizing models for missing fields.

Evaluated Models

Two architecturally distinct, general-purpose LLMs were tested without task-specific fine-tuning:

Gemini 1.5 Pro: A large-scale Mixture-of-Experts (MoE) model with a massive context window.
Mistral-small: A dense 24B-parameter model optimized for efficiency with Sliding Window and Grouped-Query Attention.

Experimental Framework

The study isolated two main variables:

Inference Parameters: A grid search with 19 configurations (Temperature, Top-K, Top-P) was conducted on Gemini 1.5 Pro to test sensitivity.
Prompt Engineering Strategies: Six different strategies were compared under deterministic settings (Temperature = 0) for both models:
- Zero-shot: Only task instructions and schema.
- Few-shot_v1: One annotated example with explicit label markers.
- Few-shot_v2: One unannotated example (plain text).
- Cross-valid_v1: Three examples from sequential templates (T1-T3), tested on T4-T6.
- Cross-valid_v2: Three examples from structurally similar templates (T1, T2, T4), tested on structurally different ones (T3, T5, T6).
- Iterative: Sequential extraction field by field.

Evaluation Metrics

Performance was measured at the field level using Precision, Recall, and F1-Score, comparing the model's JSON output against the ground truth after normalization.

3. Key Results

A. Inference Parameters vs. Prompt Engineering

Parameter Sensitivity: The variation in F1-score across all 19 parameter configurations was negligible (0.58 percentage points). The difference between deterministic decoding (Temp=0) and stochastic sampling was insignificant.
Conclusion: Hyperparameter tuning has a negligible impact on extraction accuracy. Practitioners can use deterministic settings for reproducibility without compromising performance.

B. Impact of Prompt Engineering

Zero-shot Baseline: Achieved an F1-score of approximately 78% (Gemini) and approximately 83% (Mistral).
Few-shot Improvement: The introduction of context examples yielded massive gains (>11 percentage points).
Best Performance: The Cross-valid_v2 strategy (grouping by visual similarity) achieved the highest scores:
- Gemini 1.5 Pro: 97.61% F1-score.
- Mistral-small: 96.11% F1-score.
Key Insight: Prompt quality is the dominant lever for performance, far surpassing differences in model architecture or parameter tuning.

C. Model Comparison (Gemini vs. Mistral)

Gemini 1.5 Pro: Showed higher Precision (95.24% in cross-validation), meaning it was better at recognizing when a field was absent (avoiding hallucinations).
Mistral-small: Showed higher Recall (often 100%) but lower Precision, indicating a tendency to "hallucinate" values for fields not present in the document.
Generalization: Although Mistral is smaller, it achieved results only 1–2% away from Gemini, proving that smaller, efficient models can be highly effective with suitable prompt strategies.

D. Comparison with Classical ML

Classical Baseline (SVM): Achieved 91.86% on seen templates but dropped to 67.20% on unseen templates (T7), demonstrating severe overfitting.
LLM Performance: Maintained 95.24% (Gemini) and 91.50% (Mistral) Precision on structurally different, unseen templates.
Conclusion: LLMs generalize the extraction task through semantic understanding, whereas classical ML memorizes layout patterns.

E. Structural Influence

The structure of the document template was identified as the primary determinant of difficulty. Templates with clear tabular layouts (T1) performed significantly better than dense, multi-column designs (T5), which lost quality during PDF-to-Markdown conversion.

4. Key Contributions

Empirical Validation of Prompt Engineering: It was demonstrated that prompt design is the critical factor for LLM-based document processing, rendering extensive hyperparameter tuning unnecessary for high-precision extraction.
Generalization Capability: It was proven that general LLMs can achieve state-of-the-art results on unseen document layouts without fine-tuning, outperforming classical ML methods in generalization scenarios by approximately 24 percentage points.
Architectural Efficiency: It was shown that efficient dense models (Mistral-small) can compete with massive MoE models (Gemini) when combined with optimal prompt strategies.
Practical Framework: A reproducible pipeline was established (PDF $\to$ Markdown $\to$ LLM), achieving near-perfect extraction (97.61% F1) for business automation.

5. Significance and Implications

Operational Transformation: This study provides enterprises with a viable path to automate invoice processing without the high costs and data requirements for training custom models.
Strategic Shift: The focus for document automation is shifting from "model construction" (training/fine-tuning) to "model orchestration" (prompt engineering and pipeline design).
Limitations & Future Work: The study relied on synthetic data and purely text-based processing (Markdown). Future work should address real-world noise, the loss of visual information during conversion, and the integration of Vision-Language Models (VLMs) to capture layout features directly from images.

In summary, the paper concludes that Prompt Engineering is the primary driver of success in LLM-based document extraction, enabling robust, template-agnostic automation that significantly outperforms traditional machine learning approaches.