Technical Summary: Enhancing Generative Information Extraction with Two-step Validation

Problem Statement

Information Extraction (IE) in the product domain faces significant challenges due to the scarcity of domain-specific labeled data, the heterogeneity of product information, and strict data privacy constraints. These issues are particularly acute in the context of the European Commission's Digital Product Passport (DPP), which mandates the structured disclosure of product attributes (e.g., materials, components, manufacturing details) starting in 2027.

While Large Language Models (LLMs) offer strong generalization capabilities suitable for low-data regimes, they often underperform compared to fine-tuned Pre-trained Language Models (PLMs) on classification-oriented IE tasks. Conversely, fine-tuned PLMs require substantial labeled data and struggle with generalization. Furthermore, commercial LLMs are often unsuitable for this use case due to data privacy concerns, necessitating the use of open-source, locally deployable models. However, smaller open-source LLMs frequently suffer from hallucinations, span boundary mismatches, and poor performance on weakly expressed or low-salience entities.

Methodology

The authors propose a two-step validation method that reformulates the conventional generative IE task. Instead of asking an LLM to extract information directly from raw text, the pipeline operates as follows:

1. First Step (PLM Prediction): A fine-tuned PLM (specifically RoBERTa or DeBERTa) processes the text to generate an initial structured prediction (or an empty dictionary as a baseline).
2. Second Step (LLM Validation): An open-source LLM is instructed to validate and correct the PLM's output rather than extracting from scratch. The LLM acts as a post-hoc verifier, leveraging its general knowledge and text-refinement capabilities to fix errors, fill gaps, or refine the initial prediction.

This approach mimics a self-refine framework without recursive prompting. The study utilizes two open-source datasets (Amazon Product Descriptions and E-commerce Text Classification) annotated with six entity classes: Size, Weight, Product Number, Component, Material, and Manufacturer. Experiments were conducted across seven instruction-tuned LLMs from the Llama, Mistral, and Gemma families, ranging from 3B to 70B parameters.

Key Contributions

1. Hybrid Two-Step Pipeline: The introduction of a generative IE method that integrates a PLM block into the pipeline, shifting the LLM's role from primary extractor to validator/corrector.
2. Enhanced Performance on Low-Salience Entities: The method specifically improves the extraction of weakly expressed entities (e.g., Component and Manufacturer) that appear sparsely in text, where standard extraction often fails.
3. Efficiency and Privacy: The approach enables mid-size LLMs to achieve performance comparable to larger models, facilitating local deployment on limited hardware (e.g., consumer laptops) while ensuring data privacy by avoiding commercial API calls.
4. Prototype Application: Development of a demo application that implements this two-step method for product information extraction, tailored for DPP use cases.

Results

• Overall Performance: The two-step validation method generally improved F1 scores for mid-size and large models. The largest gains were observed with the Llama-3.3 70B model.
• Baseline Utility: Even when the PLM provided an empty dictionary (baseline), the LLM's ability to revise empty predictions often resulted in better IE performance than direct extraction, suggesting the gains are not solely dependent on high-quality initial PLM predictions.
• Model Size Sensitivity: The smallest open-source LLMs (e.g., Llama-3.2 3B and Gemma-3 4B) often experienced a performance drop under the validation task, likely due to the increased difficulty of processing longer prompted contexts.
• Entity-Specific Gains: The validation task significantly boosted Recall for weakly expressed entities like Component and Manufacturer. In contrast, explicit entities like Size and Weight were often better handled by direct extraction.
• PLM Quality Impact: The quality of the final output was positively correlated with the quality of the first-step PLM prediction. Using a PLM fine-tuned on the same dataset yielded the best results, boosting performance by up to 7% in some cases.
• Local Deployment: The demo application successfully processed product descriptions on an M1 MacBook Pro in under 5 seconds using Llama-3.2, demonstrating the feasibility of local deployment without traditional GPUs.

Significance and Claims

The paper claims that reformulating generative IE as a validation task offers a practical solution for the DPP use case. By leveraging the correction capabilities of LLMs to refine PLM outputs, the method achieves a more efficient performance-to-size ratio. This allows smaller, locally deployable models to handle complex extraction tasks that typically require larger models or extensive labeled data.

The authors emphasize that this approach addresses the specific bottlenecks of the DPP: the need for structured data from scarce, heterogeneous, and confidential sources. By enabling local deployment, the method mitigates data privacy risks associated with commercial LLMs. However, the paper remains modest regarding the smallest models, acknowledging that while they improve recall, they may still struggle with precision in the validation setting. The work serves as a proof-of-concept for integrating PLMs and LLMs in a pipeline to balance efficiency, generalizability, and privacy in real-world industrial applications.