Technical Summary: MADP – A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop

1. Problem Statement

Document processing automation in enterprise environments remains a critical bottleneck. Traditional approaches relying on manual verification are labor-intensive and error-prone, while standard Optical Character Recognition (OCR) systems struggle with complex layouts, multi-page structures, and domain-specific terminology. Although Large Language Models (LLMs) have demonstrated remarkable capabilities in extracting structured information from unstructured documents, their direct deployment in production faces three significant hurdles:

1. Reliability: Concerns regarding hallucinations and non-deterministic accuracy in mission-critical applications.
2. Cost & Sustainability: High computational and environmental costs associated with large-scale inference.
3. Auditability: A lack of interpretability and auditability required in regulated industries.

Furthermore, existing Human-in-the-Loop (HITL) systems often lack mechanisms for continuous learning from human feedback, and prior work has not comprehensively analyzed the full sustainability impact of AI-assisted processing compared to manual baselines.

2. Methodology: The MADP Architecture

The authors propose MADP, a multi-agent architecture that orchestrates specialized AI agents under human oversight to achieve end-to-end document processing. The system integrates deep learning-based classification and parsing with LLM-based extraction, governed by a novel Prompt Fine Tuning with Feedback Inheritance (PFTFI) mechanism.

2.1 System Components

The pipeline consists of five sequential specialized agents:

• Classificator Agent: Utilizes a ResNet-18 Convolutional Neural Network (CNN) to identify document types and supplier categories. It is optimized for low latency by freezing the first three convolutional blocks and fine-tuning the fourth on document headers. Input images are cropped to the top 40% to focus on supplier identification marks.
• Splitter Agent: Segments multi-page documents into logical units (e.g., individual invoices) using visual features (page breaks, headers) and semantic analysis to prevent context mixing.
• Parser Agent: Converts raw document formats into structured Markdown representations optimized for LLM processing. Using the Docling library, it analyzes layout, identifies tables, and determines reading order. This step reduces token count by 35% compared to raw OCR while preserving semantic information.
• Extraction Agent: Employs LLMs to infer structured information (JSON) based on carefully designed prompts that include constraints, data types, and illustrative examples to minimize hallucinations. The system supports parallel extraction strategies using multiple LLM backends (e.g., Granite, Mistral, DeepSeek) with consensus voting to boost confidence.
• Validator Agent: Acts as a sentinel, enforcing data quality through atomic consistency checks (e.g., verifying that subtotal + tax = total, checking date logic, and validating VAT rates). It routes documents with low confidence scores or validation failures to human reviewers.

2.2 The PFTFI Mechanism

A core innovation is the Prompt Fine Tuning with Feedback Inheritance (PFTFI) loop. When human reviewers correct extraction errors, the system:

1. Captures the original data and the correction.
2. Generates structured feedback describing the error pattern.
3. Updates extraction and parsing prompts by incorporating new examples and refining instructions.
4. Propagates these updates bidirectionally to pending documents in the queue that exhibit similar characteristics.
   This allows for continuous improvement of the system's behavior without retraining the underlying neural models.

2.3 Human-in-the-Loop (HITL) Interface

The validation interface presents documents side-by-side with extracted metadata. It highlights low-confidence fields and offers rapid correction mechanisms. Crucially, human operators correct the extracted metadata rather than annotating source documents directly, generating rich structured feedback for the PFTFI mechanism.

3. Key Contributions

• Multi-Agent Orchestration: A modular pipeline combining CNNs for classification, specialized parsing for layout understanding, and LLMs for semantic extraction, all coordinated under a validation framework.
• PFTFI Mechanism: A novel approach to continuous learning that refines prompts and rules based on human feedback without requiring model retraining, enabling rapid adaptation to new suppliers and document variants.
• Sustainability Analysis: The paper presents what it claims is the first detailed sustainability analysis of AI-assisted document processing, quantifying reductions in CO2 emissions, energy consumption, and water usage relative to fully manual and fully automated baselines.
• Comprehensive Benchmarking: A comparative evaluation of multiple LLM backends (Granite-Docling, Mistral-Small, DeepSeek-OCR) regarding accuracy, latency, and resource footprint in production-like conditions.

4. Experimental Results

The system was evaluated on a production dataset of 955 real-world documents processed through January 2026, covering over 50 suppliers and multiple languages (Italian, English, German, French, and Arabic).

• Automation Rate: The system achieved a 97.0% full-pipeline automation rate, with only 3% of documents requiring non-AI fallback (legacy OCR or manual portal).
• Accuracy:
  • On a stratified ablation subset of 100 documents (5 per category across 20 categories), the full MADP configuration with HITL supervision attained 98.5% document-level accuracy.
  • The Parser Agent contributed the largest single improvement (+17.5 percentage points) over a baseline direct LLM approach, validating the importance of layout-aware preprocessing.
  • Mistral-Small-3.2 was identified as the optimal LLM backend for the production deployment, achieving the highest F1 score (92.9%) and recall (98.2%).
• Operational Efficiency: In a projected scenario of 100,000 invoices per year, the hybrid AI+HITL approach reduced Full-Time Equivalent (FTE) requirements by approximately 70% compared to manual processing (7 FTEs vs. 23 FTEs), while maintaining 98.5% accuracy (compared to 85% for a "Pure AI" approach).
• Sustainability Impact: Compared to the manual baseline, the AI+HITL approach resulted in:
  • 69% reduction in CO2 emissions (12.3 tons/year).
  • 69% reduction in energy consumption.
  • 63% reduction in water usage.
    The authors attribute these gains primarily to the reduction in office-related overhead (FTEs) rather than just the efficiency of the AI inference itself.

5. Significance and Claims

The paper claims that MADP demonstrates that intelligent orchestration of AI agents with strategic human intervention achieves a superior balance of accuracy, cost, and environmental impact compared to fully manual or fully automated approaches.

Key significance points include:

• Practical Viability: The 97% automation rate on a diverse, real-world dataset validates the architecture's robustness in operational settings.
• Adaptability without Retraining: The PFTFI mechanism proves that systems can adapt to new document types and suppliers rapidly by updating prompts and rules, avoiding the high cost and latency of model retraining.
• Sustainability Leadership: The study challenges the notion that AI is inherently less sustainable than manual work, showing that a hybrid model can significantly reduce the carbon and resource footprint of document processing by drastically reducing the need for human labor hours.
• Modular Design: The architecture allows for graceful degradation; if a document fails at classification or parsing, it is routed to manual inspection rather than producing unreliable extractions, ensuring data integrity.

The authors conclude that while the system is currently optimized for invoices, the modular multi-agent pattern is transferable to other business documents, provided that specific classification and extraction logic are adapted.