🤖 AI

Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production

This paper presents a microservice architecture and operational experience for deploying document AI pipelines at scale, highlighting design decisions like separating GPU and CPU tasks and revealing that OCR latency and shared GPU capacity, rather than model complexity or worker count, are the primary bottlenecks in production.

Original authors: Yao Fehlis, Benjamin Bengfort, Zhangzhang Si, Vahid Eyorokon, Prema Roman, Patrick Deziel, Devon Slonaker, Steve Veldman, Ben Johnson, Joyce Rigelo, Michael Wharton, Steve Kramer

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yao Fehlis, Benjamin Bengfort, Zhangzhang Si, Vahid Eyorokon, Prema Roman, Patrick Deziel, Devon Slonaker, Steve Veldman, Ben Johnson, Joyce Rigelo, Michael Wharton, Steve Kramer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a massive, high-speed mail sorting facility. Every day, thousands of complex documents arrive—some are crumpled, some are blurry, some are multi-page contracts, and some are just photos of receipts. Your goal is to read every single one, understand what it is, and pull out specific details (like dates, names, or amounts) to put into a neat digital spreadsheet.

This paper describes how the team at Kungfu.ai built a "factory" to do exactly that, but with a twist: they realized that trying to do everything with one giant machine (a monolith) was slow, expensive, and prone to breaking. Instead, they built a microservice architecture, which is like turning that factory into a team of specialized workers, each with their own specific job, passing documents down a conveyor belt.

Here is how their system works, explained simply:

1. The Three Specialized Teams (The Microservices)

Instead of one robot trying to scan, read, and think all at once, they split the work into three distinct teams:

The Gateway (The Receptionist):
This is the front door. Its only job is to take the documents from the outside world, store the pictures safely in a digital warehouse (Object Storage), and put a "ticket" in a queue saying, "Hey, we have a new document to process!" It doesn't do any heavy lifting; it just manages the flow. If the Receptionist gets sick, no new mail enters, but the workers inside keep working on what they already have.
The Workers (The Managers):
These are the CPU-powered managers. They pick up tickets from the queue, look at the document, and decide what needs to happen next. They are great at organizing, waiting for answers, and moving data around, but they aren't very good at the heavy "thinking" or "seeing" tasks. They act as the conductor of the orchestra, telling the specialists when to play.
The Inference Service (The Heavy Lifters):
This is the team with the expensive, super-fast GPUs (graphics cards). They are the only ones allowed to do the hard visual work: OCR (turning blurry images into text) and LLM Parsing (using a giant AI brain to understand the text and pull out specific fields). Because these machines are expensive, the team keeps them separate so they don't have to pay for them when the Workers are just waiting around.

2. The Assembly Line (The Pipeline)

When a document arrives, it goes through a specific sequence of steps, like a car on an assembly line:

Classification (The Sorter):
First, the system needs to know what kind of document it is (e.g., is it an invoice or a medical form?).
- The Trick: They use a "Hybrid Strategy." First, they use a cheap, fast, local AI (CLIP-KNN) to guess the type. It's 92% accurate and free. If the AI is unsure (less than 70% confidence), then they send it to the expensive, super-smart AI (Claude Sonnet) to double-check. This saves a ton of money because the cheap AI gets it right 96% of the time on its own.
OCR (The Translator):
The document is an image. The system uses the GPU team to turn that image into actual text, word by word.
- The Surprise: The authors found that this step is the biggest bottleneck. It takes the most time. Even though the "thinking" AI (LLM) is complex, it only reads the text once for the whole document. The OCR has to look at every single page individually. It's like the difference between reading a whole book (fast) vs. translating every single word of a book one by one (slow).
Text Stitching (The Gluer):
If a document has 10 pages, the system takes the text from page 1, then page 2, and so on, and glues them together into one long story.
Structured Parsing (The Extractor):
Finally, the "Thinking AI" (LLM) reads the glued-together text and fills in a digital form (like a JSON file) with the specific data needed (e.g., "Invoice Date: 2023-10-01").

3. The Secret Sauce: How They Scale

The paper highlights two major "aha!" moments they had while running this system:

The Bottleneck is the Eyes, Not the Brain:
Everyone assumed the "Thinking AI" (LLM) would be the slow part. They were wrong. The "Eyes" (OCR) were the slow part. Because OCR is the bottleneck, they realized they need to scale the GPU team specifically for OCR, not just add more managers (Workers). If you add more managers but not more "eyes," the managers just sit around waiting.
The Queue is the Safety Net:
They use a message queue (like a digital waiting room). If the "Heavy Lifters" are busy, the documents just wait in line. This prevents the system from crashing. It also means if one worker crashes, the document just goes back to the line to be picked up by someone else, ensuring nothing gets lost.

4. The Results

By using this architecture, they achieved two amazing things:

Cost: They dropped the cost of processing a page from $0.01 to $0.001 (a 10x reduction). They did this by using the cheap AI for most tasks and only paying for the expensive AI when absolutely necessary.
Reliability: They maintained 96% accuracy while processing thousands of pages per hour.

Summary

The paper argues that building a production AI system isn't just about having the smartest model; it's about engineering. You have to separate the "thinking" from the "organizing," use queues to manage traffic, and realize that the slowest part of your process (in this case, reading the text) is the one you need to optimize, not the part you think is the most complex.

They turned a chaotic, expensive process into a streamlined, cost-effective factory where every worker knows exactly what to do, and the expensive machines are only used when they are truly needed.

Technical Summary: Operationalizing Document AI via Microservice Architecture

Problem Statement

Despite a steady stream of innovation in document understanding models (e.g., LayoutLM, Donut, VLMs), a significant gap exists between academic model definitions and their reliable deployment at production scale. Practitioners face substantial engineering challenges when moving from model checkpoints to on-demand systems that process thousands of heterogeneous documents daily. These challenges include:

Resource Mismatch: Balancing CPU-bound orchestration (I/O, data marshaling) with GPU-bound inference (OCR, VLMs) within fixed memory constraints.
Operational Complexity: Managing failures, retries, timeouts, and model meta-information (e.g., confidence scores) while ensuring observability.
Scalability & Cost: Achieving horizontal scaling without over-provisioning expensive GPU resources, particularly when I/O operations dominate execution time.
Security & Compliance: Processing sensitive data within secure enclaves or on-premise platforms while maintaining auditability.

Existing literature offers few guidelines on the system-level architecture required to wrap multiple model paradigms (classification, OCR, structured extraction) into a fault-tolerant, cost-effective pipeline.

Methodology: Microservice Architecture

The authors propose a microservice architecture that decomposes document understanding into three independently scalable services, decoupling GPU-bound inference from CPU-bound orchestration.

1. System Components

Gateway (Ingestion Service): A lightweight, I/O-bound entry point responsible for storing page images in object storage, creating tracking records in a relational database, and enqueuing document IDs. It handles synchronous REST submissions and asynchronous queue handoffs but performs no inference.
Worker (Pipeline Orchestration): CPU-bound execution engines that pull document IDs from a message queue, orchestrate the workflow, and manage I/O (downloading images, uploading JSON results). Workers utilize asynchronous coroutines to overlap I/O waits with task execution, enabling vertical concurrency within a pod.
Inference Service: A dedicated GPU-bound service exposing REST APIs for OCR models (e.g., DocTR) and proxying VLM capabilities (e.g., Claude Sonnet). This isolation allows GPU resources to be scaled independently of worker count.

2. Communication & Coordination

The system relies on message queues (Ingestion, Worker, Status) to coordinate document ownership, provide backpressure, and manage retries. This ensures:

Idempotency: Only one worker processes a document at a time.
Fault Isolation: Failures in one tier (e.g., Gateway) do not crash the entire control plane; in-flight processing continues.
Checkpointing: Processing status is updated in the database at each step, allowing resumption from failure points without reprocessing the entire document.

3. Pipeline Design

Documents flow through a configurable sequence of steps:

Hybrid Classification: A cost-accuracy tradeoff strategy using a local CLIP-KNN classifier (92% accuracy, $0 cost) as the primary filter. If confidence is low (<0.7), the system falls back to a VLM classifier (98% accuracy, higher cost). This reduces API costs by ~10x while maintaining 96% overall accuracy.
Auxiliary Metadata Extraction: Uses object detection (RF-DETR) or VLMs to extract dates, barcodes, or signatures.
OCR: Converts images to text with word-level bounding boxes using DocTR (GPU-accelerated).
Text Stitching: Concatenates per-page OCR output into a coherent document-level representation.
Structured Parsing: Sends stitched text to a VLM (Claude Sonnet) with a JSON schema to extract structured field-value pairs.

Key Contributions

Architectural Decomposition: The paper details the decision to separate inference from orchestration, enabling independent scaling of GPU resources based on demand rather than worker count.
Pipeline Engineering: A detailed walkthrough of a hybrid classification strategy and the flow from raw images to structured JSON, balancing cost and accuracy.
Operational Insights: Qualitative findings from batch profiling that challenge common assumptions about document AI bottlenecks.

Results & Findings

Through batch profiling of synthetic multi-page documents, the authors identified two critical, counter-intuitive findings:

OCR Dominates Latency: Contrary to the intuition that LLM parsing is the bottleneck, OCR consumes the majority of end-to-end wall-clock time. For an 8-page document, OCR accounts for roughly two-thirds of the processing time, while parsing (which runs once per document) is sublinear relative to page count.
Saturation Dynamics: System throughput saturates based on shared GPU-inference capacity, not the number of worker pods. Adding workers beyond the point where the Inference Service is saturated yields diminishing returns; requests queue behind the GPU layer rather than processing in parallel.

Cost Reduction: By implementing this architecture, the authors reduced the cost of processing documents from $0.01 per page to $0.001 per page while maintaining 96% accuracy.

Significance & Lessons Learned

The paper argues that bridging the gap between model research and production requires addressing system-level constraints that benchmarks ignore. Key lessons include:

Model Accuracy $\neq$ System Reliability: High-accuracy models still produce errors that require human review; systems need defense-in-depth strategies (e.g., hybrid classification) rather than relying solely on model performance.
Staggering vs. Batching: Utilizing different models for different pipeline stages (classification, OCR, parsing) allows for "staggering" of compute resources, improving utilization more effectively than simple batching.
Observability is Critical: Distinguishing between orchestration bottlenecks (queue depth) and inference bottlenecks (GPU saturation) is essential for debugging. A growing queue may indicate underprovisioned workers, but it may also signal an inference bottleneck or downstream API throttling.

The authors conclude that while end-to-end VLMs or retrieval-augmented approaches may evolve the field, the current practical solution for structured form processing requires a modular, microservice-based approach that treats OCR as the primary cost and latency driver. This architecture provides a concrete pattern for operationalizing document AI beyond the benchmark.