From PDF to RAG-Ready: Evaluating Document Conversion Frameworks for Domain-Specific Question Answering

This paper systematically evaluates four open-source PDF-to-Markdown conversion frameworks and their associated preprocessing pipelines on a Portuguese administrative document corpus, finding that metadata enrichment and hierarchy-aware chunking strategies significantly outweigh the choice of conversion tool in determining RAG system accuracy, with the best automated approach achieving 94.1% performance compared to a 97.1% manual baseline.

The Gist

Imagine you have a massive library of old, dusty, and sometimes messy paper documents (PDFs) from the Portuguese Army. You want to build a super-smart robot librarian (an AI) that can answer any question about these documents instantly. This is called a RAG system (Retrieval-Augmented Generation).

The big question the researchers asked was: "Does it matter how we turn those messy paper documents into digital text for the robot to read?"

Many people assume the "brain" of the robot (the AI model) is the most important part. This paper argues that the preparation of the food (the documents) is actually more important than the chef. If you feed the robot garbage, it will give you garbage answers, no matter how smart the chef is.

Here is the breakdown of their experiment using simple analogies:

1. The Problem: The "PDF" Puzzle

PDFs are like frozen sculptures. They were designed to look good on a screen or when printed, not to be read by a computer.

• The Issue: When you try to copy-paste text from a PDF, the computer often gets confused. It might mix up columns in a table, lose the "bold" headings, or turn the Portuguese letter "ç" (as in caça, meaning "hunting") into "caca" (which means... well, something much less polite).
• The Goal: The researchers wanted to find the best "sculptor" (software tool) to melt these frozen PDFs down into clean, readable text (Markdown) so the AI could understand them.

2. The Contest: Four "Sculptors"

They tested four different open-source tools to see which one did the best job of cleaning up the documents:

• PDFLoader: The "Lazy Sweeper." It just grabs whatever text it sees without caring about structure.
• MinerU: The "OCR Specialist." It tries to read the text like a human eye scanning a page.
• DeepSeek OCR: A high-tech "Vision AI" that looks at the page like a picture.
• Docling: The "Architect." It uses special models to understand the layout, tables, and headings.

3. The Experiment: 19 Different Recipes

They didn't just use the tools; they tried 19 different "recipes" (pipelines). They mixed in:

• Cleaning: Removing HTML code or fixing math formulas.
• Rebuilding: Trying to figure out what is a "Chapter Title" and what is just "Body Text." They tried doing this by looking at font sizes (easy, like seeing a big sign) vs. asking the AI to guess (hard, because the AI can get confused).
• Chunking: Cutting the long documents into bite-sized pieces. They tried cutting them randomly vs. cutting them by logical sections (like cutting a book by chapters, not by random sentences).
• Metadata: Adding "sticky notes" to the chunks so the AI knows, "Hey, this paragraph is from Chapter 3, Section 2."

4. The Results: The "Garbage In, Garbage Out" Rule

The results were surprising and clear:

• The Winner: Docling combined with smart cutting (keeping chapters together) and adding image descriptions won the race. It got 94.1% of the answers right.
• The Loser: DeepSeek OCR got only 71.2% right.
• The "Lazy" Baseline: Even the "Lazy Sweeper" (PDFLoader) got 86.9% right, which was surprisingly high, but still far from the best.
• The "Gold Standard": When humans manually cleaned the documents, they got 97.1%.

The Big Takeaway:
The difference between the worst setup and the best setup was 23 percentage points. That is a huge gap!

• Analogy: It's like the difference between giving a student a textbook with torn pages, missing chapters, and typos (DeepSeek) versus giving them a pristine, well-organized textbook with a table of contents (Docling). The student's intelligence (the AI) didn't change, but their performance skyrocketed because the material was better.

5. The "Graph" Detour: A Fancy Map That Didn't Help

The researchers also tried building a Knowledge Graph (a giant web of connections between people, places, and ideas) to help the AI.

• The Idea: "If we map out how everything connects, the AI will be smarter!"
• The Reality: It actually made things worse (dropping to 82%).
• Why? The map was messy and full of duplicates. It was like trying to navigate a city with a map that has 20,000 names for the same street. The researchers concluded that for now, a clean, well-organized book (basic RAG) is better than a messy, over-complicated map (GraphRAG) unless you have a very specific plan for how to build it.

6. The "Font" vs. "AI" Debate

They tried to rebuild the document structure (headings) in two ways:

1. Font-based: "If the text is big and bold, it's a title." (Simple, reliable).
2. AI-based: "Let the AI guess what is a title based on the context." (Complex, prone to errors).

• Result: The simple Font-based method won every time. The AI got confused by the complex legal text, while the simple rule of "Big Text = Title" worked perfectly.

Summary: What Should You Do?

If you are building an AI system to read documents:

1. Don't obsess over the AI model first. A smart AI fed bad data will fail.
2. Focus on the "Kitchen Prep." Spend your time and energy cleaning the documents, fixing the tables, and organizing the chapters.
3. Use the right tools. Tools like Docling that understand document structure are better than simple text extractors.
4. Keep it simple. Don't try to build complex knowledge graphs unless you really know what you are doing. A clean, well-structured text file is often the secret sauce.

In short: The quality of your answer depends entirely on the quality of your input. Clean your data, and the AI will do the rest.

Technical Summary

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems are widely used to ground Large Language Models (LLMs) in domain-specific knowledge, yet most research focuses on retrieval algorithms, embedding models, or the LLM itself. There is a critical gap in understanding how upstream document preprocessing—specifically the conversion of raw PDFs into machine-readable text—affects downstream question-answering (QA) accuracy.

PDFs are designed for visual rendering, not structural extraction. Converting them often results in:

• Loss of document hierarchy (headings, sections).
• Misinterpretation of tables and merged cells.
• Corruption of special characters and diacritics (e.g., Portuguese "ç" becoming "c").
• Distorted reading order in multi-column layouts.

The authors argue that "garbage in, garbage out" applies strictly to RAG: if the conversion introduces errors, they propagate through retrieval and generation, causing hallucinations or factual inaccuracies. No prior study had systematically evaluated PDF conversion frameworks based on their impact on RAG QA accuracy rather than just parsing metrics (like edit distance).

2. Methodology

Corpus and Benchmark

• Dataset: 36 Portuguese administrative documents from the Portuguese Army's Personnel Command (1,706 pages, ~492k words).
• Characteristics: Highly heterogeneous, containing 94.4% tabular content, 44.4% form fields, and 4,198 images.
• Evaluation: A manually curated benchmark of 50 questions covering table extraction, hierarchy-dependent queries, and image-based content.
• Scoring: "LLM-as-judge" (using gpt-4o-mini) averaged over 10 runs to mitigate stochastic variance.

System Architecture

The authors built a modular pipeline inspired by the Medallion Architecture:

• Bronze: Raw PDFs.
• Silver: Intermediate artifacts (raw conversion output).
• Gold: RAG-ready Markdown with metadata.
• Key Feature: The pipeline allows swapping conversion frameworks via a single configuration parameter without code changes.

Evaluated Frameworks & Configurations

The study tested 19 distinct pipeline configurations varying across four dimensions:

1. Conversion Framework:
   • Docling: Modular pipeline with specialized models.
   • MinerU: OCR-based (local pipeline and HTTP-client VLM).
   • DeepSeek OCR: Vision-Language Model approach.
   • PDFLoader (LangChain): Naïve baseline (no preprocessing).
   • Hand-made Markdown: Upper baseline (manually curated).
   • (Note: Marker was excluded due to privacy concerns with cloud processing and poor local results.)
2. Transformations: HTML table cleaning, LaTeX cleaning, Font-based hierarchy rebuilding (HR-F), LLM-based hierarchy rebuilding (HR-LLM), and VLM-based image descriptions.
3. Splitting Strategy: Recursive, Markdown Recursive, and Hierarchical Recursive (with breadcrumb metadata).
4. Retrieval Depth (K): Tested K=50, 20, and 5.

GraphRAG Exploration

An exploratory GraphRAG implementation was built using LangChain's LLMGraphTransformer to extract entities and relationships into a Neo4j graph, followed by semantic deduplication.

3. Key Results

Overall Accuracy Rankings

• Best Automated Result: Docling with hierarchical splitting and image descriptions achieved 94.1% accuracy.
• Upper Baseline: Hand-made Markdowns achieved 97.1%.
• Lower Baseline: Naïve PDFLoader achieved 86.9%.
• Worst Results: DeepSeek OCR (71.2%) and MinerU Pipeline without cleaning (74.7%).
• Gap: The difference between the worst and best automated configurations was nearly 23 percentage points, driven entirely by data preparation choices.

Key Findings on Factors

1. Splitting & Metadata > Framework Choice:

   • Switching from "Recursive" to "Hierarchical Recursive" splitting (with breadcrumb metadata) on Docling improved accuracy by 3.8 points (89.4% → 93.2%).
   • This gain was larger than the difference between different conversion frameworks using the same splitting strategy.
   • Insight: Preserving logical document structure during chunking is more critical than the raw text extraction tool.

2. Cleaning Transformations:

   • HTML Table Cleaning was the single most impactful transformation for MinerU (+5.4 points), as the tool outputs HTML tables that standard splitters mishandle.
   • Font-based Hierarchy Rebuilding (HR-F) consistently outperformed LLM-based Hierarchy Rebuilding (HR-LLM) by 3.5–4.6 points. The deterministic analysis of PDF fonts proved more reliable than LLM inference for structured legal documents.

3. Retrieval Depth (K):

   • High-quality pipelines (Hand-made) at K=5 matched the performance of low-quality pipelines (PDFLoader) at K=50 (both ~86.9%).
   • This suggests that high-quality preprocessing reduces the need for aggressive retrieval (large K).

4. GraphRAG Failure:

   • The basic GraphRAG implementation scored 82%, significantly underperforming the basic RAG (94.1%).
   • Entity deduplication further dropped scores to 81%.
   • Reasoning: The graph was shallow and sparse, and the lack of a domain-specific ontology led to noise and redundancy that hindered retrieval.

4. Key Contributions

1. First RAG-Centric Benchmark: The study is the first to evaluate PDF conversion frameworks based on downstream QA accuracy rather than parsing metrics, revealing a disconnect between "clean text" and "retrievable context."
2. Quantification of Preprocessing Impact: Demonstrated that data preparation quality is the dominant factor in RAG performance, outweighing the choice of LLM or embedding model in this context.
3. Practical Pipeline Architecture: Proposed a modular, configuration-driven ETL pipeline (Bronze/Silver/Gold) that enables reproducible experimentation and easy framework swapping.
4. Negative Results on GraphRAG: Provided evidence that naive, ontology-free GraphRAG construction can degrade performance compared to well-structured vector RAG, challenging the current hype around GraphRAG for general document retrieval.
5. Non-English Insights: Highlighted specific failure modes in Portuguese (e.g., "ç" misrecognition) that are often missed in English-centric benchmarks.

5. Significance and Implications

• For Practitioners: The paper argues that before optimizing retrieval algorithms or prompt engineering, teams must invest in data preparation. A well-configured pipeline (e.g., Docling + Hierarchical Splitting) can achieve near-human curation levels (94.1% vs. 97.1%) without manual labor.
• Strategy Shift: The focus should shift from "which LLM is best" to "how do we structure the document chunks?" The study proves that hierarchy-aware chunking and metadata enrichment are the new bottlenecks in RAG systems.
• GraphRAG Caution: The findings suggest that GraphRAG is not a silver bullet; without careful ontology design and graph densification, it adds complexity and cost without improving accuracy over standard vector RAG.
• Language Sensitivity: The study underscores the necessity of selecting tools that support specific language diacritics and layouts, as standard English-optimized tools may fail catastrophically in other languages.

In conclusion, the paper establishes that data quality is the primary driver of RAG success, and that simple, deterministic preprocessing strategies often outperform complex, AI-heavy alternatives when dealing with structured administrative documents.