PDF Retrieval Augmented Question Answering

1. Problem Statement

Existing Retrieval-Augmented Generation (RAG) systems for Question Answering (QA) are predominantly designed for unstructured text. However, PDF documents contain rich, multi-modal data including text, images, vector diagrams, graphs, and tables. Current systems often fail to:

• Accurately extract and interpret non-textual elements (e.g., charts, tables).
• Handle complex, multi-faceted questions that require synthesizing information across different data types within a single document.
• Filter out noise (headers/footers) that degrades retrieval accuracy.
• Effectively integrate diverse data sources to prevent hallucinations and improve reasoning.

2. Methodology: PIER-QA System

The authors propose PIER-QA (PDF Integrated Enhanced Retrieval Question Answering), an end-to-end system designed to process, retrieve, and generate answers from multi-modal PDF content. The architecture consists of three primary stages:

A. PDF Preprocessing

To ensure high-quality input for retrieval, the system performs rigorous cleaning and formatting:

• Header/Footer Removal: Uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to identify and remove repeating header and footer elements based on their spatial coordinates (bounding boxes) across pages. This is applied in sliding windows (e.g., every 10 pages) to handle document variations.
• PDF to Markdown Conversion: Utilizes Marker, a specialized tool, to convert cleaned PDFs into Markdown. This step extracts text, images, and tables while preserving layout.
• Image Captioning: Images are processed using LLaVA (Large Language and Vision Assistant) to generate descriptive captions. This converts visual data into text, allowing the LLM to "read" images and enabling text-based indexing.
• Table Reformatting: Tables are converted from verbose Markdown syntax into a compact dictionary format to optimize storage and retrieval efficiency.

B. Embedding and Storage

• Chunking: The processed Markdown documents are segmented into 1,000-character chunks.
• Embedding: Chunks (containing text, image captions, and dictionary-formatted tables) are embedded into high-dimensional vector spaces using the GTE-large model.
• Indexing: Vectors are stored in ElasticSearch. To enhance retrieval precision, the system employs RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), which creates a semantic tree structure for hierarchical indexing and clustering.

C. Question Answering & Fine-tuning

• RAG-Aware Fine-tuning: The authors fine-tuned Llama3-70B-Instruct using LoRA (Low-Rank Adaptation). The training data was synthetically generated by creating question-context-answer pairs from the specific document domain and Markdown structure. The model was trained to understand the specific format of retrieved chunks (including image IDs and table dictionaries).
• Inference Pipeline:
  1. User query is embedded and used to retrieve the top-10 relevant chunks.
  2. Chunks are injected into the prompt for the LLM.
  3. The LLM generates an answer. If the answer references an image or table, it outputs a specific ID (e.g., [image_1.png]).
  4. Post-processing: The system retrieves the corresponding image or converts the table dictionary back to Markdown for display to the user.

3. Key Contributions

• Multi-Modal RAG Framework: A comprehensive system that goes beyond text-only retrieval to integrate images, diagrams, and tables into the RAG pipeline.
• Advanced Preprocessing: The novel application of DBSCAN for automated header/footer removal and the use of LLaVA for image captioning to bridge the modality gap.
• Domain-Specific Fine-tuning: Demonstrated that fine-tuning an open-source LLM (Llama3-70B) on the specific document format and retrieval context significantly improves performance, closing the gap with proprietary models like GPT-4o.
• Structured Data Handling: A method for compressing tables into dictionary formats for efficient storage and retrieval.

4. Experimental Results

The system was evaluated on a dataset of 100 question-answer pairs derived from 8 private industrial PDFs.

• Comparison with Baseline:
  • PIER-QA significantly outperformed a baseline system (which only converted PDFs to plain text without preprocessing or multi-modal retrieval).
  • Similarity Score: PIER-QA with GPT-4o achieved 0.8837 vs. Baseline's 0.8647.
  • Accuracy (@0.95 threshold): PIER-QA achieved 0.191 vs. Baseline's 0.1111, showing a marked improvement in high-precision answers.
• LLM Agent Performance:
  • GPT-4o achieved the highest similarity score (0.8837).
  • RAG-Llama3-70B (fine-tuned) achieved the highest accuracy at the strictest threshold (0.2135 at @0.95), outperforming the base Llama3-70B-Instruct and trailing GPT-4o closely despite being a 4-bit quantized open-source model.
• Non-Textual Retrieval:
  • Image Retrieval Accuracy: 65.66%.
  • Table Retrieval Accuracy: 48.38%.
  • These results confirm the system's ability to correctly identify and retrieve non-textual elements without relying on expensive multimodal LLMs for the entire generation process.

5. Significance and Future Work

• Significance: This work pushes the boundaries of RAG by proving that complex, multi-modal PDFs can be effectively queried using a text-centric LLM if the non-textual data is properly preprocessed (captioned/structured) and the model is domain-adapted. It offers a cost-effective alternative to proprietary multimodal models by leveraging open-source LLMs and specialized preprocessing.
• Limitations: The DBSCAN algorithm may struggle with highly variable PDF layouts, potentially removing essential content. The Marker tool may occasionally fail with complex table structures. The system currently retrieves for every query, which may introduce latency.
• Future Directions: Improving robustness against diverse PDF formats, optimizing retrieval to reduce unnecessary steps, and further refining the integration of complex vector graphics.

In conclusion, PIER-QA demonstrates that by combining advanced preprocessing, semantic indexing, and domain-specific fine-tuning, RAG systems can effectively handle the complexity of real-world multi-modal PDF documents.