MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

1. Problem Statement

Engineering design documents (e.g., rulebooks, technical standards) are inherently multimodal, containing dense text, complex tables, mathematical formulas, and critical illustrations (CAD drawings, stress-strain graphs). Existing Retrieval-Augmented Generation (RAG) systems struggle with these documents because:

• Text-Only Limitations: Traditional RAG systems (like those used in the original DesignQA benchmark) rely on text extraction and semantic similarity. This process often strips away visual context, layout hierarchy, and spatial relationships essential for understanding engineering rules.
• Ineffective Retrieval: Simple text-based retrieval fails to locate relevant information when the answer depends on interpreting a diagram or a table rather than just a keyword.
• Context Window Constraints: Providing the entire document (All-Rules approach) to a Large Language Model (LLM) ensures context but is computationally expensive (up to 25x more costly) and often exceeds token limits.
• Performance Gap: There is a significant accuracy gap between models given the full document context and those using standard RAG, particularly in tasks requiring visual reasoning.

2. Methodology: The MCERF Framework

The authors propose the Multimodal ColPali Enhanced Retrieval and Reasoning Framework (MCERF). Unlike previous approaches, MCERF treats document pages as visual inputs to preserve layout and structure, coupling this with adaptive reasoning strategies.

A. Multimodal Information Retriever (ColPali)

Instead of extracting text, MCERF uses ColPali, a vision-language model that:

• Treats each PDF page as a discrete visual input.
• Splits pages into smaller patches.
• Generates unified embeddings for these patches that capture both textual semantics and visual attributes.
• Uses MaxSim scoring (late interaction) to match query tokens against all encoded page regions, enabling fine-grained relevance calculation that preserves document hierarchy and visual patterns.

B. Reasoning Strategies (Variants)

The framework employs four distinct processing pipelines tailored to specific task complexities:

1. Hybrid Lookup (Variant A): Combines ColPali's semantic retrieval with a keyword-based search (BM25). An LLM extracts critical technical terms from the query, which are then used for precise lexical matching. Best for explicit rule extraction.
2. Self-Consistency (Variant B): Executes five independent retrieval-reasoning passes for a single query. A "blind" adjudicator LLM aggregates these candidates via consensus ranking to reduce hallucinations and stabilize responses, particularly for complex compilation tasks.
3. High-Reasoning Mode (Variant C): Utilizes a more capable reasoning model (GPT-5-mini with extended reasoning chains) for tasks requiring complex logical inference, such as spatial reasoning or multi-step compliance checks.
4. Vision-to-Text Fusion (Variant D): Converts visual inputs into structured text descriptions before reasoning. Images are split into overlapping quadrants, upscaled, and described by an image-to-text module. This reduces multimodal input complexity, allowing the reasoning model to perform arithmetic or logical operations on structured text rather than raw pixels.

C. Adaptive Routing

To optimize efficiency and accuracy, MCERF implements dynamic routing to select the best pipeline for each query:

• Single-Case Router (Router 1): A unified router that selects the best variant for an entire task category (e.g., all "Retrieval" questions use Hybrid Lookup) based on ensemble voting.
• Agent-Based Router (Router 2): A multi-agent system where a Supervisor LLM analyzes individual questions and routes them to specialized agents (DocumentAgent for text/rules, VisionAgent for images) in real-time.

3. Key Contributions

1. Modular Multimodal Interface: A model-agnostic framework separating the retriever, reasoner, and router, allowing for easy swapping of underlying models.
2. Layout Preservation: Demonstrates that maintaining document layout and visual structure during retrieval (via ColPali) is a dominant factor in QA accuracy, outperforming text-extraction methods.
3. Specialized Routing Patterns: Establishes that different engineering tasks require different strategies (e.g., Hybrid for rules, Vision-to-Text for dimensions) and provides a mechanism to automate this selection.
4. Open-Source Implementation: The framework and code are publicly available, including support for open-source reasoning backbones (e.g., Llama-3.2).

4. Experimental Results

The framework was evaluated on the DesignQA benchmark (derived from Formula SAE rules), which includes six task categories: Retrieval, Compilation, Definition, Presence, Dimension, and Functional Performance.

• Overall Performance: MCERF achieved an average accuracy of 0.79, representing a 41.1% relative improvement over the best baseline RAG model (0.56).
• Surpassing Full-Context Baselines: In most categories, MCERF surpassed the "All-Rules" baseline (where the model sees the entire 140-page document), proving that targeted multimodal retrieval can compensate for reduced context.
  • Retrieval: Improved from 0.19 (Baseline RAG) to 0.95 (Hybrid variant).
  • Dimension: Improved from 0.68 to 0.82 (Vision-to-Text variant).
  • Functional Performance: Achieved 0.94, matching the full-document baseline.
• Efficiency: The system avoids the high computational cost of feeding full documents to LLMs while maintaining or exceeding their accuracy.
• Router Effectiveness: The Single-Case Router successfully identified the optimal variant for each task, achieving the highest scores across the board.

5. Significance and Conclusion

This work addresses a critical bottleneck in engineering AI: the inability of current systems to effectively reason over multimodal technical documentation.

• Scalability: MCERF offers a scalable alternative to full-document ingestion, making it feasible for organizations with massive libraries of technical standards.
• Practical Application: The framework is directly applicable to intelligent design assistants, automated compliance checking, and technical documentation analysis in aerospace, automotive, and manufacturing sectors.
• Future Directions: The authors suggest that while ColPali is powerful, future work should focus on domain-adaptive fine-tuning of the vision encoder and developing more efficient retrieval methods for massive document collections (thousands of pages) without sacrificing accuracy.

In summary, MCERF demonstrates that vision-language retrieval combined with adaptive, modular reasoning is the key to unlocking accurate and efficient comprehension of complex engineering documents, moving beyond the limitations of text-only RAG systems.