Developing an AI Assistant for Knowledge Management and Workforce Training in State DOTs

This paper proposes a multi-agent Retrieval-Augmented Generation framework that integrates open-weight large language models and vision-language models to enhance knowledge management and workforce training in state Departments of Transportation by enabling context-aware, evidence-grounded responses from both textual and visual technical documentation.

The Gist

Imagine a massive, bustling library where the books are the secret manuals, safety guides, and engineering reports of a State Department of Transportation (DOT). This library holds the collective brainpower of thousands of engineers who have built and fixed roads for decades.

The Problem: The "Lost Knowledge" Crisis
Here's the trouble: The library is huge, messy, and getting messier.

1. The Retirement Cliff: The senior engineers who know the "secret tricks" of fixing potholes or managing traffic are retiring. When they leave, they take their experience with them, like a librarian walking out the door and taking the only map to the treasure chest.
2. The Needle in the Haystack: New engineers are drowning in paperwork. They need to find a specific rule about "crack sealing" right now to fix a road, but searching through thousands of PDFs and charts is like trying to find a specific grain of sand on a beach.
3. The "Hallucination" Risk: If you ask a standard AI (like a basic chatbot) a question, it might make up an answer because it's guessing based on general knowledge, not the specific, strict rules of the DOT. In road safety, a wrong guess can be dangerous.

The Solution: The "Super-Team" AI Assistant
The authors of this paper built a new kind of AI assistant. Instead of one robot trying to do everything, they created a team of specialized robots (called "Multi-Agent RAG") that work together like a high-end research firm.

Here is how this team works, using a simple analogy:

1. The Librarian (The Retriever Agent)

When you ask a question (e.g., "How long does a chip seal last?"), the first robot doesn't just guess. It acts like a super-fast librarian. It dives into the library, scans the shelves, and pulls out the top three most relevant documents.

• The Twist: This librarian is smart enough to read pictures and charts, not just text. If a chart shows a graph of road wear over time, this robot can "read" the graph and understand what it means, turning the visual data into words so the team can use it.

2. The Researcher (The Generator Agent)

Once the librarian hands over the documents, the Researcher robot reads them carefully. It doesn't make things up. It says, "Based only on these specific documents, here is the answer." It writes a clear, easy-to-understand response, citing exactly which page or chart the info came from.

3. The Quality Control Inspector (The Evaluator Agent)

Before the answer goes to you, the Inspector robot checks it. It asks:

• "Is this answer clear?"
• "Did we actually find the right documents?"
• "Is it safe to use this advice?"
  If the answer is weak or the documents weren't a perfect match, the Inspector sends it back.

4. The Detective (The Query Refiner Agent)

If the Inspector says, "This answer is missing something," the Detective steps in. It realizes the original question might have been too vague. It rephrases the question to be more specific (e.g., changing "How long does it last?" to "How long does a chip seal last in hot, rainy climates?") and sends the new question back to the Librarian to try again.

Why This is a Big Deal
The team tested this system with over 500 real-world DOT documents and 100 tricky questions about road maintenance.

• The Result: The system was incredibly accurate. It found the right information 94.4% of the time within its top three guesses.
• The Comparison: A standard "one-shot" AI (which asks once and gives an answer) often misses the mark. This "team" approach, which checks its work and asks better questions, is much more reliable.

The Bottom Line
This paper proposes a way to save the "institutional memory" of transportation agencies. It turns a chaotic library of dusty manuals and confusing charts into a friendly, 24/7 expert assistant. It ensures that when a new engineer has a problem, they don't have to guess or wait for a senior mentor; they can get a precise, fact-checked answer instantly, keeping our roads safer and our workforce smarter.

In short: It's like giving every road worker a personal team of expert librarians, researchers, and detectives who never forget a fact and never leave the building.

Technical Summary

Here is a detailed technical summary of the paper "Developing an AI Assistant for Knowledge Management and Workforce Training in State DOTs."

1. Problem Statement

State Departments of Transportation (DOTs) face critical challenges in Knowledge Management (KM) and workforce training:

• Knowledge Loss: The retirement of senior engineers leads to the loss of tacit institutional knowledge.
• Information Overload & Fragmentation: Agencies maintain vast volumes of unstructured or semi-structured technical documents (manuals, reports, guidelines). Engineers struggle to locate specific, relevant information quickly to solve field problems.
• Limitations of Traditional KM: Static documentation and classroom training are inefficient, leading to steep learning curves for new hires and fragmented knowledge transfer.
• AI Limitations: While Large Language Models (LLMs) offer potential, they suffer from hallucinations, outdated training data, and a lack of domain-specific grounding. Furthermore, standard Retrieval-Augmented Generation (RAG) systems often rely on single-pass retrieval, lacking iterative refinement and quality control.
• Visual Data Neglect: Technical documents contain critical information in figures (charts, graphs, performance trends). Standard RAG systems typically ignore these visual elements because they cannot be embedded or retrieved without conversion to text.

2. Methodology

The authors propose a Multi-Agent Retrieval-Augmented Generation (RAG) framework designed specifically for State DOT environments. The system integrates text and visual data processing through four main components:

A. Figure Preprocessing (Vision-Language Integration)

To address the "visual data gap," the system treats figures as retrievable knowledge units:

• Extraction: Figures are extracted from PDF documents.
• Semantic Conversion: An open-weight Vision-Language Model (VLM) (specifically Qwen3-VL-2B-Instruct) processes each figure.
• Prompting Strategy: The VLM is prompted to generate:
  1. A caption for context.
  2. A detailed description including visualization type, axis variables, units, quantitative trends, and qualitative interpretations.
• Indexing: These generated text descriptions are embedded alongside standard text chunks in the vector database, allowing the system to retrieve figure-based evidence during question answering.

B. Multi-Agent Architecture

Unlike single-pass RAG, the system employs a collaborative workflow using the AutoGen framework with specialized agents:

1. User Proxy Agent: Receives and relays user queries.
2. Retriever Agent: Fetches relevant context (text and figure-descriptions) from the vector database (ChromaDB) using cosine similarity.
3. Generator Agent: Formulates the response based strictly on retrieved context, ensuring beginner-friendly tone and mandatory citations. It avoids external knowledge to prevent hallucinations.
4. Evaluator Agent: Assesses the generated response for clarity, relevance, and completeness.
   • If Satisfactory: The response is finalized.
   • If Unsatisfactory: Feedback is sent to the next agent.
5. Query Refiner Agent: Analyzes evaluator feedback, refines the original query for better specificity, and triggers a re-retrieval loop (up to $k$ iterations).

C. Core Models

• Embedding Model: all-MiniLM-L6-v2 (384 dimensions).
• Vision-Language Model: Qwen3-VL-2B-Instruct (for figure description).
• Response Generation Model: Qwen3-4B-Instruct-2507 (for final answer generation).

3. Key Contributions

1. Multi-Agent RAG Framework: Moves beyond single-pass retrieval by introducing a closed-loop system with evaluation and query refinement, significantly improving retrieval accuracy and answer quality.
2. Figure-Based Knowledge Retrieval: A novel approach to converting technical figures into semantic text representations, enabling the RAG system to index and retrieve visual evidence (e.g., performance trends, comparative analyses) alongside text.
3. Domain-Specific Implementation: Tailored for State DOTs, focusing on pavement maintenance and management, with a strict grounding mechanism to minimize hallucinations in high-stakes engineering contexts.
4. Transparent Performance Metrics: The system reports granular metrics (Precision/Recall) and operates under low-cost deployment constraints (standard hardware, no specialized servers).

4. Experimental Setup & Results

• Dataset: A curated corpus of 521 technical and research documents from multiple State DOTs (including Texas, California, etc.), spanning 30 years (1990s–2024). Topics cover pavement management, service life, costs, and rehabilitation strategies.
• Evaluation: The system was tested on 100 domain-specific queries regarding pavement maintenance.
• Metrics: Precision@3 and Recall@3.

Performance Results:

• Recall@3: 94.4% (Nearly all relevant evidence was found within the top 3 retrieved items).
• Precision@3: ~1.0 (All top 3 retrieved items were relevant).
• Comparison:
  • Single-Pass RAG Baseline: Achieved only 58% Recall@5.
  • Multi-Agent RAG: Achieved 94% Recall@3.
  • Analysis: The multi-agent approach significantly outperformed the baseline because the Query Refiner Agent successfully reformulated queries when initial retrieval failed, recovering documents missed by single-pass systems.

5. Significance and Future Work

• Significance: The study demonstrates that Agentic RAG combined with Vision-Language processing is highly effective for complex, document-heavy engineering domains. It provides a scalable solution for preserving institutional knowledge and accelerating workforce training in public infrastructure agencies.
• Limitations:
  • Scope: Currently limited to pavement engineering; generalizability to traffic safety or bridge management is untested.
  • Automation: Figure processing currently requires manual extraction/screenshotting; full automation is a future goal.
• Future Work: Expanding the dataset to include broader transportation domains (traffic operations, safety analysis) and automating the figure ingestion pipeline to improve scalability.

In conclusion, this paper presents a robust, evidence-based AI assistant that bridges the gap between vast technical documentation and practical engineering decision-making, offering a blueprint for modernizing knowledge management in critical public infrastructure sectors.