Technical Summary: TelcoAI

Problem Statement

The 3rd Generation Partnership Project (3GPP) produces the foundational technical specifications for global mobile telecommunications (3G, 4G, 5G, and beyond). These documents are characterized by extreme complexity, spanning hundreds of pages across multiple releases, and containing dense formatting, thousands of equations, tables, and interdependencies. While Large Language Models (LLMs) offer potential for processing this data, existing approaches face significant limitations:

• Structural Ignorance: Standard Retrieval-Augmented Generation (RAG) systems often use flat or generic chunking strategies that fail to preserve the deeply hierarchical nature of 3GPP documents, leading to the loss of crucial contextual cues.
• Multi-Modal Deficits: Current methods largely neglect the visual components of specifications, such as technical diagrams, tables, and schematics, which are vital for accurate interpretation.
• Reasoning Limitations: Most implementations rely on single-step retrieval patterns, lacking the agentic capabilities required for structured query decomposition, multi-hop planning, and cross-document information fusion.
• Evaluation Gaps: Existing benchmarks often focus on isolated, atomic queries that do not reflect the complex, multi-faceted information needs of real-world engineers.

Methodology: The TelcoAI Framework

TelcoAI is an agentic, multi-modal RAG system designed specifically to address the intricacies of 3GPP documentation. The architecture consists of two primary pipelines: Document Ingestion and Question Answering.

1. Document Ingestion

The ingestion pipeline processes Microsoft Word (.docx) specifications through four key stages:

• Metadata Extraction: Structured metadata (Release number, Series number, Specification number, and Version) is extracted and stored alongside document chunks to enable precise filtering during retrieval.
• Section-Aware Chunking: Unlike fixed-size chunking, TelcoAI employs a recursive, depth-first traversal of the document's hierarchical structure. Chunks are created based on section and subsection headings, preserving semantic coherence and the logical relationships between concepts.
• Image Tagging and Integration: Visual elements (images and tables) are processed using an LLM (Anthropic's Claude 3.5 Sonnet v2) to generate natural language descriptions. These descriptions are concatenated with the surrounding text chunks, along with unique element markers, enabling natural language matching for multi-modal queries while retaining access to the original visual elements.

2. Question Answering Pipeline

The QA pipeline operates through four stages:

• Pre-Retrieval (Planning & Reformulation): A planner LLM decomposes complex user queries into a sequence of standalone sub-queries and suggested follow-up questions. It also extracts metadata (e.g., specific releases or specification numbers) from the query to guide filtering.
• Context Retrieval: The system performs a hierarchical hybrid search, combining dense vector similarity (using Amazon Titan Text Embeddings v2) with sparse lexical matching (BM25). Retrieval begins with smaller sub-section chunks and expands to include parent sections to ensure comprehensive context.
• Post-Retrieval Processing: Retrieved chunks undergo metadata-based filtering to match query constraints, followed by re-ranking based on similarity scores. The top 5 chunks are selected for generation.
• Answer Generation: The system synthesizes information from multiple retrieved chunks and modalities. It fuses textual content with retrieved images and tables, then generates a structured response using an LLM. The prompt instructs the model to perform technical reasoning, cite specific sections, and integrate visual evidence.

Key Contributions

The paper outlines five primary contributions:

1. Agentic Multi-Stage RAG Architecture: A modular pipeline integrating hierarchical reasoning, query planning, and answer synthesis to mimic expert decision-making in navigating telecommunications documents.
2. Multi-Modal Document Understanding: A custom ingestion pipeline that fuses visual and textual content via metadata-aware alignment, enabling comprehensive interpretation of specifications containing diagrams and tables.
3. Section-Aware Chunking: A retrieval strategy that leverages document hierarchy to preserve semantic coherence, improving the fidelity of downstream answers.
4. Agentic Query Decomposition and Fusion: A planner that breaks down complex queries into manageable sub-queries and fuses answers across diverse content slices to capture nuanced technical insights.
5. Comprehensive Evaluation: Extensive experiments on curated and synthetic datasets tailored to the 3GPP domain, demonstrating superior performance across retrieval and reasoning tasks.

Results

The system was evaluated on multiple benchmarks, including Human-QA (expert-curated complex questions), Synthetic-QA, and the TSpec-LLM benchmark.

• Performance on Complex Queries (Human-QA): TelcoAI achieved a Recall of 87% and a Claim Recall of 83%, with a Faithfulness score of 92%. This represents a 16% improvement in Recall and a 13.7% improvement in Claim Recall over the strongest baseline (Chat3GPP with Claude 3.5 Sonnet v2).
• Ablation Studies: Incremental testing revealed that section-based chunking provided the initial significant boost (+14.3% Recall). The addition of multi-modal fusion in the generation stage yielded the highest final performance, contributing to a total 38.1% improvement in Recall over the Naive RAG baseline.
• Benchmark Accuracy (TSpec-LLM): On fact-based multiple-choice questions, TelcoAI achieved 93% overall accuracy, outperforming specialized baselines like Chat3GPP (92%) and Naive RAG (82%).
• Robustness: The system demonstrated consistent performance across different underlying LLMs (Claude 3.5, Llama 3.3, Mistral Large 2), with Recall scores varying by only 2-4%.
• Latency: The total latency for TelcoAI is 11.81 seconds, which is higher than Naive RAG (6.76s) but comparable to other specialized systems like Telco-RAG (9.91s). The authors note this trade-off is justified by the substantial gains in accuracy.

Significance and Claims

The paper claims that TelcoAI represents a significant step forward in creating domain-specialized, multi-modal language systems for high-stakes technical environments. By addressing the specific structural and visual challenges of 3GPP documentation, the system advances practical solutions for telecommunications research and engineering. The authors assert that their approach effectively reduces the cognitive load on engineers by providing accurate, context-aware, and verifiable responses to complex technical queries.

The paper maintains a modest stance regarding its limitations, acknowledging that the current implementation is tailored to .docx formats and single-turn queries. It identifies future work in supporting diverse file formats (PDFs, Markdown), enabling multi-turn conversational agents, and optimizing for inference latency and production integration.