Retrieval-Augmented Reasoning for Chartered Accountancy

1. Problem Statement

The paper addresses the limitations of Large Language Models (LLMs) in the specialized field of Indian Chartered Accountancy (CA). While LLMs are increasingly adopted in finance, they face critical challenges in regulated sectors:

• Hallucinations and Accuracy: Generalist models (e.g., GPT-4o) struggle with complex, multi-step reasoning required for legal regulations (Income Tax Act, ICAI standards) and GST computations, often achieving low accuracy (13–20%) on domain-specific benchmarks.
• Resource Constraints: State-of-the-art models require massive computational resources, high inference costs, and cloud dependency, raising privacy concerns and making them unsuitable for resource-constrained environments.
• Document Parsing Issues: Standard OCR fails to handle the intricate layouts of tax documents (multi-column tables, mathematical symbols), leading to data loss during retrieval.
• Lack of Grounding: Models often fail to base their answers on original source documents, which is a strict requirement for financial and legal compliance.

2. Methodology: CA-ThinkFlow

The authors propose CA-ThinkFlow, a parameter-efficient Retrieval-Augmented Generation (RAG) framework designed to operate on a 14B parameter model while matching the performance of much larger proprietary models.

A. Core Architecture

• Base Model: Utilizes 14B-DeepSeek-R1 (specifically DeepSeek-R1-Distill-Qwen-14B) in a 4-bit quantized (Q4_K_M) format. This model is selected for its built-in Chain-of-Thought (CoT) reasoning capabilities, allowing it to mimic natural cognitive thinking and handle multi-step problems.
• Retrieval System:
  • Document Extraction: Uses Docling, a layout-aware extractor (leveraging TableFormer and DocLayNet) to preserve document structure, tables, and mathematical symbols from financial texts, overcoming the limitations of standard OCR.
  • Data Source: A curated database of Indian financial knowledge, including ICAI open-source materials, tax regulations, and textbooks.
  • Chunking: Text is segmented into 1000-character chunks with a 200-character overlap, optimized for embedding continuity and retrieval precision.
  • Embedding: Text chunks are encoded using the Qwen-Embedding-0.6B model for multilingual support and efficiency.
  • Indexing: Embeddings are stored in a FAISS index for L2-based similarity search.
• Inference Workflow:
  1. The system receives a user query.
  2. It retrieves the top-k relevant context from the vector store.
  3. The retrieved context is injected directly into a standardized prompt template (based on the CA-Ben benchmark).
  4. The 14B-DeepSeek-R1 model processes the query and context, utilizing its internal CoT reasoning to synthesize the final answer.

B. Evaluation Setup

• Benchmark: Evaluated on the CA-Ben 2 benchmark using a zero-shot setting (no fine-tuning on the test data), simulating a cold-start scenario.
• Metrics: Performance measured via F1 scores across 14 subjects (Foundation, Intermediate, and Final levels) and the Scholastic Reliability Coefficient (SRC).
• Hardware: Tested on a workstation with 2x Intel Xeon CPUs, 65GB RAM, and 2x NVIDIA GTX 1080 Ti GPUs.

3. Key Contributions

1. Parameter-Efficient High Performance: Demonstrated that a quantized 14B model, when combined with a robust RAG pipeline, can achieve performance levels comparable to massive proprietary models (e.g., GPT-4o, Claude 3.5 Sonnet, LLaMA-3.1-405B) in a specialized domain.
2. Layout-Aware Document Processing: Integrated Docling to solve the specific challenge of parsing complex financial documents, ensuring that tabular data and mathematical symbols are retained during the retrieval process.
3. Synergy of RAG and CoT: Showed that a "basic" RAG approach (retrieving context and injecting it into the prompt) is sufficient when paired with a model possessing strong internal Chain-of-Thought reasoning, eliminating the need for complex fine-tuning or external reasoning engines.
4. Domain-Specific Benchmarking: Provided a granular analysis of model performance across 14 distinct CA subjects, highlighting specific strengths and weaknesses in areas like Taxation vs. Business Mathematics.

4. Results

• Overall Accuracy: CA-ThinkFlow achieved a Scholastic Reliability Coefficient (SRC) of 68.75%, matching the performance of GPT-4o and Claude 3.5 Sonnet, and significantly outperforming the base 14B-DeepSeek-R1 model (46.87%) and other open-source models like LLaMA-3.3-70B (59.38%).
• Level-wise Performance:
  • Foundation Level: Achieved 80.39% accuracy, outperforming all other models.
  • Final Level: Achieved 48.63% accuracy, surpassing most models except GPT-4o and Claude 3.5 Sonnet.
• Subject-Specific Insights:
  • Strengths: The model excelled in Business Mathematics, Business Economics, and Advanced Financial Management.
  • Weaknesses (Systemic Bottlenecks): The system struggled with Taxation (I3) and Indirect Tax Laws (FN5), failing to clear the "Systemic Bottleneck" (defined as a subject where no model exceeds 40% accuracy). This indicates that even with RAG, the base model lacks the deep regulatory knowledge required for complex tax logic without further fine-tuning.
• Efficiency: The framework operates effectively on consumer-grade hardware (GTX 1080 Ti), proving that high-accuracy financial AI does not necessarily require massive GPU clusters.

5. Significance and Future Work

• Significance: The paper validates that parameter efficiency and retrieval augmentation can bridge the gap between small, private, cost-effective models and massive proprietary LLMs in high-stakes financial domains. It offers a viable path for deploying compliant, private AI in the Indian CA sector without reliance on cloud-based, expensive APIs.
• Limitations: The system currently fails to fully resolve complex regulatory reasoning in Taxation, suggesting that retrieval alone is insufficient for highly nuanced legal logic.
• Future Directions:
  • Developing advanced retrieval pipelines for multi-chunk synthesis and dynamic context scaling.
  • Implementing domain-specific Supervised Fine-Tuning (SFT) on Indian financial statutes to deepen the model's regulatory knowledge before retrieval.
  • Integrating real-time tax regulation updates and multilingual support to create a deployable, expert-level financial AI system.