Content-Based Smart E-Mail Dispatcher Using Large Language Models

Technical Summary: Content-Based Smart E-Mail Dispatcher Using Large Language Models

Problem Statement
The paper addresses the inefficiency and cognitive burden associated with manually processing high volumes of email in large organizations, specifically within an engineering college context. While email remains ubiquitous, the manual perusal and forwarding of academic notifications and circulars from departmental inboxes to specific student WhatsApp groups is described as error-prone, time-consuming, and stressful for staff. Current manual workflows create bottlenecks where staff must read, interpret, and route diverse content to the correct batches (defined by academic year and program) without automation. Existing automated solutions often rely on simple keyword matching in subject lines or forwarding emails from specific accounts without analyzing the semantic content, failing to handle the nuance required for accurate routing.

Methodology
The proposed solution is a "Content-Based Smart E-Mail Dispatcher" (SEDS) that leverages Large Language Models (LLMs) and an agent-based framework to automate the classification and routing of emails. The system operates through a three-step process:

1. Data Capturing: The system ingests email messages and their attachments. During development, this data is read offline; in deployment, it connects directly to the mail inbox.
2. Content Analysis: This is the core computational step. The system employs an AI agent framework (utilizing the Langchain library) where agents query LLMs via APIs. The agents use prompt engineering to analyze the textual content of the email and its attachments. The prompts provide the email content, instructions, and context, enabling the LLM to determine the semantic relevance of the message.
3. Dispatching: Based on the LLM's analysis, the agent identifies the specific target WhatsApp groups (e.g., specific student batches or faculty groups) and automatically posts the message and attachments to those groups.

The system is designed to function without labeled datasets, relying instead on the LLM's inherent ability to understand context and perform zero-shot or few-shot classification. The dataset used for validation consisted of over 1,000 carefully selected emails related to students and faculty.

Key Contributions

• LLM-Driven Routing: The paper introduces a mechanism that moves beyond keyword-based filtering to semantic content analysis for routing emails to specific organizational groups.
• Agent-Based Architecture: The implementation utilizes a multi-agent framework where one agent identifies the email category and subsequent agents handle the specific routing actions to custom-built dispatcher tools.
• Dataset-Free Operation: The system is highlighted for not requiring extensive manual effort to create labeled training datasets, a common hurdle for traditional machine learning approaches in this domain.
• Handling Dynamic Structures: The approach addresses the challenge of dynamic group structures (e.g., new batches added annually) which often complicate static rule-based or clustering-based systems.

Results
The authors evaluated the system using four different LLM platforms: Gemini, ChatGPT, DeepSeek, and Perplexity AI. The performance metric was strict classification accuracy, defined as the model identifying all correct target groups with no extra or missing groups.

• Gemini achieved the highest accuracy at 93.5%.
• ChatGPT followed closely with 92%.
• Perplexity and DeepSeek scored 82.5% and 80%, respectively.

The analysis of misclassifications revealed that errors often occurred with circulars regarding "valuation and revaluation," where the model sometimes incorrectly recommended sharing faculty-only content with student groups. However, the system successfully mapped circulars mentioning batch series (even without specific years) to the correct targets.

Significance and Claims
The paper claims that this system represents a significant step forward in automating institutional email management. By integrating transformer-based models and unsupervised learning techniques, the system addresses challenges related to semantic understanding and unstructured data processing. The authors assert that the system:

• Enhances Productivity: It significantly reduces the manual workload and cognitive load on staff, allowing them to focus on high-priority tasks.
• Ensures Timeliness: It facilitates a smooth flow of information, ensuring students receive academic updates promptly via their preferred communication channels (WhatsApp).
• Reduces Errors: Automation minimizes the human errors associated with manual forwarding.
• Versatility: The system is capable of processing diverse formats, including PDFs, images, and Word documents.

The authors position this work as a "stepping stone" for future innovations, noting that future iterations will focus on summarizing email content before dispatching to provide concise notifications. They explicitly state that replacing the LLM with traditional machine learning or deep learning models would be challenging due to the need for labeled data and the difficulty in handling multi-label selection (multiple target groups) and dynamic group additions.