💻 computer science

Beyond Keywords: Intent-Aware Message Routing for Gmail and Slack Using Large Language Models

This paper presents the design, implementation, and preliminary evaluation of an intelligent message routing system for Gmail and Slack that leverages the LLaMA 3.1 model via Groq to perform semantic intent classification, enabling users to define routing rules in plain English and significantly outperforming traditional keyword-based filters in a feasibility study.

Original authors: Ijegwa Acheme, Emmanuella Amukwuo, Ayodeji Makinde, Shangbum Faveren, Chukwuka Obionwu

Published 2026-10-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ijegwa Acheme, Emmanuella Amukwuo, Ayodeji Makinde, Shangbum Faveren, Chukwuka Obionwu

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern workplace, communication has fractured into a dozen different streams. A single professional might juggle emails, instant messages, and project updates across multiple platforms, creating a constant, low-level hum of incoming information. For decades, the tools designed to manage this flow have relied on a simple, rigid logic: if a message contains a specific word, do a specific thing. This keyword-based approach works well for obvious cases, but it struggles with the nuance of human language. A person asking for help with a bill might write "I was charged twice," "my invoice is wrong," or simply "payment issue." A system looking only for the word "invoice" would miss the first two, leaving the sender waiting and the recipient overwhelmed by a cluttered inbox. The challenge for computer scientists has long been to build systems that understand the intent behind the words rather than just the words themselves, a task that requires machines to grasp context and meaning in the same way humans do.

A team of researchers has now built a prototype system that attempts to solve this problem by using a new generation of artificial intelligence to read and sort messages for small businesses. Instead of relying on rigid lists of keywords, their platform uses a large language model—a type of artificial intelligence trained on vast amounts of text to understand how humans communicate—to interpret the meaning of incoming emails and chat messages. The system, designed to work with Gmail and Slack, allows a business owner to write routing rules in plain English, such as "if someone asks about billing, send this to the finance team." The computer then reads the incoming message, figures out what the person actually wants, and automatically forwards it to the right place, sends a reply, or sets it aside. This approach aims to replace the brittle, keyword-dependent filters of the past with a more flexible, intelligent assistant that can handle the messy reality of daily business communication.

The researchers constructed this system as a complete software application that connects directly to a user's email and chat accounts. Behind the scenes, a background process checks for new messages every five minutes. When a new message arrives, the system does not just scan it for specific terms; it sends the content to a powerful artificial intelligence model along with the user's plain-English instructions. The model analyzes the message to determine its purpose, such as whether it is a billing inquiry, a request for a meeting, or a general question. Based on this understanding, the system decides what to do. If the message clearly matches a rule and the AI is confident, it acts immediately by forwarding the message or sending an automated response. If the message is vague, the AI is unsure, or the topic is critical, the system pauses and flags the message for a human to review. This "abstain and escalate" strategy ensures that the computer never guesses when it is not sure, preventing the kind of errors that could annoy customers or delay important work.

To test whether this idea actually works, the researchers created a small set of fourteen test messages covering four common business topics: billing, scheduling, partnerships, and general inquiries. They wrote these messages carefully to avoid using the exact words found in the system's rules, forcing the computer to rely on understanding rather than simple word matching. For example, if the rule was written for the word "invoice," the test messages used phrases like "my card was charged twice" or "I need help with my bill." The results showed that the intelligent system correctly identified the intent of twelve out of the fourteen messages, routing them to the right category. In contrast, a simulated system using traditional keyword matching, similar to the filters found in standard email programs, only correctly identified five of the fourteen messages. The two messages the intelligent system missed were not sent to the wrong place; instead, the system recognized they were difficult to categorize and correctly flagged them for a human to handle.

The study highlights a significant shift in how automation can work for small businesses, particularly in regions where resources are limited and teams are small. The researchers found that by using a pre-trained artificial intelligence model, they could create a system that understands language without needing to collect thousands of examples or hire experts to teach it. The system is designed to be accessible, allowing non-technical users to define their own rules in everyday language. While the test was small and the messages were written by the researchers themselves rather than drawn from real-world chaos, the results suggest that this approach is feasible. The system successfully demonstrated that it could interpret the meaning of a message and make a routing decision that a simple keyword filter would have missed. The authors emphasize that this is a preliminary demonstration of the concept, not a final solution, and that the system's true value will be determined by how well it performs with the diverse, unpolished, and often messy communications that real businesses handle every day.

Technical Summary: Beyond Keywords: Intent-Aware Message Routing for Gmail and Slack Using Large Language Models

1. Problem Statement

The paper addresses the persistent issue of message misrouting in corporate environments, particularly within small and medium-sized enterprises (SMEs) in Nigeria and Africa. While tools like Gmail filters, Slack Workflows, Zapier, and Make exist, they rely heavily on keyword matching and rigid condition logic. These systems fail to capture the semantic intent of natural language, leading to frequent misclassifications when users phrase requests differently (e.g., "I can't access my account" vs. "my login is broken").

Furthermore, existing sophisticated automation tools often require significant technical expertise to configure and impose recurring costs in foreign currencies (USD), creating barriers for resource-constrained African SMEs. Current academic solutions for automated ticket classification often require supervised fine-tuning on large, labeled datasets, which is infeasible for organizations lacking dedicated machine learning engineering capacity. The gap identified is the absence of an accessible, affordable platform that combines Large Language Model (LLM) semantic classification with a no-code, plain-English rule interface for multi-channel routing.

2. Methodology and System Architecture

System Design

The authors designed and implemented a full-stack intelligent message routing platform using a client-server architecture:

Frontend: Next.js (React, TypeScript) for a web-based dashboard.
Backend: Express.js (TypeScript) handling authentication, rule management, and API orchestration.
Database: MongoDB (via Mongoose) for storing user accounts, routing rules, and activity logs.
Integrations:
- Gmail & Slack: Connected via OAuth 2.0 and their respective APIs (Google Gmail API, Slack Web API).
- LLM Engine: Utilizes the LLaMA 3.1 (8B) model accessed via the Groq API for high-speed inference.

Core Workflow

Polling Service: A background Node.js process polls connected Gmail and Slack accounts every five minutes ( $T_p$ ) for new messages.
Classification Engine: Upon retrieving a message, the system constructs a structured prompt containing the message metadata (sender, subject, truncated body up to 400 chars), the user's active plain-English rules, and a few semantically varied example phrasings per category. This prompt is sent to the LLM.
Decision Logic: The LLM returns a structured JSON object containing:
- Matched rule index ( $i^*$ )
- Urgency level ( $g$ : low, medium, high)
- Confidence score ( $c$ )
- requiresUserAction flag ( $h$ )
Routing Policy (Abstain-and-Escalate): The system applies a conservative decision rule $d(M)$ $d (M)$ :
- Automatic Action: Executed only if a rule is matched ( $i^* \neq \emptyset$ ), the model does not flag for review ( $h = \text{false}$ ), and urgency is not high.
- Escalation: If no rule matches, the model flags for review, or urgency is high, the message is routed to a human-review queue rather than being acted upon speculatively.
Auditability: Every event is logged to the Activity collection, recording the classification outcome and action taken without storing the full message body.

Implementation Details

Prompting Strategy: The system employs example-augmented prompting rather than strict zero-shot classification. It includes small numbers of example phrasings per rule category to encourage intent-based matching over literal keyword overlap.
Configuration: The model temperature was set to 0.1 to minimize output variance. No task-specific fine-tuning or gradient updates were performed; the pre-trained LLaMA 3.1 was used as-is.
Development: The system was built iteratively over four cycles, culminating in functional testing (34 tests) and integration testing.

3. Key Contributions

The paper's primary contributions are:

System Design: A documented architecture for embedding LLM-based semantic classification into multi-channel business tools (Gmail/Slack) without requiring fine-tuning infrastructure.
No-Code Interface: A mechanism allowing non-technical users to define routing logic in plain English, bridging the gap between complex LLM capabilities and SME usability.
Abstain-and-Escalate Mechanism: A specific decision rule that prioritizes safety by escalating ambiguous or high-urgency messages to human reviewers, preventing speculative automated actions.
Contextual Relevance: A design tailored for the economic and technical constraints of Nigerian and African SMEs, offering an alternative to expensive, subscription-based enterprise automation.

4. Results and Evaluation

The evaluation was modest, focusing on functional correctness and a preliminary feasibility check on a small, author-constructed dataset.

Functional Testing: Achieved a 100% pass rate across 34 tests covering authentication, CRUD operations, integrations, and logging.
Classification Feasibility:
- Dataset: 14 messages across four categories (Billing, Scheduling, Partnership, General Inquiry). Messages were deliberately phrased to avoid exact keyword matches with the rules.
- LLM Performance: Correctly matched 12 of 14 messages (85.7%). The two unmatched messages were escalated to human review (False Negatives), not misrouted.
- Baseline Comparison: A simulated keyword-matching baseline (mimicking standard Gmail filters) matched only 5 of 14 messages (35.7%).
- Metrics: Precision was 1.00 (no misroutings), Recall was 0.857, and F1 score was 0.923.
Error Analysis: The two failures occurred in the "Scheduling" category where the phrasing was less tied to the specific rule conditions (e.g., "find a time to speak" vs. "schedule a call"). This highlights the system's sensitivity to rule specificity.

Limitations Acknowledged by the Authors:

The test set was small (14 messages), constructed by the authors, and lacked real-world noise (typos, code-switching, Nigerian Pidgin).
Only one model configuration and one run were evaluated.
The keyword baseline was simple; stronger baselines (e.g., TF-IDF, embedding similarity) were not tested.
Statistical significance is limited due to the small sample size and overlapping confidence intervals between the LLM and baseline.

5. Significance and Claims

The paper explicitly frames its significance as a technical feasibility demonstration and a design contribution, rather than a definitive proof of superior accuracy over all existing methods.

Feasibility: The study demonstrates that LLMs can effectively classify intent using example-augmented prompting without the need for labeled training data or fine-tuning, making semantic routing accessible to organizations without ML engineering resources.
Safety: The "abstain-and-escalate" policy is highlighted as a critical design feature that ensures the system does not cause disruption through incorrect automated routing, a common risk in business automation.
Accessibility: The work provides a blueprint for affordable, accessible automation for African SMEs, addressing the cost and complexity barriers of current enterprise solutions.

The authors conclude that while the results are consistent with the feasibility of semantic matching, they do not establish that this approach is universally more accurate than keyword matching or other baselines in a production environment. Future work is identified as necessary to validate the system with real SME users, larger datasets, and diverse linguistic inputs.