Technical Summary: Neural Router: Semantic Content Matching for Agentic AI

1. Problem Statement

As autonomous AI agents operate across the computing continuum (cloud, edge, IoT), they must discover and consume information from diverse sources with varying data formats, vocabularies, and modalities. Traditional content-based publish/subscribe (pub/sub) systems rely on keyword matching or attribute-based filters, which fail to bridge the "semantic gap" when publishers and subscribers use different terminologies or modalities (e.g., raw sensor activations vs. natural language activity descriptions). While embedding-based semantic matching offers an intermediate solution, it lacks the abductive reasoning required when modalities do not align.

The paper addresses the design of a matching engine for a pub/sub broker that utilizes Large Language Models (LLMs) to evaluate whether a piece of content satisfies a natural-language interest description. The core challenge is to integrate LLMs into the matching loop while managing the trade-offs between semantic accuracy, latency, and computational cost, particularly given the constraints of context windows and the non-deterministic nature of LLM inference.

2. Methodology and Architecture

The authors propose the Neural Router, a content-based pub/sub broker where the LLM serves as the primary matching engine. The system is framed as an offline multi-label retrieval problem and employs a two-level architecture:

• Offline Subscription Path:

  • Embedding & Clustering: Free-text subscription descriptions are embedded (using Sentence-BERT) and clustered via $k$-means into semantically coherent groups.
  • Compression: Within each cluster, a CoverAndMerge algorithm (Algorithm 4) uses the LLM to identify subscriptions that are semantically subsumed by broader ones (covering) or can be combined (merging). This reduces the token footprint of the subscription set.
  • Backend Assignment: Clusters are assigned to specific LLM backends based on a Quality-of-Experience (QoE) score, which is a weighted scalarization of accuracy, cost, and latency.

• Online Event Path:

  • Routing: Incoming events are embedded and routed to candidate clusters based on cosine similarity to cluster centroids.
  • Batching & Matching: Events are accumulated into batches and packed into LLM prompts alongside the compressed subscription set of the target cluster. The LLM returns the top-$\kappa$ matching subscriptions for each event.

Key Algorithms

1. OptimizeSubscriptions: Prepares the matching table by embedding, clustering, and compressing subscriptions, then assigning backends.
2. MatchEvents: Routes events to clusters, batches them, and invokes the assigned LLM for semantic matching.
3. CoverAndMerge: An iterative LLM-assisted process that reduces the subscription set size while aiming to preserve recall.

Cost Model and Crossover Analysis

The paper derives a cost model predicting two critical crossover points:

1. Context-Window Crossover ($W_{cross}$): A theoretical threshold below which compression (reducing subscription tokens) increases the batch size of events per LLM invocation, thereby reducing total invocations and latency. Above this threshold (typical of modern cloud LLMs with large context windows), the overhead of clustering and multiple prompts outweighs the savings from compression.
2. Discrimination-Capacity Crossover ($|S|_{cross}$): An empirical threshold where the number of subscriptions ($|S|$) exceeds the LLM's ability to make accurate simultaneous binary relevance decisions. Beyond this point, matching accuracy collapses, independent of the context window size, due to attention budget constraints.

3. Experimental Setup

The evaluation spans three public datasets representing different domains and semantic difficulties:

• CardiffNLP (D1): Social media tweets (19 topics, low semantic gap).
• EUR-Lex (D2): EU legal documents (201 subject matters, moderate semantic gap, long context).
• CASAS (D3): Smart-home sensor logs (19 activities, high modality gap requiring abductive reasoning).

Backends: Six LLMs were evaluated:

• Open-weight: Qwen 2.5 (1.5B, 7B, 32B) and Mistral 7B.
• Closed-API: Claude 3 Haiku and Claude 3 Sonnet.

Baselines: Seven baselines included keyword matching (BM25, TF-IDF), embedding similarity (Sentence-BERT, GloVe, Word2Vec), pairwise neural scoring (Cross-Encoder), and zero-shot classification (DistilBART-MNLI).

Configurations: Seven ablation configurations (A0–A6) tested the impact of clustering, compression, reunification, and filtering.

4. Key Results

Cost-Accuracy Trade-offs

• Context-Window Crossover: On datasets with large context windows (cloud APIs, $W \ge 128K$), the Raw LLM (A0) configuration dominates. Clustering and compression (A1, A3, A4) increase the number of LLM invocations without improving accuracy, as the context window is not the binding constraint.
• Edge Regime: In constrained environments ($W \approx 4K$), compression (A4) reduces invocations. However, the paper observes an empty-prediction collapse: as the subscription set grows via duplication, the LLM's vocabulary narrows, and it increasingly returns empty matches, causing accuracy to drop significantly despite the compression.

Discrimination Capacity

• Model Dependency: Accuracy declines when $|S|$ exceeds a model-specific threshold ($|S|_{cross}$). This threshold is not a global scalar but depends on the joint factor of parameter count and training generation.
• Findings on D2 ($|S|=201$):
  • Smaller models (Qwen 1.5B, 7B) collapsed to near-zero accuracy.
  • Larger models (Qwen 32B) and newer generations (Mistral 7B) showed partial recovery but remained below pairwise baselines.
  • Only the frontier closed-API model (Claude Sonnet) significantly outperformed pairwise baselines.
• Conclusion: Backend selection (model choice) is the primary lever for performance; pipeline tuning (clustering/compression) is secondary.

QoE Backend Assignment

• A QoE-based assignment framework was tested against a Round-Robin baseline.
• On a 7B/32B gradient, Round-Robin achieved 83% of the accuracy gap closure of the larger tier at 0.7x the latency.
• The QoE-optimized approach did not consistently outperform Round-Robin due to calibration noise; the small calibration sample (10%) was insufficient to reliably distinguish backend performance at the cluster level. Separation was only observed under specific "cost-first" weight presets.

5. Significance and Claims

The paper claims the following contributions and significance:

• First LLM-in-the-Loop Pub/Sub: It is the first system to place an LLM directly inside the content-matching loop of a pub/sub broker, moving beyond using LLMs merely for orchestration or post-match notification generation.
• Two-Crossover Characterization: The paper provides a principled framework for understanding when LLM-based matching is viable. It identifies that discrimination capacity (the ability to reason over many simultaneous options) is often a tighter constraint than context window size.
• Practical Guidance:
  • For large context windows, simple raw matching (A0) is superior to complex pipelines.
  • For high subscription cardinality, model selection is more critical than configuration tuning.
  • Compression pipelines are only beneficial in constrained context-window regimes and are conditional on the LLM's ability to handle merged, compound descriptions without collapsing.
• Modality Bridging: The system successfully bridges the modality gap (e.g., sensor data to natural language) where embedding-based filters fail by construction.

The authors remain modest regarding future work, noting that the current evaluation is offline, single-broker, and single-tenant. They explicitly defer issues of online operation, distributed federation, and multi-modal extensions to future studies. They also acknowledge that benchmark results may be upper bounds due to potential data contamination (memorization of public datasets by LLMs).