LLM-Driven Online Aggregation for Unstructured Text Analytics

The paper introduces OLLA, an LLM-driven online aggregation framework that accelerates unstructured text analytics by incrementally processing data and employing semantic stratified sampling to achieve high accuracy with significantly reduced latency compared to full-batch processing.

The Gist

Imagine you are a detective trying to solve a mystery by reading thousands of handwritten letters. Your goal is to answer questions like, "How many letters mention 'money'?" or "What is the average sentiment of these letters?"

The Old Way: The Exhaustive Librarian

In the past, if you wanted to analyze these letters using a computer, you had to hire a very smart but incredibly slow librarian (the LLM). This librarian had to read every single letter, one by one, from start to finish, before they could give you a single answer.

If you had 10,000 letters, the librarian would sit there for hours, reading every page. By the time they finished, you might have forgotten why you asked the question in the first place. This is the problem with current AI tools: they are too smart but too slow for real-time answers.

The New Way: OLLA (The Smart Scout)

The authors of this paper, OLLA, propose a new strategy. Instead of waiting for the librarian to read everything, they act like a smart scout who gives you a running report while the work is still happening.

Here is how OLLA works, broken down into simple steps:

1. The "Speed Reading" Scan (Embedding)

Before the slow librarian reads the letters, OLLA uses a "speed reader" (an Embedding Model) to glance at the letters. It doesn't read the words; it just looks at the "vibe" or "shape" of the text.

• Analogy: Imagine sorting a pile of mixed-up mail into envelopes based on the color of the stamp or the shape of the handwriting, without reading the content yet.

2. The "Smart Grouping" (Semantic Stratified Sampling)

Instead of picking letters randomly (which might mean you pick 50 letters that all say "Hello" and miss the ones about "Money"), OLLA groups the letters into clusters based on their "vibe."

• Analogy: Think of a music festival. Instead of asking every single person in the crowd what their favorite song is (which takes forever), you divide the crowd into sections: the "Rock" section, the "Jazz" section, and the "Pop" section. You then pick a few people from each section to ask. This ensures you get a fair picture of the whole crowd much faster.

3. The "Running Tally" (Online Aggregation)

As the slow librarian reads the letters in these small, smart groups, OLLA starts giving you answers immediately.

• Analogy: Imagine you are watching a live sports game. You don't wait until the final whistle to know the score. You see the score update every time a goal is scored. OLLA gives you a "live score" of your data analysis.
• It tells you: "Right now, based on the 100 letters I've read, 60% are about money. I'm 95% sure the final answer will be between 58% and 62%."

4. The "Self-Correcting" Loop (Adaptive Adjustment)

This is the magic trick. If OLLA realizes one of its groups is messy (e.g., the "Rock" section actually has a lot of "Jazz" fans mixed in), it instantly reorganizes the groups and focuses its attention there.

• Analogy: It's like a GPS that realizes you took a wrong turn. Instead of waiting until you reach the destination to tell you, it immediately recalculates the route and says, "Hey, let's try this new path; it will get us there faster."

Why is this a Big Deal?

The paper tested this system on real-world data (like product reviews and news articles). Here is what they found:

• Speed: OLLA can give you an answer that is 99% accurate using only 4% of the time it would take to read everything.
• The "Good Enough" Factor: In the real world, you often don't need 100% perfection. You just need to know if a product is generally good or bad right now. OLLA lets you stop the analysis early once you are confident enough, saving massive amounts of time and money.
• Versatility: It works whether you are counting things, averaging numbers, or sorting text into categories.

The Bottom Line

OLLA is like upgrading from a slow, methodical accountant who waits until the end of the year to give you a report, to a live dashboard that updates your financial status every second. It uses AI to understand text, but it uses smart math to make sure you don't have to wait hours for the answer.

In short: It turns "Wait 2 hours for the answer" into "Here is a very good answer in 5 minutes, and it gets better the longer you wait."

Technical Summary

Here is a detailed technical summary of the paper "LLM-Driven Online Aggregation for Unstructured Text Analytics" (OLLA).

1. Problem Definition

The paper addresses the latency bottleneck in analyzing large-scale unstructured text (e.g., reviews, logs, documents) using Large Language Models (LLMs).

• The Bottleneck: While LLMs excel at semantic understanding and extracting structured fields from text, they are significantly slower than traditional relational database operations. Processing a dataset row-by-row with an LLM can take minutes or hours, whereas standard SQL queries take seconds.
• The Limitation of Current Approaches: Existing systems (e.g., LOTUS, UQE) integrate LLMs into SQL but typically rely on batch processing. Users must wait for the entire dataset to be processed before seeing any results, which is impractical for interactive analytics and real-time decision-making.
• The Goal: To enable interactive, low-latency analytics on unstructured text by providing progressively refined approximate results (Online Aggregation) rather than waiting for full batch completion.

2. Methodology: The OLLA Framework

The authors propose OLLA (Online Large Language model Aggregator), a framework that combines LLM-driven text transformation with online aggregation and a novel Semantic Stratified Sampling strategy.

Core Workflow

1. Unstructured-to-Structured Stream: Raw text is processed by an LLM to extract structured fields, creating a continuous data stream rather than a static batch.
2. Online Aggregation: The system incrementally aggregates this stream, providing running estimates with confidence intervals (e.g., "Average price is $50 ± $2 with 95% confidence") that tighten as more data is processed.
3. Semantic Stratified Sampling: To ensure rapid convergence to the ground truth, OLLA does not sample randomly. Instead, it uses a three-step adaptive process:
   • Embedding: Text is converted into high-dimensional vectors using an embedding model (e.g., SentenceTransformers).
   • Clustering: Vectors are clustered (e.g., K-means) to form initial strata based on semantic similarity.
   • Adaptive Adjustment: Since initial clusters may not perfectly align with the LLM's eventual classification, OLLA employs an iterative Sampling-Recording-Adjustment loop:
     • Sampling: Samples are drawn from strata using Neyman allocation (balancing stratum size and variance).
     • Recording: The LLM processes samples, and stratum statistics (category frequency, heterogeneity) are updated.
     • Adjustment: If a stratum is found to be heterogeneous (containing mixed categories), it is split or merged with other strata to improve purity. This dynamic refinement ensures the sampling distribution aligns with the true data distribution.

Query Types Supported

OLLA supports three types of LLM-integrated queries:

• SELECT: Aggregation on unstructured columns (e.g., AVG(LLM(extract_price))).
• WHERE: Filtering based on unstructured text (e.g., WHERE LLM(is_positive) = TRUE).
• GROUP BY: Grouping by unstructured categories (e.g., GROUP BY LLM(sentiment)).

3. Key Contributions

1. OLLA Framework: The first system to integrate LLMs with online aggregation principles, enabling users to obtain progressively refined insights in real-time without waiting for full data processing.
2. Semantic Stratified Sampling: A novel mechanism that converts unstructured text into vector spaces, clusters them, and dynamically adjusts strata based on LLM inference results. This significantly improves convergence speed compared to random sampling or static stratification.
3. Prototype Implementation & Evaluation: A working prototype built on modern stack components (vLLM, Kafka, Spark Streaming, Faiss) and extensive evaluation on diverse real-world datasets (reviews, news, resumes).

4. Experimental Results

The system was evaluated on datasets including BBC News, Amazon Product Reviews, Movie Reviews, and Chinese Resumes.

• Accuracy & Convergence:
  • OLLA achieves a 1% absolute error bound compared to the labeled ground truth using less than 4% of the full-data processing time.
  • In filtering scenarios (WHERE clause), it reaches a 5% error bound in as little as 2.6% of the total time (a 38× speedup).
  • In GROUP BY scenarios, it achieves a 5.6× speedup (reaching 5% error in 17.7% of the time).
• Comparison with Baselines:
  • vs. Random Sampling: Semantic sampling accumulates valid samples much faster, leading to tighter confidence intervals early in the process.
  • vs. Static Stratification (UQE): The adaptive adjustment mechanism in OLLA significantly outperforms static clustering, which often fails to align with semantic labels initially.
  • vs. Batch Systems (LOTUS): OLLA provides immediate, useful insights, whereas batch systems offer zero output until completion.
• Scalability: The system scales effectively with multiple LLM serving instances, showing near-linear speedups (up to 2.03× with 4 instances) for certain workloads.

5. Significance

• Paradigm Shift: OLLA shifts the paradigm of LLM-based text analytics from "batch-only" to "interactive streaming," making large-scale text analysis feasible for production environments where low latency is critical.
• Cost Efficiency: By converging to accurate results using a tiny fraction of the data (often <5%), OLLA drastically reduces the number of LLM API calls required, lowering inference costs and latency.
• Generalizability: The framework is designed to handle diverse query patterns (SELECT, WHERE, GROUP BY) and works with various unstructured data types, offering a robust solution for the growing volume of text data in modern databases.

In conclusion, OLLA successfully bridges the gap between the semantic power of LLMs and the speed requirements of interactive data analytics, offering a practical path forward for real-time unstructured text processing.