Technical Summary: Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

Problem Statement

While Large Language Models (LLMs) have demonstrated exceptional proficiency in standard Natural Language Processing (NLP) tasks, a significant gap remains in their application to practical data analysis scenarios involving unstructured text. Specifically, current LLMs struggle when required to process long sequences of unstructured documents, such as social media posts, to perform rigorous quantitative analysis. Existing benchmarks predominantly focus on structured data (e.g., CSV files, SQL queries) or single-document retrieval, failing to address the challenges of aggregating information across collections of short, noisy, and independent instances. Furthermore, prior research has highlighted LLM vulnerabilities regarding hallucinations and inadequate numerical and logical reasoning, yet the extent to which these limitations hinder comprehensive data analysis over large text corpora remains an open question.

Methodology

To empirically assess LLM capabilities in this domain, the authors introduce a question-based evaluation framework and a dedicated benchmark constructed from social media data.

Benchmark Construction

The dataset is built upon two established social media NLP benchmarks: TweetEval and SuperTweetEval. It comprises 470 manually curated questions with human-verified ground truths, derived from eight datasets covering diverse tasks:

• Single-Label Classification (SLC): Sentiment analysis, hate speech detection, offensive language detection.
• Multi-Label Classification (MLC): Emotion recognition, topic classification.
• Targeted Classification (TC): Stance detection, targeted sentiment analysis.

The questions are categorized into four logical reasoning types reflecting increasing complexity:

1. Existence: Detecting the presence/absence of specific labels (Binary Yes/No).
2. Count: Quantifying instances within the context (Numerical).
3. Comparison: Performing binary comparisons between label groups (Binary Yes/No).
4. Calculation: Deriving secondary metrics like percentages (Numerical/Proportional).

To evaluate robustness across data volumes, the authors generated subsets at seven scales: 10, 50, 100, 250, 500, 750, and 1,000 instances. Crucially, labels and metadata were stripped from the input to simulate a purely unstructured text analysis environment.

Experimental Setup

The study evaluated a diverse range of models:

• Closed-source: GPT-5.4-Mini, Gemini-3.1-Flash-Lite, Grok-4.1-Fast.
• Open-weights: Various sizes of LLaMA (3B to 70B) and Qwen (4B to 397B MoE).

Models were tested using a standardized prompting strategy that enforced strict output formats (Yes/No, Integer, or Percentage) without explanations. Performance was measured using Macro-F1 for binary tasks, Range-Normalized Root Mean Squared Error (RNRMSE) for percentages, and Normalized Root-Mean-Square Error (NRMSE) for numerical counts.

Key Results

1. Performance Dependency on Task Complexity

• Existence vs. Reasoning: Models perform well on simple existence questions (identifying if a label exists), with F1 scores often exceeding 0.88 on SLC data. However, performance drops significantly as tasks require comparison and numerical reasoning.
• Numerical Deficits: Closed-source models generally outperform open-weights models, particularly in percentage estimation and numerical regression. Notably, the entire LLaMA series struggled with percentage tasks, often performing below random baselines.
• Data Complexity: Performance degrades as data complexity increases. Targeted classification tasks (requiring context-aware binding to specific entities) yield the lowest performance across all models, followed by Multi-Label tasks, with Single-Label tasks being the most manageable.

2. The "500-Instance" Threshold

A critical finding is the identification of a performance degradation threshold at approximately 500 input instances.

• Open-weights Models: Exhibit severe performance collapse across all tasks when input size exceeds 500 instances.
• Closed-source Models: While more robust in semantic understanding, they also show a marked increase in numerical errors (RNRMSE/NRMSE) beyond this threshold, indicating a shared architectural bottleneck in aggregating quantitative data over large collections.

3. Prompting Strategies

The study compared "No Instruction," "Standard," and "One-Shot" prompting.

• Open-weights models benefited significantly from more complex prompts (Standard and One-Shot).
• Closed-source models showed mixed results; in numerical tasks, One-Shot prompting sometimes led to performance decreases compared to the Standard prompt.
• The Standard prompt emerged as the most balanced and robust strategy overall.

Key Contributions

1. A Specialized Benchmark: The introduction of a 470-question dataset specifically designed to evaluate LLMs on unstructured text aggregation, moving beyond structured data analysis benchmarks.
2. Empirical Evidence of Limitations: Systematic demonstration that LLM reliability in data analysis is heavily contingent on input scale and task complexity, with a specific focus on the failure of numerical reasoning in long-context scenarios.
3. Identification of Architectural Bottlenecks: The paper highlights that current LLMs, regardless of being open or closed-source, face significant challenges in maintaining precision when processing large volumes of independent text instances for quantitative tasks.

Significance and Claims

The authors claim that their findings highlight critical limitations in current LLM architectures for performing rigorous quantitative analysis over large text collections. While LLMs are proficient at semantic identification, their ability to perform accurate counting, comparison, and calculation diminishes as the scale of data and the complexity of the reasoning required increase.

The paper concludes that future work must prioritize the development of specialized architectures tailored for long-context synthesis and methods to ensure high-precision reasoning. The authors modestly note that their evaluation represents a "performance lower bound" as it excludes agentic workflows with external tools (e.g., code interpreters), suggesting that current native LLM capabilities alone are insufficient for reliable large-scale unstructured data analysis.