💬 NLP

Moneyball with LLMs: Analyzing Tabular Summarization in Sports Narratives

This paper introduces SPORTABSET, a diagnostic benchmark for long-context tabular summarization in sports narratives, which reveals that while decomposition strategies improve accuracy by mitigating multi-entity interference, current LLMs remain fundamentally limited by poor multi-entity memory and high sensitivity to surface-level cues.

Original authors: Ritam Upadhyay, Naman Ahuja, Rishabh Baral, Aparna Garimella, Vivek Gupta

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ritam Upadhyay, Naman Ahuja, Rishabh Baral, Aparna Garimella, Vivek Gupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a sports commentator describing a game in real-time. You say things like, "Smith hits a four!" or "Johnson misses a shot." Now, imagine you have a robot assistant whose job is to listen to your entire commentary and instantly fill out a perfect scorecard for every single player.

That is the challenge this paper tackles. The researchers wanted to see if modern AI (Large Language Models or LLMs) can act like a super-fast, super-accurate sports statistician. They called their project "Moneyball with LLMs" because, just like the movie Moneyball used data to find hidden value in baseball, this paper uses data to find hidden weaknesses in how AI "thinks" about sports.

Here is the breakdown of their findings using simple analogies:

1. The Problem: The "Overwhelmed Librarian"

The researchers found that when you ask an AI to listen to a whole game (which is a lot of text) and update a scorecard, it often gets confused. It's like asking a librarian to organize a library while the books are being thrown at them one by one.

The Issue: The AI tends to forget who did what, mix up players, or hallucinate (make up) stats that never happened.
The "Cheat Code" Discovery: They found that the AI was actually "cheating." In cricket, the commentary often says, "Smith is out, he scored 50 runs." The AI didn't actually calculate the runs ball-by-ball; it just grabbed that final number and wrote it down. When the researchers hid those summary sentences, the AI's performance crashed. It realized the AI wasn't reasoning; it was just extracting easy clues.

2. The Solution: Breaking the Task Down (The "Assembly Line")

To fix this, they tried different ways to help the AI. They found that breaking the massive task into smaller chunks worked best. Think of it like this:

The Old Way (Monolithic): Asking one person to do the whole game's scorecard in one go. They get tired and make mistakes.
The New Way (Decomposition):
- Divide & Generate: Splitting the game into 8 small chunks and having the AI do one chunk at a time, then stitching the results together.
- Entity-COT (The "One-Player" Focus): Instead of asking the AI to track everyone, they asked it to track just Player A, then just Player B, and so on. It's like having a dedicated accountant for every single player rather than one accountant trying to do the whole company's taxes at once.
- Text-Tuple-Table: This was the most accurate method. The AI first wrote down tiny, atomic facts like "(Player A, Scored, 4)" for every single event, and then a computer program (not the AI) added them up. It's like having the AI write down every single receipt, and then a calculator doing the math.

3. The "Robustness" Test: The "Identity Crisis"

The researchers wanted to see if the AI was actually smart or just memorizing names. They ran three tricky tests:

The "Fake Names" Test: They swapped real player names (like "Kohli") with fake ones (like "Lionel Cristiano").
- Result: In basketball, the AI got confused and failed. In cricket, it surprisingly did okay. This suggests the AI relies heavily on recognizing specific names in basketball but uses logic in cricket.
The "Anonymization" Test: They replaced all names with "Player 1," "Player 2."
- Result: The AI struggled massively. It started making up stats (hallucinating) or forgetting players entirely. This proved the AI wasn't truly understanding the game; it was just recognizing the names.
The "Role Confusion" Test: In cricket, they messed up the sentence structure so it wasn't clear who was bowling and who was batting.
- Result: The AI got the roles completely mixed up. It thought the bowler was the batter. This showed the AI relies on the structure of the sentence, not the actual logic of the sport.

4. The Big Takeaway: "Memory is the Bottleneck"

The paper concludes that the biggest problem isn't that the AI can't do math. The problem is memory.

The Analogy: Imagine trying to remember a conversation with 20 different people happening at once. If you forget who said what 10 minutes ago, your summary will be wrong.
The Finding: The AI is great at doing the math right now, but it forgets the "state" of the game as it goes on. It loses track of the "scoreboard" in its head.

Summary

This paper is a diagnostic report. It tells us that while AI is getting better at reading sports commentary, it's still a bit like a student who memorizes the answers to a practice test but fails when the questions are slightly different.

To make AI reliable for things like sports stats, finance, or medical records, we can't just ask it to "read and summarize." We need to build systems that force the AI to break the task down, track every single entity (person) individually, and verify its work, because right now, the AI's "short-term memory" for long stories is still quite fragile.

1. Problem Statement

The paper addresses the challenge of Long-Context Tabular Summarization, specifically the task of generating structured statistical tables (e.g., scorecards) from evolving, unstructured narrative text (e.g., sports commentary).

While Large Language Models (LLMs) have improved at structured generation, they struggle with long-context, multi-entity reasoning. The core difficulties identified are:

State Maintenance: Tracking the evolving state of multiple interacting entities (e.g., batsmen and bowlers in cricket, or 24+ players in basketball) over thousands of tokens.
Numerical Fidelity: Performing accurate arithmetic and aggregation (summing runs, counting balls) without hallucination or drift.
Reliance on Surface Cues: Models often rely on explicit summary cues (e.g., "run out" summaries) rather than performing genuine ball-by-ball or play-by-play reasoning.
Evaluation Gaps: Existing benchmarks (like LiveSum) often evaluate single updates to fixed schemas, failing to capture the complexity of dynamic, multi-entity state tracking in long narratives.

2. Methodology

A. The Benchmark: SPORTABSET

The authors introduce SPORTABSET, a diagnostic benchmark designed to evaluate LLMs on dynamic text-to-table generation across two complementary sports domains:

Cricket (ODI):
- Data: 498 innings (2006–2025) from Cricinfo, comprising ~6,500 ball-by-ball commentaries.
- Challenge: Requires maintaining two interdependent tables (Batsman and Bowler) with distinct logic. Batsmen have continuous roles, while bowlers have episodic roles. Rules are complex (e.g., handling wides, no-balls, and byes).
- Ground Truth: Programmatically derived from commentary using regex parsers, validated against official scorecards (99.7% match).
Basketball (NBA):
- Data: 998 games from the 2025 season via ESPN.
- Challenge: Requires generating a single, wide table covering ~24 players. Updates are dense and overlapping, lacking the clear role separation found in cricket.
- Ground Truth: Official final box scores scraped from ESPN, as commentary alone cannot deterministically reconstruct all stats.

Perturbations: To test robustness, the authors apply three types of perturbations:

Anonymization: Replacing player names with generic placeholders (e.g., "Player1").
OOD Entity Substitution: Replacing names with plausible but synthetic names from other sports (e.g., "Lionel Cristiano").
Entity Entanglement: Paraphrasing commentary to obscure standard role patterns (e.g., breaking the "Bowler to Batsman" syntax), forcing the model to infer roles from context.

B. Experimental Setup

Models Evaluated: GPT-4.1, Gemini-2.5-Flash, Llama-3.3-70B-Instruct, and Qwen-2.5-7B-Instruct.
Baselines & Strategies:
1. Zero-Shot Chain-of-Thought (ZS-CoT): Standard prompting with step-by-step reasoning.
2. Divide & Generate: Splitting the commentary into $n$ chunks ( $n \in \{2, 4, 8\}$ ), generating intermediate tables, and merging them.
3. Entity CoT: Extracting the list of entities first, then generating a separate table row for each entity via parallel LLM calls.
4. Text-Tuple-Table (T3): Extracting atomic tuples (Entity, Attribute, Value) for every event, then using a rule-based compiler to aggregate them into the final table.
Metrics:
- Accuracy: Cell-level exact match.
- Numerical Fidelity: RMSE and Symmetric Mean Absolute Percentage Error (SMAPE).
- Robustness: Hallucination-Omission Index (HOI) = Over-Count% - Under-Count%.

3. Key Contributions

SPORTABSET Benchmark: A large-scale, multi-sport dataset with programmatically verified ground truth and a suite of label-invariant perturbations to diagnose model failures.
Systematic Decomposition Analysis: A comprehensive comparison showing that while decomposition improves performance, the gains stem primarily from mitigating multi-entity interference rather than improving local arithmetic.
Diagnostic Insights: Identification of multi-entity memory as the primary bottleneck. The paper reveals that models are brittle to surface-level cues (names, syntax) and often fail via structured hallucinations (role confusion) or omissions when these cues are removed.

4. Key Results

A. Impact of Decomposition

General Improvement: All decomposition strategies significantly outperformed the monolithic ZS-CoT baseline.
- Text-Tuple-Table (T3) achieved the highest aggregate performance (+20.6 pp accuracy gain, 62.8% RMSE reduction).
- Entity CoT showed comparable accuracy gains (+15.1 pp) but with the most significant reduction in numerical drift (RMSE), particularly in basketball.
Source of Gains: The improvements are driven by reducing the cognitive load of tracking multiple entities simultaneously. When the model focuses on a single entity (Entity CoT) or small chunks, it performs better, suggesting the bottleneck is state maintenance, not calculation capability.

B. Model-Specific Behaviors

Gemini-2.5-Flash: Showed the most consistent gains across all methods, benefiting strongly from decomposition and symbolic reasoning.
GPT-4.1: Exhibited unstable behavior with "Divide & Generate" (gains were inconsistent) but performed reliably with Entity CoT.
Llama-3.3: Showed monotonic improvements with decomposition, especially in basketball.
Reliability Issues: T3 suffered from "degenerate repetition" (models outputting thousands of identical tuples) and cumulative hallucinations, requiring multiple regeneration passes, which increased computational costs significantly.

C. Robustness & Perturbation Analysis

Surface Cue Reliance: When explicit dismissal summaries were masked, accuracy dropped sharply (e.g., GPT-4.1 dropped from 89% to 61%), proving models often extract rather than reason.
Anonymization: Caused catastrophic failure in basketball (accuracy dropped ~27pp) and increased hallucinations. Models struggled to track entities without lexical anchors.
Entity Entanglement: In cricket, breaking the "Bowler to Batsman" syntax caused role confusion, where models swapped batsmen and bowlers in the output tables (high Jaccard overlap between predicted and ground-truth roles).
Memorization: Chronological splits (Pre-2025 vs. Post-2025) revealed that some models (like GPT-4.1) rely on memorized patterns for older data, while others (Gemini) showed improved state tracking on newer data.

5. Significance and Conclusion

The paper argues that current LLMs lack native mechanisms for maintaining persistent, structured state over long contexts.

The Bottleneck: The primary limitation is not arithmetic ability but multi-entity memory. Models fail when they must track the state of many interacting entities simultaneously without external scaffolding.
Trade-offs: While decomposition strategies (like T3) achieve high accuracy, they incur high computational costs and reliability risks (e.g., infinite loops, hallucination).
Future Directions: The authors call for foundational models that can natively maintain structured, entity-consistent states, rather than relying on brittle prompting tricks or external symbolic pipelines. SPORTABSET serves as a diagnostic tool to drive this development, emphasizing that robustness and faithfulness are as critical as raw accuracy for real-world deployment.

In summary, the paper demonstrates that while LLMs can summarize sports narratives, they currently do so by exploiting surface regularities and short-term memory. True long-context tabular summarization requires architectural advances in state management to handle the complexity of multi-entity interactions reliably.