💻 computer science

Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls

This paper addresses the challenges of extracting Key Performance Indicators (KPIs) from unstructured earnings call transcripts by introducing new benchmarks, demonstrating the limitations of models trained on structured SEC filings, and proposing a human-verified LLM-based system that achieves 79.7% precision for open-ended information extraction.

Original authors: Rasmus T. Aavang, Rasmus Tjalk-Bøggild, Alexandre Iolov, Giovanni Rizzi, Mike Zhang, Johannes Bjerva

Published 2026-05-06

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rasmus T. Aavang, Rasmus Tjalk-Bøggild, Alexandre Iolov, Giovanni Rizzi, Mike Zhang, Johannes Bjerva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Formal Report" vs. The "Chatty Phone Call"

Imagine you are trying to understand how a company is doing. You have two main sources of information:

The SEC Filing (The Formal Report): This is like a filled-out tax form. It is strict, follows a rigid template, uses specific boxes, and has no room for creativity. If you ask a computer to read this, it's like reading a spreadsheet; it's easy to find the numbers because they are always in the same spot.
The Earnings Call (The Chatty Phone Call): This is like a live radio interview with the company's CEO and CFO. They are talking to investors, answering questions, telling stories, and sometimes correcting mistakes on the fly. There are no boxes, no templates, and the language is conversational. One minute they are talking about "revenue," and the next they are joking about "record-breaking growth" or clarifying a math error.

The Problem:
The researchers found that while computers are great at reading the "Formal Reports" (SEC filings), they are terrible at understanding the "Chatty Phone Calls" (Earnings calls). The computers get confused by the slang, the back-and-forth, and the lack of structure.

The Mission: Building a Better Translator

The team wanted to see if they could build a system that could listen to these messy phone calls and pull out the important numbers (Key Performance Indicators, or KPIs) just as well as a human expert.

They set up a "race" between two types of AI:

The "Old School" AI (Encoders): These are like librarians who have memorized the strict rules of the "Formal Reports." They tried to apply those same strict rules to the "Chatty Phone Calls."
The "New School" AI (Large Language Models or LLMs): These are like super-smart interns who have read almost everything on the internet. Instead of following a rigid rulebook, they use "context" (the flow of the conversation) to guess what the important numbers are.

The Experiment: The Three Datasets

To test their ideas, they created three new "training grounds" (datasets):

SECB: A test using the old, strict "Formal Reports" to see how well the AI handles the rules.
ECB: A massive collection of "Chatty Phone Calls" (unlabeled).
ECB-A: A small, high-quality sample of phone calls that a human expert carefully labeled. Think of this as the "Answer Key" for the test.

The Results: Who Won the Race?

1. The "Old School" Librarians Failed:
When the researchers tried to use the AI trained on the strict "Formal Reports" to read the "Chatty Phone Calls," it failed miserably.

Analogy: It's like trying to use a metal detector to find a specific type of fish in the ocean. The metal detector is great at finding metal on the beach (the formal reports), but it just buzzes uselessly when you take it underwater (the phone calls). The AI couldn't handle the shift from rigid text to messy conversation.

2. The "New School" Interns Showed Promise:
The researchers then used the super-smart LLMs (like Llama, Qwen, and Gemini) with a special "prompt" (a set of instructions) to act as the extractor.

The Good News: These models were much better at understanding the context. They could figure out that "iPhone revenue" in March is the same concept as "iPhone sales" in June, even if the words were slightly different.
The Bad News: They weren't perfect. While they understood the meaning (semantic understanding) very well, they sometimes struggled with the exact wording (exact match).
- Analogy: Imagine a student taking a test. The "Old School" AI got a 0% because it didn't know the questions were different. The "New School" AI got a high score on the essay portion (it understood the concept) but lost points on the multiple-choice section because it spelled the answer slightly differently than the teacher expected.

The "Human-in-the-Loop" System

Since the AI wasn't perfect, the researchers built a system that combines the AI with human logic:

Extraction: The AI listens to the call and pulls out numbers and labels.
Clustering: The system groups similar answers together. (e.g., If one AI says "iPhone Sales" and another says "iPhone Revenue," the system realizes they are the same thing and groups them).
Human Check: They had humans check the final results.
- Result: The system was 79.7% accurate. This means that out of 100 numbers the system pulled, about 80 were correct.

Why This Matters (According to the Paper)

The paper highlights a specific, tricky moment in a real earnings call (involving a company called Lyft).

The Scenario: The company released a report with a wrong number. The stock price jumped. Then, during the phone call, the CFO said, "Wait, that was a mistake, the number is actually lower." The stock price crashed immediately.
The Lesson: The "Chatty Phone Call" contains the real-time truth that the "Formal Report" misses. If an automated system can't read the call, investors miss the most critical information.

The Limitations (What the Paper Admits)

The "Gold Standard" isn't perfect: Because there are very few experts who can annotate these calls, they only had one expert label the data. This means the "Answer Key" might have missed some things, making the AI look worse than it actually is.
Company Cultures Vary: Some companies (like JPMorgan) speak very formally, while others are very casual. The system works better on some than others.
Risk: If this system makes a mistake, it could cause investors to make bad financial decisions. The paper warns that human oversight is still necessary.

Summary in One Sentence

The paper shows that while computers are great at reading strict financial forms, they struggle with messy earnings calls, but new "super-smart" AI models are getting much better at understanding the conversation, offering a promising (though not yet perfect) way to track company performance in real-time.

Technical Summary: Effective Performance Measurement: Challenges and Opportunities in KPI Extraction from Earnings Calls

Problem Statement

While earnings calls are a critical source of real-time financial information for investors, extracting Key Performance Indicators (KPIs) from them remains a significant challenge. Unlike Structured SEC filings (e.g., 10-Qs and 10-Ks), which are templated and often use XBRL tags, earnings calls are unstructured, conversational, and lack built-in labels. This "domain shift" creates a linguistic barrier where models trained on structured filings fail to generalize to the free-flowing nature of conference calls. Furthermore, existing datasets and methods often rely on predefined taxonomies, which cannot capture the emergent, open-ended KPIs frequently discussed in these calls.

Methodology

The authors propose a comprehensive framework to address these challenges, moving from established structured domains to unstructured conversational data.

1. Benchmark Creation

The study introduces three novel datasets derived from 20 S&P 500 companies:

SEC Filings Benchmark (SECB): A dataset designed to preserve broader context than previous works (like HiFi-KPI), bridging the gap between templated filings and conversational text. It utilizes pseudo-tags (regex_dollar, regex_percentage) to capture unannotated KPIs.
Earnings Calls Benchmark (ECB): A large-scale dataset of 10,477 chunks from 2023–2024 earnings calls, segmented by speaker turns.
ECB-A (Annotated): A high-quality evaluation subset containing 587 chunks with 2,460 expert-annotated entities and 934 relational groups. Annotations were performed by a domain expert using a two-stage process (identification followed by relation verification) to distinguish between traditional (financial) and non-traditional (operational) metrics.

2. Experimental Paradigms

The authors evaluate two distinct approaches:

Encoder-Based Models (Fine-tuning): They assess the generalization capabilities of models fine-tuned on SEC filings (SEC-BERT-Base, BERT-SL1000, and variants) when applied to earnings calls. These are treated as sequence labeling tasks.
In-Context Learning (Generative): They utilize State-of-the-Art (SOTA) Large Language Models (Llama-3.3, Qwen3-30B-A3B, Gemma-3-27B, and Gemini 3 Pro) with structured few-shot prompting. This approach employs a relation extraction strategy to dynamically extract KPIs without relying on a closed taxonomy. The system identifies entity spans (KPI name, value, scope, date, modality) and aggregates them into relational groups.

3. Evaluation Metrics

Evaluation on ECB-A utilizes a multi-faceted scoring system:

Exact F1: Requires exact string matches for both value and label.
Semantic F1: Uses cross-encoder similarity (STSB-RoBERTa-Large) to allow for many-to-one mappings, capturing semantic equivalence even if lexical boundaries differ.
Match F1: A soft F1 score aligning predictions to ground truths by value.
LLM Judge: An automated evaluation using DeepSeek-V3.2 to determine if a model's extraction is semantically equivalent to the expert annotation.
Human Evaluation: A final validation step involving three evaluators to verify the precision of the aggregated system output.

Key Results

1. Domain Shift Limitations

Encoder-based models fine-tuned on SEC filings (e.g., SEC-BERT-Base) demonstrated poor generalization to earnings calls. While they performed reasonably on structured datasets (FiNER-139, HiFi-KPI), their performance dropped significantly on ECB-A, often failing to predict compatible labels for extracted numbers. This confirms that the linguistic shift from templated filings to conversational calls is a major hurdle for traditional sequence labeling approaches.

2. Generative Model Performance

Generative LLMs significantly outperformed SEC-trained encoders in the unstructured earnings call domain.

Gemini 3 Pro achieved the best performance with an Exact F1 of 11.5% and extracted 45.5% of annotated KPIs according to the LLM Judge.
Semantic Metrics: While Exact F1 scores remained low (ranging from 3.2% to 11.5%), Semantic F1 scores were substantially higher (up to 61.6% for Gemini 3 Pro). This indicates that LLMs successfully capture underlying financial concepts and relationships even when they struggle with strict lexical boundaries.
Scaling: Performance improved with model scaling, suggesting the task's complexity benefits from larger model capacities.

3. System Aggregation and Longitudinal Tracking

By aggregating outputs from multiple models (Gemma, Llama, Qwen) and applying semantic clustering (with a similarity threshold of 0.85), the authors created a system capable of consistent longitudinal KPI tracking without predefined ontologies.

The system identified 1,323 KPIs consistently tracked across at least four periods for the 20 companies.
Human Validation: The final aggregated system achieved a 79.67% precision (478/600) in human evaluation, despite moderate inter-annotator agreement (Krippendorff's $\alpha$ = 0.429).
Case Study: The system successfully tracked "iPhone Revenue" across multiple quarters, aligning perfectly with reported SEC values, demonstrating its utility for financial analysis.

Significance and Claims

The paper claims to provide a robust baseline for the emerging task of automated KPI extraction from earnings calls. Its primary contributions are:

Demonstrating the Limit of Current Methods: It empirically shows that models trained on structured SEC filings do not generalize to the subjective, promotional, and unstructured language of earnings calls.
Introducing Benchmarks: The release of SECB, ECB, and ECB-A fills a critical gap in financial NLP resources, providing the first expert-annotated dataset specifically for earnings call KPI extraction.
Validating LLMs for Open-Ended Extraction: The study establishes that LLMs, when guided by structured prompting, can effectively extract emergent KPIs from unstructured text, outperforming traditional encoders.
Practical Application: The proposed system enables the consistent tracking of KPIs over time, offering a pathway to faster, more efficient processing of new financial information for market analysis.

The authors remain modest regarding the current state of the art, acknowledging that while the system shows promise, the error rate necessitates human oversight. They note that the high variability in "call culture" across different companies and the inherent ambiguity of Q&A sessions present ongoing challenges. The work serves as a foundational step toward real-time financial decision-making tools, highlighting that while full autonomy is not yet achieved, the trajectory toward efficient, automated absorption of earnings call data is viable.