Learning to Aggregate Zero-Shot LLM Agents for Corporate Disclosure Classification

1. Problem Statement

The paper addresses the challenge of classifying corporate disclosures (e.g., earnings reports, regulatory filings) to predict short-term market reactions (next-day stock return direction). While Large Language Models (LLMs) can perform zero-shot sentiment analysis without task-specific fine-tuning, their outputs are often unstable. Predictions vary significantly based on:

• Prompt variations: Different phrasing leads to different interpretations.
• Reasoning styles: Models may prioritize different aspects of the text.
• Model families: Different architectures yield divergent results.

Corporate disclosures are particularly complex because they often contain mixed signals (e.g., strong current earnings paired with weak future guidance, or revenue growth accompanied by legal risks). A single zero-shot judgment often reflects the prompt's emphasis rather than the full balance of the disclosure. The core research question is: Can the disagreement across multiple zero-shot agents be leveraged as a feature to create a stronger classification signal than any single agent or simple voting rule?

2. Methodology

A. Multi-Agent Framework

The system employs three distinct zero-shot agents, each using a fixed prompt to analyze the same disclosure text from a specific financial perspective:

1. Performance Agent: Focuses on realized operating performance (earnings, revenue, margins, costs).
2. Guidance Agent: Focuses on forward-looking statements (management outlook, demand expectations, revisions).
3. Risk Agent: Focuses on uncertainty, litigation, regulation, liquidity, and downside risk.

Model Pool: The framework utilizes a pool of models including Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, and Qwen2.5-72B-Instruct. The base LLMs remain frozen (no fine-tuning).

B. Agent Output Structure

For each disclosure $x$, agent $k$ outputs a tuple $a_k(x) = (l_k, c_k, r_k)$:

• Label ($l_k$): Sentiment label $\{-1, 0, +1\}$ (Negative, Neutral, Positive).
• Confidence ($c_k$): A score in $[0, 1]$.
  • For models exposing token log-probabilities, confidence is calculated as the geometric mean of token probabilities.
  • For others, it uses the model's self-reported numeric confidence (clipped to $[0, 1]$).
• Rationale ($r_k$): A one-sentence explanation.

C. Aggregation Strategy

Instead of simple voting, the authors train a lightweight logistic meta-classifier to aggregate the joint outputs.

• Feature Vector ($z(x)$): Includes the three agent labels, three confidence scores, majority label, label counts, agreement counts, confidence gaps, and interaction features (identifying which agent was most confident).
• Target Variable: Next-day stock return direction ($y_t = \mathbb{I}(r_{t+1} > 0)$).
• Training: Only the meta-classifier is trained on historical agent outputs. The base LLMs are not updated.

D. Baselines

The proposed aggregator is compared against:

1. Single Agents: Individual performance of each of the three agents.
2. Majority Vote: Simple voting based on the most frequent label.
3. Confidence-Weighted Voting: A weighted sum $s(x) = \sum c_k l_k$.
4. FinBERT: A finance-specific pre-trained BERT baseline.

3. Experimental Setup

• Dataset: 18,420 U.S. corporate disclosures from Nasdaq and S&P 500 firms (2018–2024).
• Split: Chronological split (60% training, 20% development, 20% test) to prevent information leakage and simulate real-world deployment.
• Preprocessing: Lowercasing tickers, removing duplicate headers, normalizing whitespace, and truncating to model token limits. Financial terms and numbers are preserved.
• Metrics: Accuracy, Macro F1, and Balanced Accuracy (to account for mild class imbalance).

4. Key Results

Quantitative Performance

The trained aggregator achieved the best performance across all metrics, significantly outperforming all baselines:

• Balanced Accuracy:
  • Best Single Agent (Guidance): 0.561
  • Majority Vote: 0.573
  • Confidence-Weighted Vote: 0.584
  • Trained Aggregator: 0.612
• The aggregator also outperformed the FinBERT baseline (0.544).

Qualitative Insights & Error Analysis

• Handling Mixed Signals: The aggregator showed the largest gains in "conflict" scenarios (e.g., 2–1 agent splits or high-conflict cases).
  • In 2–1 splits, the aggregator improved balanced accuracy from 0.558 (Majority) to 0.603.
  • In High Conflict cases, it improved from 0.517 to 0.581.
• Specific Scenarios: The model excelled in cases where positive current performance was paired with weak guidance, or revenue growth was paired with legal risks.
• Failure Modes: The system struggled with "routine filings" that lacked clear short-horizon signals, indicating that aggregation cannot resolve ambiguity where no signal exists.

5. Key Contributions

1. Compact Multi-Agent Zero-Shot Setup: Introduces a framework where diverse financial perspectives (Performance, Guidance, Risk) are captured by separate zero-shot agents without fine-tuning the base models.
2. Learned Aggregation vs. Heuristics: Demonstrates that a trained meta-classifier significantly outperforms simple heuristics (majority vote, confidence weighting) by learning how to weigh disagreements.
3. Disagreement as a Feature: Establishes that cross-agent disagreement is not merely noise but an informative signal, particularly for complex disclosures containing competing financial cues.

6. Significance and Implications

• Efficiency: The approach is data-efficient and computationally lightweight, requiring only the training of a simple logistic regression layer rather than fine-tuning massive LLMs.
• Practicality: It is well-suited for financial settings where high-quality sentiment labels are scarce or expensive to construct, but large volumes of text are available.
• Interpretability: By treating disagreement as structured information, the method offers a more robust way to handle the inherent complexity of corporate language, where "good news" and "bad news" often coexist.
• Future Direction: The paper suggests that future work should focus on calibrating confidence scores and testing robustness across different market regimes and abnormal returns.

In conclusion, the paper proves that supervised aggregation of diverse zero-shot LLM judgments creates a superior signal for financial classification tasks, effectively turning the "weakness" of model instability into a "strength" through structured disagreement analysis.