Technical Summary: Economic Narrative Indices and Media-Based Sentiment Measures (2007–2025)

1. Problem Statement

Traditional economic forecasting relies on quantitative statistics that are published with significant lags (30–90 days) and low frequency. In contrast, real-time textual data (media narratives, corporate disclosures, public expectations) continuously reveal economic sentiment. While substantial research has emerged since Tetlock (2007) demonstrated the predictive power of media sentiment, critical gaps remain regarding methodological consistency, the robustness of forecast improvements, and severe geographic and linguistic biases. The literature is heavily concentrated on the United States and English-language text, leaving vast populations in Africa, Latin America, and South Asia (including 265 million Bangla speakers) without dedicated economic sentiment indices. Furthermore, the causal mechanisms linking narratives to macroeconomic outcomes remain largely theoretical, and validation practices vary widely in rigor.

2. Methodology

This study conducts a systematic review following PRISMA guidelines, synthesizing evidence from 46 empirical studies (2007–2025) that extract sentiment from text for economic forecasting, supplemented by 20 theoretical papers developing macroeconomic models of sentiment.

• Search Strategy: A systematic search was conducted across five databases (Web of Science, Scopus, EconLit, SSRN, Federal Reserve working papers) using strings combining sentiment analysis, economic forecasting, and NLP/machine learning terms. Backward and forward citation tracking was employed.
• Inclusion Criteria: Studies must (1) extract sentiment/narratives from text, (2) apply them to economic/financial forecasting, and (3) employ systematic validation.
• Quality Assessment: Papers were evaluated on methodological rigor, data quality, validation robustness, and reproducibility.
• Synthesis Approach: The review establishes a methodological taxonomy, performs a quantitative synthesis of forecast performance (standardizing to RMSE reduction), and documents geographic/linguistic coverage gaps.

3. Key Contributions

The paper provides five principal contributions to the field:

1. Comprehensive Taxonomy: A classification of sentiment extraction methods across four eras: dictionary-based approaches (55% of studies), traditional machine learning (6%), transformer models (6%), and emerging Large Language Models (3%).
2. Quantitative Synthesis of Forecast Value: A meta-analysis establishing that sentiment-based models improve forecast accuracy, with a median Root Mean Square Error (RMSE) reduction of 20% across all studies. However, the paper nuances this by showing that studies using strong baselines (professional forecasts, ARIMA) report 12–15% improvements, while those using weak baselines (AR(1)) report larger but less realistic gains.
3. Documentation of Research Gaps: Explicit identification of critical geographic and linguistic biases: 56% of studies focus on the US, 84% analyze English text, and zero studies construct indices for Africa, Latin America, or Bangla-speaking regions.
4. Validation Framework: A structured, tiered framework for best practices in sentiment forecasting validation, distinguishing between essential (e.g., out-of-sample testing, real-time protocols), recommended, and advanced practices.
5. Case Study Proposal: A concrete proposal for the Bangla Economic Narrative Index (BENI) to bridge the geographic gap, utilizing BanglaBERT and keyword-assisted topic modeling.

4. Key Results

Methodological Evolution

• Dictionary-Based (2007–2018): Dominated early research (e.g., Loughran-McDonald). High interpretability and low cost but struggle with context and word order.
• Machine Learning & Transformers (2018–Present): Shift toward BERT and variants (e.g., FinBERT) improved accuracy to 85–92% and captured context.
• Large Language Models (2022–Present): Emerging use of GPT-3.5/4 for zero-shot decomposition. While achieving high accuracy (88–95%), they face issues with hallucinations, cost, and potential data contamination.
• Theoretical Models: 30% of the reviewed literature consists of theoretical papers (e.g., Angeletos & La'O, Shiller) modeling sentiment as extrinsic shocks or narrative modes, often without computational text extraction.

Forecast Performance

• Overall Improvement: The unconditional median RMSE reduction is 20%.
• Domain Specificity:
  • Volatility: Highest improvement (28% median).
  • Inflation/CPI: 22% median improvement.
  • GDP: 18% median improvement.
  • Stock Returns: 15% median improvement.
  • Employment: Weakest domain (16% median).
• Baseline Dependency: The reported value-added is highly sensitive to the benchmark. Comparisons against professional forecasts (SPF, Blue Chip) yield more modest (12–15%) but realistic gains compared to naive AR(1) models.
• Publication Bias: Only 2 of 66 papers reported null or negative results, suggesting a significant upward bias in reported performance figures.

Validation Practices

• Out-of-Sample (OOS) Testing: 83% of studies employ OOS testing.
• Real-Time Protocols: Only 33% of studies strictly adhere to real-time data protocols (avoiding look-ahead bias). Approximately 35% of papers violate this discipline.
• Statistical Rigor: While 71% use Diebold-Mariano tests, only 18% employ advanced Model Confidence Sets (MCS) or Superior Predictive Ability (SPA) tests.
• Robustness: Inter-annotator agreement is reported in only 12% of studies, and temporal robustness (testing model decay over time) is largely unexplored.

Geographic and Linguistic Disparities

• Concentration: North America (56%), Europe (20%), Asia-Pacific (14%), South Asia (6%).
• Gaps: Zero coverage for Africa, Latin America, and the Middle East.
• Language: 84% English; 16% non-English (mostly German, French, Chinese). No indices exist for Bangla, Hindi, or most non-OECD languages.

5. Significance and Claims

The paper argues that sentiment analysis has matured from a niche curiosity into a credible, albeit modest, forecasting tool. Its significance lies in moving beyond descriptive claims to a rigorous, systematic assessment of the field's state.

• On Forecasting Utility: The authors claim that sentiment indices reliably improve out-of-sample forecasts by 10–15% when conditioned on strong baselines and adjusted for publication bias. This is economically meaningful for real-time policy and trading but modest relative to model uncertainty.
• On Methodology: The field is characterized by a trade-off between interpretability (dictionaries) and accuracy (LLMs). No single "best" method exists; the choice depends on specific application requirements.
• On Causality: The review emphasizes that while sentiment predicts outcomes, the literature largely fails to establish causation. Most studies rely on Granger causality or temporal precedence, which are insufficient for policy design. The authors note that for forecasters, prediction is sufficient, but for policymakers seeking to influence outcomes via communication, the lack of causal identification is a binding constraint.
• On Future Directions: The paper posits that recent advances in multilingual NLP (e.g., mBERT, XLM-R, BanglaBERT) make geographic expansion technically feasible. It proposes that the next decade will determine whether narrative-based analysis becomes a global standard or remains accessible only to well-resourced developed economies.

The paper concludes by calling for methodological standardization (tiered validation frameworks, pre-registration, standardized effect-size reporting) and immediate expansion into underrepresented regions to ensure the robustness and generalizability of economic sentiment research.