💻 computer science

The Say-Do Gap Index: A Reproducible NLP Pipeline for Detecting ESG Rhetoric–Reality Divergence in Corporate Media Coverage

This paper presents a fully reproducible NLP pipeline, including a curated corpus, a six-category ESG taxonomy, and a fine-tuned Spanish BERT classifier, designed to quantify the divergence between corporate ESG rhetoric and observable outcomes in media coverage through a novel "Say-Do Gap Index" while addressing construct validity and cross-context applicability.

Original authors: Alfredo Merlet

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alfredo Merlet

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern business world, companies frequently make public promises about how they will protect the environment, treat their workers, and act ethically. These promises, often called environmental, social, and governance or ESG commitments, are meant to show that a firm cares about more than just profit. However, a persistent question haunts investors, regulators, and the public: do these companies actually follow through? There is a growing concern that some firms engage in "greenwashing," a practice where they talk a good game in their reports and press releases but fail to take meaningful action in the real world. Measuring the distance between what a company says and what it actually does has traditionally been a slow, manual process, requiring researchers to read thousands of documents and cross-reference them with news reports of scandals or violations. This gap between rhetoric and reality is difficult to track at scale, leaving a blind spot in how we understand corporate accountability.

A new study by Alfredo Merlet from the University of Seville offers a way to shine a light on this gap using a tool from the field of artificial intelligence. Rather than simply reporting on whether Spanish companies are keeping their promises, this paper focuses on building the machine that can check them. The researcher developed a fully documented, reusable system that can read thousands of news articles, understand what they are saying about a company's behavior, and automatically calculate a score that represents the difference between a firm's words and its deeds. The system was tested on a massive collection of 131,151 news articles covering 34 major Spanish companies over a seven-year period. By teaching a computer to distinguish between genuine actions, vague promises, and negative controversies, the study created a reliable method for spotting when a company's public image does not match its actual performance.

The core of this work is a pipeline, which is essentially a step-by-step assembly line for processing information. First, the system gathers news stories from a global database that tracks media coverage. It filters these stories to find only those mentioning specific large Spanish companies. Next, it uses a specialized computer program trained on the Spanish language to read each article and sort it into one of six categories. These categories range from clear evidence of positive environmental action to vague commitments that have not been proven, and finally to reports of negative controversies or scandals. The system was taught to make these distinctions by showing it thousands of examples that human experts had already labeled, a process known as training. The researchers found that the computer struggled at first to identify vague promises because there were very few examples of them in the initial data. To fix this, they specifically searched for more articles containing those types of promises and added them to the training set, which significantly improved the computer's ability to spot them.

Once the system could accurately classify the news, the researchers used it to build a score for each company for every year in the study. This score, called the Say-Do Gap Index, measures the balance between unverified promises and actual positive actions or negative controversies. If a company makes many vague promises but few concrete achievements, the score reflects a large gap. If the company's actions align with its words, the gap is small. The final result is a dataset covering 238 different company-years, providing a clear, numerical view of how well these firms have matched their talk with their walk. The computer model achieved a high level of accuracy in its classifications, successfully identifying the different types of news stories with a performance level that researchers consider robust enough for further study.

Crucially, the paper emphasizes that this tool is designed to be used by other scientists and analysts, not just for this specific study. The entire code, the rules for how the computer was taught, and the steps for gathering the data are made available to the public. This allows anyone to apply the same method to different countries, different time periods, or different languages, provided they adjust the system to fit the local context. The author notes that while the structure of the tool is universal, the specific computer brain used here was trained only on Spanish text, so it would need to be retrained for other languages to work correctly.

The study also draws a careful line between what the tool can do and what it cannot. The system is excellent at measuring the observable difference between a statement and an outcome. However, the paper warns that a large gap does not automatically prove that a company is lying or acting with bad intent. A company might fail to meet a goal because of unexpected economic shifts, new laws, or legitimate changes in strategy. The tool identifies the discrepancy, but it cannot explain the reason behind it. Determining whether a gap represents deception or simply a change in circumstances requires human judgment and an understanding of the specific situation.

Ultimately, this work provides a new way to scale the study of corporate honesty. Before this pipeline, checking the consistency of corporate promises required immense human effort and was limited to small samples. Now, researchers can systematically analyze vast amounts of media coverage to see where the gap between words and actions exists. The study does not claim to have solved the problem of corporate accountability, nor does it declare that all companies with a gap are guilty of misconduct. Instead, it offers a reproducible, transparent method for measuring the phenomenon, turning a vague suspicion about greenwashing into a concrete, data-driven observation that can be verified and improved upon by others.

Technical Summary: The Say-Do Gap Index Pipeline

Problem Statement
The paper addresses the challenge of detecting the "say-do gap"—the divergence between corporate Environmental, Social, and Governance (ESG) rhetoric and observable outcomes. While existing literature often relies on self-reported disclosures or manual analysis, there is a need for a scalable, reproducible methodology to verify corporate claims against external signals, specifically large-scale media coverage. The paper distinguishes itself from substantive empirical studies by treating the methodological infrastructure required to measure this gap as the primary object of contribution, rather than the empirical findings of a specific application.

Methodology
The author presents a fully documented, reproducible Natural Language Processing (NLP) pipeline designed to process Spanish-language media coverage of the IBEX 35 index (Spain) from 2019 to 2025. The pipeline consists of four core components:

Corpus Extraction:
- Source: Data is extracted from the GDELT Global Knowledge Graph via Google BigQuery.
- Entity Resolution: To avoid corruption of short acronyms (e.g., "BBVA") inherent in GDELT's native V2Organizations field, the pipeline utilizes URL-slug matching for firm identification.
- Scope: The final corpus comprises 131,151 deduplicated news articles covering 34 of the 35 IBEX 35 constituent firms.
Taxonomy and Annotation:
- A six-category ESG behavior taxonomy was developed, adapted from Walker and Wan (2012)'s substantive/symbolic distinction:
  1. Substantive ESG action.
  2. Unverified/aspirational commitment.
  3. Ambiguous disclosure/greenwashing signal.
  4. Negative ESG controversy.
  5. Informative/neutral ESG mention.
  6. Non-ESG noise.
- Annotation Protocol: A stratified seed sample of 700 articles was manually annotated. To address class imbalance, a targeted keyword-search enrichment procedure was applied to increase the representation of underrepresented categories (specifically unverified commitments).
- Data Cleaning: A critical inconsistency was identified where the model learned input length as a spurious proxy for specific categories. Standardizing the input format (URL-slug only) across training and inference corrected a 15.6% over-prediction rate down to 2.4%.
Classifier Development:
- Model: The pipeline employs a fine-tuned BETO (Spanish BERT) model, selected for its exclusive pre-training on Spanish corpora.
- Iterative Training: The model underwent three rounds of development:
  - v1: Six separate categories yielded a Macro F1 of 0.49, with categories 2 and 3 failing to capture due to severe under-representation.
  - v2: Categories 2 and 3 were merged into a single "unverified commitment" class, raising Macro F1 to 0.59.
  - v3 (Final): Following targeted enrichment of the merged class, the final model achieved a Macro F1 of 0.76. The merged category improved from F1=0.00 to F1=0.73, with all categories reaching usable performance (0.63–0.91).
- Precision Note: Manual verification of the critical "unverified commitment" category revealed a precision of approximately 56%, a limitation explicitly acknowledged by the author.
Index Construction:
- The Say-Do Gap Index is calculated as a firm-year metric:
  $\text{Say-Do Gap Index} = \frac{N_{2/3} - N_1 - 2 \times N_4}{N_{ESG}}$
- Where $N_{2/3}$ is the count of unverified/ambiguous articles, $N_1$ is substantive action, $N_4$ is negative controversy, and $N_{ESG}$ is the total count of ESG-relevant articles.
- The resulting panel covers 238 firm-year observations.

Key Contributions

Reproducible Infrastructure: The paper provides a complete, reusable architecture for ESG divergence detection, including extraction scripts, annotation tools, and classifier training code, all deposited on Zenodo under a CC-BY 4.0 license.
Methodological Transparency: It documents the iterative process of taxonomy design, the handling of class imbalance, and the correction of spurious model learning, offering a blueprint for similar research.
Construct Validity Discussion: The author explicitly frames the distinction between measuring a textual discrepancy and interpreting it as behavioral misconduct. They argue that while the pipeline reliably detects observable gaps, the causal inference regarding accountability requires contextual judgment beyond the NLP model.

Results and Significance
The paper does not present new empirical findings regarding firm value or market reactions; those are reserved for a separate manuscript (Merlet Echavarría, under review). Instead, the significance of this work lies in its operationalization of the Say-Do Gap.

Performance: The pipeline successfully achieves a Macro F1 of 0.76 on a complex, multi-class classification task involving nuanced ESG concepts in Spanish.
Generalizability: The author distinguishes between architectural generalizability (the pipeline's logic and structure, which they argue is transferable to other markets and languages) and empirical generalizability (specific parameters and models, which require recalibration for different linguistic or institutional contexts).
Limitations: The paper modestly claims that the pipeline measures observable discrepancies but does not resolve the question of why those discrepancies exist (e.g., distinguishing between strategic adjustment, external shocks, and deception). The author emphasizes that the tool scales the identification of the phenomenon but does not replace the need for theoretical and contextual interpretation by researchers.

In summary, this paper serves as a methodological foundation, offering a validated, open-source tool for researchers to study ESG rhetoric-reality divergence, while rigorously defining the boundaries of what the tool can and cannot infer.

Technical Summary: The Say-Do Gap Index Pipeline

More like this