Assessing the Credibility of Corporate SDG Claims: An Agentic AI Framework for Sustainability Report Analysis

Technical Summary: Assessing the Credibility of Corporate SDG Claims via an Agentic AI Framework

1. Problem Statement

Corporate engagement with the United Nations Sustainable Development Goals (SDGs) has expanded rapidly, with 74% of the world's largest 250 companies referencing the SDGs in their 2024 reporting. However, the quality of these disclosures has not kept pace with their quantity. A significant portion of corporate SDG claims remain symbolic—generic references lacking measurable targets, quantitative metrics, or verifiable evidence. This phenomenon, termed "SDG-washing," involves appropriating the normative authority of the SDGs to construct legitimacy without substantive action.

Current assessment methods face a "scalability–rigour paradox": manual content analysis is rigorous but cannot scale to the tens of thousands of annual reports, while automated keyword matching is scalable but fails to distinguish between the mere presence of SDG references and the credibility of the claims. Existing AI tools (e.g., OSDG, ClimateBERT) classify SDG relevance but do not evaluate the depth, specificity, or evidentiary support of the claims. Furthermore, there is a lack of automated, claim-level credibility assessment specifically applied to the Thai corporate context, where reporting is mandatory but content quality varies significantly.

2. Methodology

The study develops and validates a four-phase Agentic AI Framework designed to assess the credibility of corporate SDG claims at scale. The framework utilizes a multi-agent architecture implemented in Python using LangGraph, featuring feedback loops that allow downstream agents to trigger re-examination of upstream assessments.

The Four-Phase Architecture:

1. Planner Agent (Ingestion & Structuring): Converts raw PDF sustainability reports into structured, hierarchical JSON text units using LandingAI's Agentic Document Extraction (ADE). This vision-based system preserves relational context and links assessments to precise source locations. It distinguishes discrete sustainability claim units from boilerplate text and creates a persistent "chunk pool" of non-claim evidence (e.g., data tables, executive summaries).
2. Mapper Agent (Target-Level Mapping): Maps each extracted claim against the 169 specific SDG targets (rather than just the 17 high-level goals). It employs a two-stage prompting process: goal-level screening followed by target-level relevance judgment. Claims are mapped with a confidence score; those below a 0.5 threshold are flagged as unmappable.
3. Scorer Agent (Multi-Dimensional Credibility Assessment): The core methodological component. It evaluates each mapped claim on a five-dimensional credibility rubric (SMTEC), grounded in legitimacy, signalling, and accountability theories:
   • Specificity: Concreteness of action/initiative.
   • Measurability: Presence of quantitative metrics, KPIs, and baselines.
   • Temporality: Existence of deadlines, milestones, and timeframes.
   • Evidence: Type and strength of substantiation (e.g., third-party verification).
   • Consistency: Internal coherence across the report (checked via cross-section retrieval).
   • Scoring: Each dimension is scored on a 0–3 ordinal scale. A composite score (mean of the five) classifies claims into three tiers: Symbolic (<1.0), Partially Substantive (1.0–1.99), and Substantive (≥2.0).
   • Retrieval Mechanism: The agent uses a chunk-aware retrieval pipeline that searches the entire report (including the non-claim chunk pool) for corroborating evidence, rather than relying solely on the immediate context of the claim.
4. Comparator Agent (Cross-Report Analysis): Aggregates outputs to identify patterns, outliers, and "washing hotspots" (high frequency, low credibility) across sectors and goals.

Validation Protocol:
The framework was applied to 30 sustainability reports from Stock Exchange of Thailand (SET) listed companies (2024 reporting year), covering 6,325 mapped claims. Validation involved:

• Expert Reliability Benchmark: A stratified sample of 63 claims was scored by two domain experts and the AI framework. Agreement was measured using Krippendorff's ordinal alpha and Cohen's quadratic-weighted kappa.
• Retrieval Robustness: Three retrieval pipelines (OpenAI embeddings, local open-source Nomic-Embed, and LLM-as-judge) were compared to ensure findings were not artifacts of a specific embedding model.
• Prompt Iteration: The scoring prompt was refined through three versions based on expert feedback, treating the prompt as a methodological artefact subject to empirical calibration.

3. Key Results

• Reliability: The framework achieved a three-rater Krippendorff's ordinal alpha of 0.744, falling within the "substantial agreement" range. The framework's agreement with individual experts was statistically indistinguishable from the agreement between the two human experts, indicating the AI reached the empirical ceiling for this task.
• Credibility Distribution: Across the corpus of 6,325 claims:
  • 8.7% were classified as Symbolic.
  • 68.6% were Partially Substantive.
  • 22.8% were Substantive.
• Dimensional Weaknesses: Measurability was the weakest dimension (mean score 0.71), indicating most claims lack quantitative metrics. Consistency was the strongest (mean 2.63), largely due to the chunk-aware retrieval finding cross-references.
• Under-Claiming vs. Over-Claiming: Contrary to the assumption that firms over-claim, the study found firms tend to under-claim their SDG engagement. Companies declared fewer goals than their reports substantively addressed.
• SDG-Washing Hotspots: A frequency-credibility matrix identified SDGs 3, 8, 9, and 16 as hotspots (high reporting frequency, low credibility), particularly regarding Measurability. Conversely, SDG 13 (Climate Action) showed high reporting frequency but retained above-median credibility, challenging the view that climate disclosure is purely "cheap signalling."
• Methodological Impact: Using the advanced "chunk-aware" retrieval and refined prompts reduced the estimated prevalence of symbolic reporting from 20.7% (baseline) to 8.7%, demonstrating that single-pass, claims-only methods significantly overestimate symbolic engagement.

4. Key Contributions

• Methodological: Development of the first automated framework capable of assessing the credibility (not just relevance) of SDG claims at scale. The study operationalizes the SMTEC rubric and introduces a chunk-aware retrieval pipeline that surfaces cross-section evidence, treating the absence of evidence as a signal.
• Theoretical: Advances Legitimacy, Signalling, and Accountability theories by operationalizing them within a computationally tractable instrument. It identifies specific patterns such as the "SDG-13 institutional exception" and the "under-claiming asymmetry."
• Empirical: Provides the first claim-level evidence for the Thai corporate context, revealing that while reporting adoption is high, substantive engagement remains limited (22.8% of claims), and that firms often under-report their actual performance.
• Practical: Offers a deployable instrument for regulators, investors, and standard-setters to screen high-risk disclosures and construct credibility-weighted SDG scores.

5. Significance and Claims

The paper claims to resolve the scalability–rigour paradox in sustainability reporting analysis. By moving beyond simple keyword classification to a multi-agent, rubric-based assessment, the framework distinguishes between substantive and symbolic disclosures with a reliability comparable to human experts.

The study emphasizes that the distinction between the presence of SDG alignment and the quality of that alignment is critical. It argues that current automated tools fail to address this gap. The findings suggest that while the volume of SDG reporting is high, the depth of commitment is often shallow, characterized by a lack of measurable targets and quantitative evidence. The framework provides a scalable mechanism to hold corporations accountable for the substance of their claims, offering a tool for stakeholders to navigate the "form-versus-substance" gap in global sustainability reporting.

The authors remain modest regarding the "truth" of the findings, acknowledging that results are sensitive to instrument choices (e.g., retrieval design and prompt iteration). However, they assert that the framework offers a robust, theory-grounded approach to assessing corporate SDG claims that was previously unavailable.