Evaluating the scientific rigour of climate scenario analysis: LLM-assisted evidence from New Zealand’s mandatory disclosures

Technical Summary: Evaluating the Scientific Rigour of Climate Scenario Analysis in New Zealand

Problem Statement
Global financial stability is increasingly threatened by climate change, prompting a wave of mandatory climate-related financial disclosures (e.g., TCFD, IFRS S1/S2, EU CSRD). New Zealand (NZ) pioneered this regulatory shift, legally requiring approximately 200 large entities (Climate Reporting Entities or CREs) to disclose climate resilience against distinct scenarios starting January 1, 2023. However, a critical gap exists between regulatory compliance and genuine scientific utility. While companies incur significant costs (up to NZ$2 million per entity) to produce these disclosures, the scientific rigour of their underlying scenario analyses remains unexamined. There is a risk that these disclosures are merely narrative "box-ticking" exercises, prone to greenwashing, misinterpretation of climate science, and "internal inconsistency" where disparate models and data sources are conflated. Furthermore, the manual auditing of hundreds of complex, multi-page documents to verify scientific grounding creates an operational bottleneck for regulators and investors.

Methodology
This study employs a novel, Large Language Model (LLM)-assisted framework to audit the scientific rigour of NZ's first two mandatory reporting periods (FY23/24 and FY24/25).

• Sample: The researchers compiled and filtered a dataset of 227 climate statements from 173 unique reporting entities across financial and non-financial sectors.
• Assessment Framework: A multidisciplinary team of experts in climate science, finance, and adaptation developed a matrix of 13 specific questions to evaluate scenario analysis. These questions target key elements such as scenario names, temperature targets, data providers, baseline assumptions, and the integration of physical and economic models.
• LLM Architecture: The study utilized an Agentic Retrieval-Augmented Generation (RAG) pipeline powered by Anthropic's Claude 3.5 Sonnet.
  • Data Processing: Documents were parsed using PyMuPDF and Tesseract OCR, then processed via a hierarchical "small-to-big" chunking strategy (400-token child chunks linked to 2,000-token parent chunks) stored in a Qdrant vector database.
  • Agentic Workflow: Unlike rigid linear RAG pipelines, the system used an iterative, tool-driven agent capable of multi-turn reasoning. The agent dynamically adjusted search queries to locate implicit or non-linearly distributed information (e.g., specific modelling assumptions).
  • Output Structure: The agent was constrained by a strict system prompt to output structured data for each of the 13 questions, including the answer, exact textual evidence (quotes), location pointers, anomaly logs, and a confidence metric (High/Low).
• Validation: To ensure reliability, the automated pipeline was benchmarked against a human-annotated "ground truth" derived from a manual audit of 13 disclosures. The system achieved a 91% exact accuracy rate (93% including partial matches), with failures primarily occurring in complex, interpretive tasks rather than explicit text extraction.

Key Results
The analysis of 227 disclosures reveals significant structural paradoxes in the current state of climate reporting:

1. Narrative Dominance over Quantification: Disclosures are highly comprehensive in volume (averaging ~47 pages) but overwhelmingly narrative. Quantitative metrics account for only ~2.1% of total characters. Financial entities tend to use a more negative, prudent tone compared to non-financial entities, but both sectors remain heavily text-based.
2. Fragmented and Inconsistent Methodologies: While all entities adhere to the three-scenario mandate (1.5°C, ≥3°C, and a third pathway), the underlying taxonomies are highly fragmented. Entities frequently "pick-and-mix" scenarios from international providers (IPCC, NGFS) and local bodies (CCC, NIWA) without clear structural logic.
3. Lack of Scientific Rigour and Baselines:
   • 68% of disclosures do not explicitly state their baseline scenarios.
   • There is widespread conflation of data aggregators (e.g., S&P Trucost, NGFS) with primary modelling architectures (e.g., CMIP6, IAMs like GCAM).
   • Explicit macroeconomic transmission modelling (e.g., Computable General Equilibrium models) is almost entirely absent.
   • Entities often conflate structural scenario baselines with localised physical modelling baselines (e.g., historical data), creating "internal inconsistency."
4. LLM Performance and Limitations: The agentic RAG system successfully scaled the analysis but highlighted specific failure modes. The model exhibited a negative correlation between its internal confidence scores and actual retrieval success in complex areas (e.g., identifying baselines). The system frequently generated "low-confidence assertions" or defensive rationales when data was missing, rather than cleanly flagging "not found," demonstrating that prompt engineering alone cannot eliminate hallucinations in unstructured data.

Key Contributions

• First Systematic Audit: This study provides one of the first systematic evaluations of the scientific rigour of corporate climate scenario analysis under a mandatory regime.
• Automated Audit Framework: It demonstrates a scalable, LLM-assisted RAG framework capable of auditing hundreds of disclosures, offering a potential tool for international regulators as global mandates expand.
• Diagnostic of "Analytical Fog": The paper identifies that the current regulatory flexibility, combined with heterogeneous data sources, has created an "analytical fog" where costly disclosures fail to provide comparable, scientifically grounded data for capital allocation.
• Validation of Agentic RAG: It offers empirical evidence on the reliability and boundary conditions of using agentic AI for policy analysis, showing high accuracy for explicit data but significant challenges in interpreting complex, implicit scientific logic.

Significance and Claims
The paper claims that while NZ's regulatory framework successfully compelled legal compliance, it has inadvertently incentivized a "performative" compliance loop. The current disclosures, constrained by narrative styles and inconsistent methodologies, fail to deliver the market transparency and capital allocation efficiency intended by the TCFD and IFRS standards.

The authors argue that the gap between regulatory compliance and scientific utility is exacerbated by the decoupling of physical climate science from financial decision-making. They caution that without a shift toward standardised, machine-readable reporting and explicit coupling of physical and economic models, automated auditing systems will merely scale the extraction of "empty narrative." The paper concludes that the deployment of LLMs in climate auditing is not a silver bullet; rather, it exposes the need for stricter data standardization at the source to prevent the amplification of uncertainty and the risk of maladaptation in the financial system.