Technical Summary: AI-Assisted Scientific Assessment on Climate Change

Problem Statement

The paper addresses a critical gap in the application of Artificial Intelligence to scientific discovery: the distinction between verification-rich and verification-poor domains. Current "AI co-scientist" paradigms, which utilize autonomous agents in "guess and check" loops, excel in verification-rich environments (e.g., drug discovery, code optimization) where ground truth is accessible via immediate, repeatable experimental feedback. However, high-stakes fields like climate science, macroeconomics, and epidemiology are often verification-poor. In these domains, controlled experiments are impossible, systems are non-stationary, and "truth" is an authoritative state of knowledge established through expert consensus rather than direct measurement.

The central problem is how to deploy AI in these verification-poor contexts to support scientific assessment—the process of weighing reinforcing and diverging evidence against background theory to form a consensus—without compromising scientific rigor or introducing hallucinations and biases. The authors hypothesize that a hybrid intelligence architecture, where AI acts as an integrated assistant within a collaborative human workflow, can enhance efficiency and consistency while maintaining necessary expert oversight.

Methodology

The authors conducted an empirical case study involving 13 climate scientists with diverse expertise in the Atlantic Meridional Overturning Circulation (AMOC). The study utilized a proprietary AI environment built on Gemini (accessed via Vertex AI) and integrated into a standard scientific workflow. The experiment spanned five weeks (November–December 2025) and was structured into three phases to approximate IPCC-style assessments:

1. Phase 1 (Outline): Using Gemini 2.5 Pro, the AI generated an initial outline for a 10-page assessment on AMOC stability. Scientists edited this draft to produce a final outline and assigned sections.
2. Phase 2 (Sections): Transitioning to Gemini 3 Pro Preview, the AI expanded the outline into a full draft. It utilized a Retrieval-Augmented Generation (RAG) system with a corpus of ~1,660 AMOC papers (processed into Markdown) and external search capabilities. The AI iteratively identified relevant sources, incorporated citations, and drafted text. Scientists then curated these drafts in groups.
3. Phase 3 (Final Report): The AI collated sections into a coherent report with unified references. Scientists performed a final review to resolve issues, ensure rigor, and sign off on the content.

The AI Environment:
The system featured three main components:

• Editor: A Markdown-based interface allowing manual edits or AI-executed edits via document-level or text-span-level feedback (highlighting).
• Evidence Panel: A dynamic list of documents used as context, including summaries, bibliographic metadata, and retrieved snippets from the AMOC corpus and Google Search.
• Assistant Chatbox: A context-aware interface for querying the AI, requesting revisions based on rhetorical or argumentative logic (e.g., Toulmin schema), and resolving conflicts.

Evaluation Metrics:
To quantify human-AI dynamics, the authors adapted NLP evaluation metrics to the sentence level:

• Retention (Precision): The fraction of AI-proposed sentences thematically present in the final expert-curated version.
• Intervention (Recall/False Negative Rate): The fraction of final report sentences that had no thematic correspondence in the initial AI draft (representing new human work).
• Convergence: Measured via text similarity (semantic embeddings) between successive versions and the final version.

Key Results

Efficiency and Convergence

• Time Savings: The 13 scientists produced a synthesis of 79 papers into an 8,000-word report through 104 revision cycles in 46 hours and 33 minutes of logged time.
• Expert Estimation: In a post-study survey, all participants reported a speedup of at least 2–3x, with 38% estimating a 7x or greater improvement. Participants estimated that a comparable task without AI would typically require 100–200 person-hours.
• Convergence: Text similarity metrics showed the report converged almost monotonically toward the final version, with rapid changes in the first 9 versions followed by a refinement stage.

AI Contribution and Human Oversight

• High Retention: The AI draft content was largely preserved. Retention scores were 98.8% for the outline and 93.9% for the full report, indicating high initial quality.
• Significant Intervention: Despite high retention, Intervention was substantial (58.3% for the report), meaning the majority of the final content was added or significantly altered by humans.
• Revision Attribution: AI-generated revisions were preferred over manual edits, accounting for 80% of all revisions and ~75% of submitted versions. However, the authors estimate that AI contributed only **25%** of the final revision content, with humans contributing 58.3% of the final report's text.
• Critical Corrections: Human experts were essential for elevating the text from a general summary to a rigorous scientific assessment. Key corrections included:
  • Downgrading the confidence level on anthropogenic attribution of AMOC weakening from "Medium" to "Low."
  • Correcting the AI's claim of a "full scientific consensus" on weakening, replacing it with a nuanced statement reflecting conflicting evidence.
  • Replacing emotive language (e.g., "catastrophic," "pivotal") with precise, neutral scientific terminology.

Survey Findings

• Quality: 62% of experts rated the final report as a "solid foundation" requiring minor polish; 23% were neutral; 15% felt it needed more work.
• Cognitive Load: 62% reported a lower cognitive burden using the assistant compared to drafting from scratch.
• Trust and Limitations: No participant expressed "full trust" in the AI. 62% observed "sycophancy" (over-agreeableness). Experts noted the AI's difficulty in synthesizing quantitative ranges and its tendency to smooth over conflicts rather than highlighting them.

Significance and Claims

The paper claims that AI scientific assistance can extend to verification-poor tasks, but only within hybrid intelligence architectures where human expertise remains central.

1. Productive Hybrid Framework: The study demonstrates that an AI assistant can accelerate the scientific synthesis process, serving as a "starting point" that does not anchor or limit expert reasoning. The final output was deemed suitable for peer review.
2. Shift in Scientific Workflow: The findings support a "cognitive offloading" model where experts delegate mechanistic sub-tasks (drafting, retrieval, cross-referencing) to AI while retaining deliberative control over scientific judgment, uncertainty articulation, and evidence quality.
3. New Metrics for Co-Authoring: The authors propose Retention and Intervention as simple, sentence-level metrics to quantify human-AI co-authoring dynamics, offering a stepping stone for future research in this area.
4. Limitations and Safeguards: The paper emphasizes that AI cannot replace the need for expert oversight in high-stakes assessments. It identifies evidence traceability, numeracy verification, and unbiased literature selection as critical safeguards. The study acknowledges limitations, including the use of a single domain (climate science), a single topic (AMOC), and the lack of a formal human-only control group.

The authors conclude that while AI excels at information processing ("the What"), human expertise is indispensable for evaluating evidence quality and forming scientific judgment ("the So What"). This augmentative approach enables expert groups to accelerate research on long-horizon problems, suggesting a future where such environments play a central role in developing trusted, full-stack AI co-scientists for verification-poor domains.