Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

This paper proposes a structured framework for governing AI-assisted scientific software development within strict quality assurance standards like NQA-1, using the TMAP8 fusion energy code to demonstrate how transparent, traceable, and auditable verification and validation processes can ensure human accountability and software reliability.

The Gist

Imagine you are building a nuclear power plant. The software running the controls is like the plant's brain; if it has a tiny bug, the consequences could be catastrophic. For decades, the rule has been: "Only humans write this code, and other humans must double-check every single line." This ensures safety, traceability, and accountability.

Now, imagine a new, incredibly fast, and talented apprentice arrives: an AI coding agent. It can write code, run tests, and draft documentation in seconds. But here's the catch: this apprentice sometimes "hallucinates." It might write code that looks perfect and runs without crashing, but it's actually doing the wrong thing mathematically—like a chef who perfectly chops vegetables but accidentally swaps salt for sugar.

This paper, titled "Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability," tackles a big question: How do we let this AI apprentice help us build critical software without letting it sneak in dangerous mistakes?

The authors argue that banning AI isn't the answer (it will just go underground and become even more dangerous). Instead, we need a governance framework—a set of strict rules—to manage how AI helps.

The Core Idea: The "Proving Ground"

To test these rules, the authors didn't just talk about theory; they built a "training ground" using a specific scientific software tool called TMAP8.

Think of TMAP8 as a simulator for tritium (a radioactive fuel used in fusion energy). The software is already famous for being ultra-safe and strictly regulated (following "NQA-1" standards, which are like the "Gold Standard" of nuclear safety).

The authors used TMAP8 to test two scenarios, acting like a flight simulator for their new rules:

1. The "Copy-Paste" Challenge: They asked the AI to recreate a known scientific experiment from a published paper. The AI had to translate a human-written math model into code.
   • The Result: The AI was fast at the boring stuff (formatting files, making graphs). However, it missed a subtle detail in the original paper (a "defect annihilation" term). If a human hadn't checked the work, the simulation would have been wrong. The AI faithfully copied the mistake in the paper.
2. The "Inventor" Challenge: They asked the AI to solve a problem where no published model existed. The AI had to guess the physics, build a hypothesis, and test it against real data.
   • The Result: The AI was amazing at brainstorming. It quickly tried different ways to model a thin layer of rust (oxide) on a metal surface, something that would take a human weeks to prototype. It found a working solution much faster than a human could alone.

The New Rules: The "AGENTS.md" Contract

The paper proposes a simple but powerful solution: a file called AGENTS.md.

Think of this file as a contract or a flight manual that lives inside the software project. It tells the AI exactly how to behave. Here is what the contract demands:

• No Secrets: Every time the AI writes code, it must leave a "receipt" (metadata) saying, "I wrote this, and here is what I was thinking."
• The Human is the Captain: The AI is the co-pilot, but a human must always be the one to sign off on the work. The human is legally and scientifically responsible for the final product.
• The "Red Team" Check: The AI cannot just say, "I'm done." It must run a battery of automated tests (like a crash test) to prove its code works. If it fails, it gets sent back to the drawing board.
• Traceability: You must be able to look at the code years later and see exactly which AI tool was used, what version, and what the human did to fix it.

The Big Lessons Learned

Through their experiments, the authors found three key things:

1. AI is a Speed Booster, Not a Replacement: The AI can do the heavy lifting of typing and formatting, freeing up humans to do the hard thinking. But the human must still steer the ship.
2. The "Silent" Hallucination is the Real Danger: The scariest AI errors aren't when it writes gibberish; they are when it writes code that looks right but is scientifically wrong. The only way to catch this is with a human who understands the physics, not just the code.
3. Rules Must Be Hard-Coded: You can't just tell the AI, "Please remember to be careful." The AI forgets. Instead, the rules must be built into the software itself (like a gate that won't open unless the AI has attached its "receipt" and passed the tests).

The Bottom Line

The paper concludes that we don't have to choose between "Human-only" and "AI-only." We can have Governed AI.

By treating AI-assisted development like a regulated nuclear project—where every step is documented, every output is tested, and a human remains the ultimate authority—we can enjoy the speed of AI without sacrificing the safety and trust required for scientific discovery. The goal isn't to stop the AI; it's to make sure the AI's "apprenticeship" is safe, transparent, and accountable.

Technical Summary

Technical Summary: Bridging the Gap on AI-Assisted Scientific Software Development Through Transparency and Traceability

Problem Statement
The scientific software community faces a systemic risk arising from the ad hoc, unguided, and largely unacknowledged adoption of Large Language Models (LLMs) and agentic AI systems for code generation, testing, and documentation. While these tools accelerate development, their use in safety-critical domains—specifically those governed by strict Software Quality Assurance (SQA) standards like Nuclear Quality Assurance Level 1 (NQA-1)—poses significant challenges. Traditional SQA frameworks assume human authorship and independent verification. AI introduces correlated failure modes (where the same model generates both code and tests), increases code volume and complexity, and produces "hallucinations": syntactically valid but semantically incorrect implementations. Current workflows lack a standardized framework to govern AI involvement, risking the integrity of safety-relevant modeling and simulation tools where traceability, independent verification, and documented procedures are paramount.

Methodology
The authors propose a governance framework designed to integrate AI-assisted development into rigorous SQA environments without compromising quality or accountability. The framework is demonstrated through the development of two new Validation and Verification (V&V) cases for TMAP8, an open-source, NQA-1-compliant tritium migration code for fusion energy.

The methodology relies on a lightweight, repository-level specification file (AGENTS.md) that encodes governance requirements directly into the development infrastructure. Key components of the methodology include:

• Structured Disclosure: All contributions must include commit-level metadata detailing the degree and nature of AI involvement, linked to development issues describing intent and design rationale.
• Traceability: Machine-readable session logs (.jsonl) of AI interactions are stored in version control, allowing for the stochastic reconstruction of the development process.
• Human Accountability: A human developer is designated as the author and bears full responsibility for the contribution. Crucially, an independent human reviewer is required for verification; no agentic system, including adversarial review agents, is considered sufficient to satisfy NQA-1 verification requirements.
• Automated Enforcement: Pre-commit hooks enforce non-negotiable provenance requirements (e.g., presence of logs, metadata completeness) and ensure contributions pass the full V&V test suite before review.
• Case Studies:
  1. Implementation Challenge: Reproducing a published analytical model for tritium release from neutron-irradiated Li₂TiO₃ (Kobayashi et al.). The agent translated an existing model into TMAP8 and performed Bayesian parameter optimization.
  2. Hypothesis-Driven Challenge: Modeling deuterium release from self-irradiated tungsten with thin oxide films (Kremer et al.) where no published model existed. The agent engaged in hypothesis-driven model construction, proposing and testing competing physical mechanisms.

Key Contributions

1. Governance Framework: A practical, deployable framework for AI-assisted development that aligns with NQA-1 requirements. It shifts the paradigm from "human-only" to "governed AI assistance," ensuring AI involvement is visible, traceable, and subject to quality controls.
2. AGENTS.md Specification: A concrete implementation of governance as code, defining task execution, validation, documentation, and review standards that are version-controlled alongside the codebase.
3. Empirical Validation: The successful development of two distinct V&V cases demonstrates that AI agents can accelerate both the implementation of established models and the exploration of novel scientific hypotheses, provided they operate within strict bounds.
4. Identification of Failure Modes: The study documents specific failure modes, including the faithful reproduction of errors from source documents (e.g., omitting a defect annihilation term) and the generation of overly complex or hallucinated code, emphasizing the necessity of human oversight.

Results
The application of the framework yielded the following results:

• Productivity Gains: The agent-assisted workflow reduced development time for a validation case from approximately 4–5 days to roughly 6 hours. In the hypothesis-driven case, the agent enabled rapid prototyping of competing model formulations that would have been cost-prohibitive in a purely human workflow.
• Correctness and Errors:
  • In Case 1, the agent successfully translated the model and optimized parameters, reducing the Root Mean Square Percent Error (RMSPE) from 23.1% to 9.2%. However, it initially omitted a critical defect annihilation term from the source paper, an error detected only through adversarial human review.
  • In Case 2, the agent constructed a phenomenological model capturing the influence of oxide layers on deuterium release. While it successfully reproduced primary experimental trends across different oxide thicknesses, it required human intervention to correct hallucinated data digitization and to simplify overcomplicated model structures.
• Governance Efficacy: The AGENTS.md constraints and pre-commit hooks successfully enforced traceability and prevented contributions lacking provenance from entering the review process. However, the study noted that agents can still overlook instructions if they fall outside their attention window, necessitating structural countermeasures like automated hooks rather than relying solely on agent instructions.

Significance and Claims
The paper claims that the transition to agentic development is already a reality, and the critical choice for the scientific community is not whether to permit AI, but how to govern it responsibly. The authors assert that:

• Compatibility with Standards: Existing SQA and NQA-1 requirements (traceability, independent verification, accountability) are compatible with AI-assisted development when interpreted to include human-directed AI workflows, provided the human remains the accountable author and verifier.
• Systemic Risk Mitigation: Unregulated AI use risks "model collapse" in the long term, where AI-generated errors propagate through training data. The proposed provenance framework allows downstream curators to filter synthetic contributions, preserving the integrity of the scientific software ecosystem.
• V&V as a Proving Ground: V&V cases, with their objective correctness criteria and bounded scope, serve as an ideal environment for establishing and iterating on AI governance practices before extending them to broader scientific software development.
• Human Role Evolution: The developer's role shifts from direct code authorship to prompt engineering, output review, and active scientific oversight. While cognitive costs for individual tasks decrease, the burden of maintaining scientific oversight and ensuring model simplicity remains a critical human responsibility.

The paper concludes that responsible governance of AI-assisted scientific software is achievable today within existing quality assurance standards, without sacrificing the productivity gains that motivate AI adoption.