A Scientific Human-Agent Reproduction Pipeline

This paper introduces SHARP, a structured human-agent collaboration framework that treats scientific reproduction as a translation task, enabling researchers to guide AI agents in autonomously generating and testing analysis code while maintaining human oversight of scientific judgment.

The Gist

Imagine you find a brilliant, award-winning recipe in a famous cookbook. It describes a complex dish in beautiful, poetic language. You want to cook it exactly as written, but the instructions are vague, the measurements are in a different system, and you're not sure if you have the right tools. Trying to cook it from scratch, step-by-step, is exhausting, and if you mess up, you might never know why it didn't taste right.

This is exactly the problem scientists face when they try to reproduce (re-do) a published scientific study. They have the "recipe" (the paper), but turning that text into working code is hard, time-consuming, and often unrewarded.

Enter SHARP (Scientific Human-Agent Reproduction Pipeline). Think of SHARP not as a robot that replaces the chef, but as a super-intelligent, hyper-organized sous-chef who works alongside you.

Here is how SHARP works, broken down into simple concepts:

1. The Core Idea: Translation, Not Invention

The authors argue that reproducing science isn't about inventing new ideas; it's about translation.

• The Paper is a human-readable story: "Mix the ingredients until they glow."
• The Code is a machine-readable instruction: mix(ingredients, time=300, temp=200).
• The AI Agent is the translator. It doesn't need to be a genius chef; it just needs to be a meticulous translator that converts the story into the code without losing any meaning.

2. The Workflow: The "Check-In" System

SHARP doesn't just let the AI run wild. It uses a structured, step-by-step process with human "checkpoints."

• The Plan (The Menu): First, the human and the AI sit down and look at the paper. They agree on a menu (a plan) that breaks the huge task into small, bite-sized steps (like "chop onions," "sauté garlic," "bake the cake").
• The Sub-Teams (The Kitchen Crew): Once the plan is set, the AI doesn't do everything alone. It acts like a manager, delegating tasks to specialized "sub-agents":
  • The Analyst: Reads the paper to find specific details.
  • The Coder: Writes the actual code.
  • The Tester: Tries to break the code to make sure it works.
  • The Critic: Checks if the code is clean and organized.
• The Human Checkpoint (The Taste Test): This is the most important part. Every time the AI finishes a major step, it pauses. It says, "I've finished the sauce. Here is what I did. Does this taste right?"
  • The human scientist reviews the work.
  • They give feedback: "Good, but the sauce is too salty," or "Yes, this looks perfect, move on."
  • The AI then continues to the next step.

This ensures the human is always the Captain of the Ship, making the big decisions, while the AI handles the heavy lifting of the engine room.

3. The Real-World Test: Particle Physics

To prove this works, the team used SHARP to reproduce a famous experiment from the Large Hadron Collider (where they smash particles together to find new physics).

• The Task: They had to recreate a complex AI model used to identify "jets" (sprays of particles) created by top quarks.
• The Result: The AI, guided by the human, successfully recreated the model. The results were almost identical to the original paper (within a tiny fraction of a percent).
• The Efficiency: It took the human about one workday to oversee the whole process, whereas doing it manually might have taken weeks. The human didn't write code; they just directed the AI and checked the results.

4. Why This Matters

• It Saves Time: Scientists spend less time debugging code and more time understanding the science.
• It Builds Trust: Because the human checks every step, we know the results are faithful to the original paper.
• It Preserves Knowledge: If a scientist leaves a lab, the "recipe" (code) doesn't get lost. The AI helps rebuild it perfectly from the paper.

The Catch (Limitations)

The AI is smart, but it's not perfect.

• The "Gotcha" Moments: Sometimes the AI might miss a tiny, subtle detail that only a human expert would know (like a specific rule about the data that isn't written down).
• The Human is Still Needed: The AI can translate the recipe, but if the recipe has a hidden trap (like a "truth label" that ruins the experiment), the human needs to spot it. The AI is the tool; the human is the expert.

The Bottom Line

SHARP is like having a personal assistant who is a master coder. It takes the heavy, boring work of translating scientific papers into working software and hands it back to the researcher, who then uses their brain to verify, understand, and steer the science. It's not about replacing scientists; it's about giving them superpowers to understand their own work better and faster.

Technical Summary

1. Problem Statement

Scientific data analysis reproduction is critical for preserving knowledge, building extensible codebases, and deepening researcher understanding. However, it is rarely performed independently because the effort required often outweighs the academic recognition.

• The Core Challenge: Reproducing an analysis is fundamentally a translation task: converting human-readable specifications (papers, documentation) into machine-readable analysis code.
• Current Limitations: While AI coding agents exist, they often lack the structure to handle complex, real-world scientific workflows without hallucinating or deviating from the original methodology. Conversely, fully automated pipelines lack the necessary scientific judgment to handle domain-specific nuances.

2. Methodology: The SHARP Framework

The authors propose SHARP (Scientific Human-Agent Reproduction Pipeline), a structured framework for collaborative reproduction where an AI agent handles implementation while a human researcher maintains control over scientific judgment.

A. Workflow Architecture

The workflow follows an iterative "Ralph pattern" (based on Geoffrey Huntley's design) and operates in two main phases:

1. Initialization Phase:
   • The agent (built on Claude Code using the claude-opus-4.6 LLM) ingests the target paper and supporting documents.
   • Using the /plan skill, the agent proposes a decomposition of the reproduction into discrete tasks (defaulting to 10), marking specific tasks as checkpoints.
   • The human reviews, refines, and approves the plan (plan.md).
   • The agent converts the plan into a machine-readable specification (project.json) via the /setup skill.
2. Implementation Phase:
   • The agent executes tasks iteratively. Between iterations, state is managed via Git version control and project files.
   • Subagent Delegation: Within each task, the agent orchestrates specialized subagents in parallel:
     • Paper Analyst: Extracts specific details from the text.
     • Code & Test Subagents: Implement code in a test-driven style.
     • Statistician: Handles statistical analysis.
     • Critic: Enforces FlexCAST principles (Modularity, Testability, Robustness).
   • Checkpointing: When a task marked as a checkpoint is completed, the agent pauses. It summarizes progress and advises the human on how to run tests. The human reviews the output via a /chat interface, providing feedback or approval before the agent proceeds.

B. Technical Environment

• Sandboxing: The system runs inside claude-hpc, a sandboxed container on HPC systems (specifically NERSC's Perlmutter).
• Constraints: File system access is restricted to allowed directories; network access is limited to an allowlist (GitHub, arXiv, Conda).
• Hardware: Runs utilize a single NVIDIA A100 GPU.
• Code Generation: Uses the law workflow engine for modularity and PyTorch/PyTorch Lightning for implementation.

3. Experimental Setup

The framework was validated by reproducing the ParticleNet-Lite paper (arXiv:1902.08570), a graph neural network for jet classification in particle physics.

• Task: Classify jets initiated by top quarks vs. QCD jets using a public dataset.
• Target Metrics: Accuracy, Area Under the ROC Curve (AUC), and background rejection at 50% and 30% signal efficiency (R50, R30).
• Requirements: The agent had to implement dataset preparation, model training, evaluation, and uncertainty estimation, adhering to specific engineering constraints (e.g., GPU/CPU fallback, smoke-test modes, versioned outputs).

4. Key Results

The authors conducted three independent SHARP runs to evaluate performance, code quality, and human-agent interaction.

A. Reproduction Performance

The reproduced results matched the original paper with high precision (Table 1):

• Accuracy: Original (0.937) vs. SHARP Runs (0.938).
• AUC: Original (0.9844) vs. SHARP Runs (0.9844–0.9845).
• Background Rejection (R50/R30): The SHARP runs produced values within the statistical uncertainty of the original paper (e.g., R50: 325 ± 5 in paper vs. 318–323 in SHARP).
• Validation: Individual model checkpoints were independently validated by a human expert using a plain PyTorch definition, confirming the weights and architecture were correct.

B. Code Quality

• The agent generated a well-structured, law-based workflow.
• Components were covered by unit tests.
• The codebase was modular and ran in a self-contained Conda environment.

C. Human-Agent Interaction Analysis

Using a novel tool (claude-parser), the authors analyzed the human messages across three runs:

• Classification: Messages were categorized by Type (Essential, Optional, Meta) and Complexity (Hard, Medium, Easy).
• Findings:
  • The initial prompt (defining scope) was always a "Hard Essential" intervention.
  • Run 1 was highly efficient (9 messages), requiring minimal steering.
  • Runs 2 & 3 required more intervention (24 and 31 messages), dominated by "Medium Essential" interventions (concrete corrections/clarifications) and "Meta" messages (workflow steering like "commit now").
  • Time Efficiency: Each run took roughly one working day, with the human operating two runs in parallel at medium intensity.

5. Limitations

• Subtle Implementation Differences: The agent occasionally introduced minor deviations (e.g., learning rate schedules, activation functions) that usually had negligible impact but required cross-checking.
• Domain-Specific Pitfalls: Automated tests cannot catch all domain-specific errors. For example, a "truth-label leakage" issue (where a trivial particle label invalidates the classification) was encountered during development but missed by automated tests, highlighting the need for human oversight.

6. Significance and Contributions

• Paradigm Shift: SHARP demonstrates that the researcher's role in reproduction can shift from writing code to understanding, evaluating, and directing the analysis. This elevates human understanding rather than replacing it.
• Translation vs. Extrapolation: The paper successfully frames scientific reproduction as a translation task, which is uniquely well-suited for current AI agents, moving away from the expectation that agents must be creative problem solvers.
• Structured Collaboration: It provides a "middle ground" between rigid, task-specific pipelines and unstructured agent workflows, ensuring consistency and quality while remaining flexible enough for non-trivial scientific problems.
• Open Source: The authors released the SHARP template, the conversation analysis tool (claude-parser), and the sandboxing tool (claude-hpc), fostering reproducibility in AI-assisted research.

Conclusion: SHARP offers a practical, high-fidelity model for AI-assisted scientific reproduction, proving that human-agent collaboration can successfully reproduce complex particle physics analyses while significantly reducing the manual coding burden on researchers.