CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants

This paper introduces CelloAI, a suite of repeatable and automated benchmarks designed to evaluate Large Language Models on HEP/HPC-specific tasks, including code documentation, GPU kernel generation, and graphical data analysis, to address the unique constraints of scientific software development.

The Gist

Imagine you are trying to teach a brilliant, but slightly naive, new intern (the AI) how to work in a massive, ancient, and incredibly complex factory. This factory doesn't build cars; it simulates the birth of the universe using supercomputers. The blueprints are thousands of pages long, written by different people over decades, and if the intern makes a tiny mistake, the whole simulation could crash or, worse, give you the wrong answer about how the universe works.

This paper is about CelloAI, a team of researchers who decided that the standard "tests" we use to check AI coding skills aren't good enough for this high-stakes factory. They built a new set of specialized training drills (benchmarks) to see if AI can actually handle the messy, real-world job of scientific coding.

Here is a breakdown of their three main drills, explained simply:

1. The "Translator" Drill (Code Documentation)

The Problem: The factory's manuals are a mess. Some parts have no labels, and the labels that exist are vague. The AI needs to write clear, structured notes (like "Doxygen" comments) for every machine part so future workers know what it does.
The Drill: They asked different AIs to write these notes.

• The Scorecard: They didn't just check if the AI wrote something. They checked:
  • Did it miss anything? (Coverage): Did it label every single screw and wire?
  • Did it make sense? (Semantics): Did it describe the machine correctly, or did it just sound like it was speaking gibberish?
    The Result: Most AIs are great at remembering to put a label on every part (high coverage), but they often struggle to write a good description that matches what a human expert would say. It's like an intern who labels every tool but writes "This is a thing" instead of "This is a wrench for tightening bolts."

2. The "Moving Day" Drill (Code Porting)

The Problem: The factory is upgrading its machinery. They need to move a complex process from an old, slow machine (CPU) to a new, super-fast one (GPU). This is like trying to move a delicate, intricate clockwork mechanism from a wooden box to a high-tech steel case without breaking a single gear.
The Drill: They gave the AIs a specific, tricky job: take a piece of code that runs on a standard computer and rewrite it to run on a graphics card (GPU) used for high-speed physics simulations.

• The Scorecard: They didn't just ask, "Did the code look right?" They ran the code.
  • Did it compile? (Did it turn on?)
  • Did it run? (Did it work without crashing?)
  • Did it give the right answer? (Did it simulate the physics correctly?)
    The Result: This was the hardest test.
• Easy tasks: Simple tasks (like resetting a counter) were easy for the AIs.
• Hard tasks: The complex "simulation" tasks were a disaster. Even the smartest AIs failed to get the code to run correctly more than 1 or 2 times out of 10. It showed that while AI is good at writing small snippets, it struggles with the "big picture" logic needed to move complex systems without breaking them.

3. The "Detective" Drill (Graphical Data Analysis)

The Problem: Scientists look at thousands of charts and graphs to find problems. If a graph looks slightly different than yesterday's, it could mean a broken machine or a new discovery. Humans are good at spotting these "glitches." Can an AI look at a picture of a graph and say, "Hey, that line is weird"?
The Drill: They showed the AIs pictures of histograms (bar charts) with a "normal" line and a "monitored" line that had some errors.

• The Scorecard: The AI had to point out exactly where the lines didn't match and list the weird points.
  The Result: The AIs were okay at spotting the obvious errors, but they weren't great at being precise detectives. Some models missed the clues entirely, while others found them but got confused about exactly where the problem started. It's like showing a detective a crime scene photo; they can see something is wrong, but they can't always pinpoint the exact fingerprint.

The Big Takeaway

The main message of this paper is: We need better tests for AI in science.

Current tests are like asking an AI to solve a math puzzle on a clean sheet of paper. But real scientific coding is like fixing a leak in a dam while the water is rushing over it. You can't just write code; you have to make sure it fits into a giant, fragile system, compiles, runs, and doesn't break the laws of physics.

The researchers built these new "drills" so that in the future, we can fairly compare AI models and figure out which ones are actually ready to help scientists build the next generation of supercomputers, rather than just ones that are good at writing homework essays.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) are increasingly used for software development, existing coding benchmarks (e.g., SWE-bench, LiveCodeBench) fail to address the specific constraints of High Energy Physics (HEP) and High Performance Computing (HPC).

• Context Gap: HEP/HPC codebases are massive, sparsely documented, and rely on complex dependencies. Standard benchmarks often lack the context required for these environments.
• Scientific Constraints: Correctness in HPC is not just about passing unit tests; it requires adherence to scientific constraints (physics intent, numerical stability) and system constraints (memory layouts, CPU-GPU transfers).
• Evaluation Deficit: Current qualitative or anecdotal assessments do not provide a repeatable, automated basis for comparing models or strategies across large, performance-critical repositories.

2. Methodology

The authors introduce CelloAI, a locally hosted, retrieval-augmented generation (RAG) coding assistant, and a corresponding benchmark suite (CelloAI Benchmarks) designed to evaluate LLMs under realistic HEP/HPC conditions.

Core System Architecture (CelloAI):

• RAG Context: Retrieves both scientific text and source code to ground generation.
• Syntax-Aware Chunking: Preserves semantic boundaries to prevent fragmented retrieval.
• Callgraph-Aware Prompting: Enhances prompts with caller/callee dependency context to respect execution flow.

The Benchmark Suite:
The framework evaluates LLMs across three distinct tracks, emphasizing automated scoring, repeatability, and domain-relevant failure modes.

A. Code Documentation (CelloAI-Doc-Bench)

• Task: Generate Doxygen-style comments (structured documentation) for scientific functions.
• Metrics:
  1. Tag-Based Coverage (F1): Measures structural completeness (precision/recall of @param, @return tags) against ground truth.
  2. Semantic Similarity:
     • Differential Similarity: Checks consistency of parameter descriptions between caller-callee pairs (using cosine similarity of embeddings).
     • Expert Similarity: Compares LLM output against human-written expert documentation.

B. HPC Code Generation (CelloAI-Code-Bench)

• Task: Porting GPU kernels from CUDA to OpenMP within the FastCaloSim (ATLAS Liquid Argon Calorimeter simulation) workflow.
• Workload: Three kernels of increasing complexity:
  1. Reset: Simple array zeroing.
  2. Count: Identifying hits and packing results (requires atomic operations).
  3. Simulate: Complex floating-point computation with memory updates and host-device transfers (highest difficulty).
• Evaluation: Automated pipeline attempts to compile and validate the generated code. Success is binary (pass/fail) based on compilation and execution correctness, not just snippet quality.

C. Graphical Data Analysis (CelloAI-Multimodal-Bench)

• Task: Vision-enabled analysis of synthetic histograms to detect discrepancies between "reference" and "monitored" data.
• Metrics:
  • Outlier Detection: Precision/Recall/F1 for identifying specific anomalous points.
  • Discrepancy Region Identification: Precision/Recall/F1 for identifying intervals of deviation.
• Goal: Evaluate if Vision-LLMs can interpret scientific plots and flag anomalies for further investigation.

3. Key Results

Code Documentation Results

• Structural vs. Semantic: Most modern LLMs (e.g., GPT-oss-120b, Qwen3) achieve very high Recall (~1.0) and F1 (~0.95) for tag coverage, meaning they correctly identify parameters.
• Semantic Limitations: Semantic similarity scores remain moderate (Expert Similarity ~0.57–0.62). Smaller models (Llama-3-7B) struggle significantly with recall.
• Temperature Impact: Higher decoding temperatures slightly reduce semantic consistency.
• CelloAI Impact: Enabling CelloAI's context workflow yields marginal improvements in semantic scores, suggesting that domain-specific fine-tuning is likely required for high-quality scientific descriptions.

Code Generation Results

• Complexity Gradient: Performance drops sharply with kernel complexity.
  • Kernel 1 (Reset): High success rates (up to 10/10 with GPT-oss-120b + CelloAI).
  • Kernel 2 (Count): Moderate success (improved by CelloAI context).
  • Kernel 3 (Simulate): Extremely difficult. Even the best models (GPT-oss-120b) achieved only 2/10 successes.
• Context Matters: CelloAI's enhanced context (callgraphs, patterns) significantly improved success rates on Kernels 1 and 2 compared to baseline RAG, but the simulation kernel remains a major bottleneck for end-to-end correctness.

Graphical Data Analysis Results

• Model Performance:
  • InternVL 3.5 8B: Performed best for outlier detection (F1 ~0.57) at low temperatures.
  • Qwen3-VL: Consistently moderate performance (F1 ~0.5) across settings.
  • Gemma-3n: Failed to detect any ground-truth outliers (F1 = 0).
• General Trend: All models showed moderate performance, indicating that current vision-LLMs are not yet reliable for precise scientific plot analysis without further domain adaptation.

4. Key Contributions

1. Specialized Benchmark Framework: Introduced the first repeatable, automated benchmark suite specifically for HEP/HPC coding tasks, moving beyond generic coding benchmarks.
2. Three-Track Evaluation: Defined methodologies for evaluating documentation quality (structural + semantic), code generation robustness (compilation + execution), and multimodal analysis (plot interpretation).
3. Real-World Validation: Demonstrated that while LLMs are good at structural tasks (tagging, simple code), they struggle with complex scientific constraints (simulation kernels, semantic consistency, plot interpretation).
4. CelloAI System: Validated a retrieval-augmented system that improves code generation reliability by incorporating dependency and callgraph context.

5. Significance and Future Outlook

• Scientific Rigor: The paper establishes that "working code" in HPC requires more than passing unit tests; it demands scientific validation and integration into complex build systems.
• Fair Comparison: The automated scoring framework allows for fair, reproducible comparisons between models, decoding strategies (temperature), and system architectures (RAG vs. CelloAI).
• Future Directions: The authors identify a critical need for domain-adapted (fine-tuned) models to improve semantic understanding and complex kernel generation. Future work will expand the benchmark to include more repositories and fine-tuned models to better represent the diversity of HEP workloads.

In summary, this paper provides a critical foundation for developing reliable, science-aware AI assistants, highlighting that current general-purpose LLMs require significant architectural and training enhancements to be viable for high-stakes scientific computing.