⚛️ high-energy experiments

\textbf{\textsc{HepUKE}}: Uncertainty-aware Knowledge Evolution for High Energy Physics Analysis Agents

This paper introduces HepUKE, an uncertainty-aware knowledge evolution system for High Energy Physics analysis agents that quantifies evidence confidence via information entropy to enable self-evolving DSL knowledge bases and significantly reduces retrieval rounds while maintaining answer quality.

Original authors: Zijie Shang, Ke Li, Junkun Jiao, Shengchun Wang, Changzheng Yuan, Xiongfei Wang, Beijiang Liu, Weimin Song, Xiaoshen Kang

Published 2026-10-12

📖 4 min read🧠 Deep dive

CC BY 4.0

Original authors: Zijie Shang, Ke Li, Junkun Jiao, Shengchun Wang, Changzheng Yuan, Xiongfei Wang, Beijiang Liu, Weimin Song, Xiaoshen Kang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

High energy physics is the science of smashing particles together at incredible speeds to see what they are made of and how they behave. To make sense of the billions of collisions that happen every second, scientists rely on massive computers and complex software to sort the data, filter out the noise, and find the rare events that reveal new laws of nature. This process requires a deep understanding of the equipment, the math, and the specific rules of the game. In recent years, researchers have begun asking if artificial intelligence can help. They have built digital assistants that can read scientific papers and write the code needed to analyze data. However, these assistants often struggle because they treat every piece of information they find as equally true. If the computer reads a document that sounds similar to the one it needs but actually describes a different experiment or uses a different method, the assistant might copy the wrong instructions. This can lead to serious errors in the final results, because in this field, a small mistake in the code can change the entire physical conclusion.

A team of researchers has developed a new system called HEPUKE to fix this problem. Instead of just looking for documents that sound similar to the question, this system acts like a cautious expert who checks the reliability of every piece of advice it receives. The researchers realized that in high energy physics, there are two types of uncertainty. The first is statistical, which comes from the natural randomness of how a computer generates an answer. The second is systematic, which comes from the choices made in the method itself, such as which specific tools or settings were used. Most current AI systems ignore the second type. HEPUKE combines both types of uncertainty into a single measure of confidence. When the system finds a potential example to copy, it asks: "If I use this example, does my answer become more certain, or does it become confused?" If the example introduces confusion or relies on a method that doesn't fit the current task, the system rejects it.

The system works by building a living library of expert knowledge. It starts with a collection of proven analysis programs written by human scientists. When a new task comes in, the system searches this library for the best match. But unlike older systems that would simply grab the top result, HEPUKE tests the match. It generates several possible solutions based on that match and checks how consistent they are. If the solutions vary wildly, it means the match is poor or the method is unstable, so the system discards it. If the solutions are consistent and the uncertainty is low, the system accepts the advice. Crucially, if the system successfully solves a new problem that wasn't in the library before, it adds that new solution to the library for future use. This allows the system to grow smarter over time, but only by adding solutions that have been rigorously checked.

To make this process efficient, the system includes a smart stop button. In many scientific databases, the same information is repeated in many different documents. A standard computer might keep searching through page after page, wasting time and money, even after it has found the answer. HEPUKE watches the flow of information as it searches. It knows when it has gathered enough reliable evidence and when further searching is just finding the same things again. It stops the search at that point, saving a significant amount of time and computing power. In tests using data from the Beijing Spectrometer III experiment, the system reduced the number of search steps by 43 percent while keeping the quality of the answers almost the same. It also worked well on a general knowledge test that had nothing to do with physics, showing that the method is robust.

The researchers found that this approach is particularly good at handling the specific, rigid rules of particle physics. They discovered that the most important part of the code to check is the section where the scientist decides which events to keep and which to throw away. This is where the most uncertainty usually lies. By focusing on this part, the system can tell the difference between a helpful example and a misleading one. The study shows that by treating uncertainty as a tool rather than a bug, AI assistants can become much more reliable partners for scientists. They can learn from their mistakes, avoid copying the wrong methods, and help researchers get to the answers faster without sacrificing accuracy. This work suggests that the future of scientific discovery will not just be about having more data, but about having smarter ways to trust the information we find.

Technical Summary: HEPUKE – Uncertainty-Aware Knowledge Evolution for High Energy Physics Analysis Agents

Problem Statement

High Energy Physics (HEP) analysis requires agents to synthesize complex software workflows (Domain-Specific Languages or DSLs) based on few-shot precedents. Existing Retrieval-Augmented Generation (RAG) systems for HEP face three critical limitations:

Misleading Relevance: Standard semantic similarity treats all retrieved documents as equally trustworthy. In HEP, documents may share terminology but differ in physical relevance, measurement types, or trustworthiness, leading to systematic bias.
Static Knowledge: Current systems do not iteratively retain validated precedents or estimate the confidence of stored evidence, preventing the knowledge base from evolving based on successful analyses.
Uncalibrated Uncertainty: Existing uncertainty quantification methods primarily capture statistical fluctuations (output variance under fixed evidence) but fail to account for systematic uncertainty arising from methodological choices in retrieved evidence. Furthermore, retrieval loops often lack adaptive stopping mechanisms, leading to redundant calls in highly repetitive HEP corpora.

Methodology: HEPUKE Framework

The authors propose HEPUKE (Uncertainty-Aware Knowledge Evolution), a system that fuses statistical uncertainty from Large Language Model (LLM) generation with systematic uncertainty from HEP methodological choices. The framework operates on a shared knowledge base $K = K_{QA} \cup K_{DSL}$ , where $K_{DSL}$ is an evolving library of DSL programs (specifically HepScript for the BESIII experiment).

1. Uncertainty-Aware Confidence Estimator

HEPUKE quantifies the reliability of a retrieved precedent by measuring its ability to reduce uncertainty in the generated DSL.

Statistical Uncertainty ( $H_{stat}$ ): Computed via token-level predictive entropy from the LLM.
Systematic Uncertainty ( $H_{syst}$ ): Computed via structural disagreement (Jensen-Shannon Divergence) across semantically equivalent DSL generation paths. This captures sensitivity to methodological choices (e.g., selection cuts, fit configurations).
Scoring: The system calculates the reduction in entropy ( $\Delta H$ ) when a candidate precedent is used versus a baseline. A candidate score is formed by weighting the reduction in full-code entropy and selection-module entropy:
$score(q, k) = \Delta H_{stat}(q, k) + \alpha \Delta H_{full}(q, k) + (1 - \alpha) \Delta H_{sel}(q, k)$
The confidence $c(q)$ is derived from this score using a sigmoid function.

2. Self-Evolving DSL Knowledge Base

The system employs a confidence-gated, append-only mechanism for knowledge evolution.

Dual Gating: A synthesized program is admitted to the knowledge base only if it passes:
1. Validation Gate: Successful static parsing and dry-run execution.
2. Confidence Gate: The program must exhibit a specific "sweet spot" of uncertainty reduction. The authors observe an inverted-U relationship where very high uncertainty reduction indicates inappropriate copying of precedent-specific values (transcription tail), while low reduction indicates conflict. Only programs with moderate, positive uncertainty reduction are admitted.
Provenance: Each entry retains source chunks and synthesis history, allowing tracing back to human-written seeds.

3. Early-Stopping Controller

To address redundancy in HEP corpora, HEPUKE introduces an adaptive stopping controller that determines when to terminate retrieval rounds.

Mechanism: It fuses a backward stationarity test (based on Bayesian Online Changepoint Detection, BOCPD) on the answer posterior with a forward estimate of marginal information gain.
Decision: Retrieval stops when the posterior is sufficiently stable and the expected information gain from another round is low, preventing over-retrieval without requiring reflection tokens or retraining.

Key Contributions

Unified Uncertainty Signal: The paper proposes a method to fuse statistical uncertainty (LLM generation variance) and systematic uncertainty (methodological choices in evidence) into a single confidence signal for reranking and knowledge admission.
Self-Evolving Knowledge Base: It introduces a mechanism for the continuous, confidence-gated expansion of a domain-specific knowledge base, moving beyond static retrieval indices.
Information-Theoretic Early Stopping: The authors develop a controller that reduces redundant retrieval in highly repetitive scientific corpora by combining stationarity detection with information gain estimation.
BESIII Benchmark: The work releases a large-scale, expert-annotated benchmark (BESIII-PhysicsQA) containing 1,000 question-answer pairs and a library of 700 expert-authored HepScript programs.

Experimental Results

Experiments were conducted on the BESIII-PhysicsQA benchmark (in-domain) and HotpotQA (out-of-domain).

Confidence vs. Relevance: Uncertainty-aware confidence selection outperformed standard relevance-based retrieval. On the BESIII-DSL task, the best confidence configuration ( $\alpha=0.0$ ) achieved 76.08% strict match and 65.02% physics accuracy, compared to 74.94% and 62.46% for relevance-based top-1 selection.
Knowledge Evolution: The "interval gate" (admitting only moderate uncertainty reduction) improved the quality of admitted programs from 0.93 to 0.95, effectively rejecting the "transcription tail" where models blindly copy precedent-specific values.
Efficiency: On the BESIII-PhysicsQA benchmark, the early-stopping controller reduced retrieval rounds by 43.0% (from 5.23 to 2.98) and total tokens by 44.9%, while preserving 95.5% of the native answer accuracy (a drop of only 2.27 percentage points). On HotpotQA, the controller reduced retrieval time by 18.2% with a negligible accuracy drop (0.76 points in F1).
Robustness: The confidence gating remained effective under perturbations to retrieval depth ( $k=10$ ) and prompt paraphrasing, showing particular robustness in physics-sensitive evaluation metrics.

Significance

The paper claims that HEPUKE addresses the specific needs of scientific agents where "strategic guidance" is more valuable than "ready-made answers." By treating uncertainty not just as a generation artifact but as a signal for methodological reliability, the system enables a self-correcting, evolving knowledge base. The results suggest that for high-stakes, redundant domains like HEP, controlling for systematic uncertainty and employing adaptive stopping yields a superior cost–accuracy trade-off compared to standard RAG pipelines. The framework is designed to be generator-agnostic and applicable to any HEP experiment utilizing structured DSLs.