Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory: A Case Study in Spectral Reconstruction

Technical Summary: Quantifying the Reconstructability of Astrophysical Methods with Large Language Models and Information Theory

Problem Statement
Modern astrophysical research relies on complex data analysis pipelines, yet published descriptions often lack the granularity required for computational reproducibility. While open-source practices have improved, sharing code alone does not guarantee reproducibility due to hidden assumptions, specific configurations, and the reliance on tacit expert knowledge. Current literature lacks a quantitative framework to measure how effectively a scientific method can be reconstructed from its written description alone. This study addresses the gap between explicit textual documentation and the implicit "algorithmic ellipsis" required to execute a method, asking whether Large Language Models (LLMs) can serve as diagnostic tools to audit methodological transparency.

Methodology
The authors propose an information-theoretic framework to quantify the reconstructability of scientific methods. The core hypothesis treats algorithmic reconstruction as an inference problem where an LLM generates a probability distribution of candidate algorithms ($\hat{Y}$) given a text input ($X$).

1. Information-Theoretic Framework:

   • Entropy ($H$): Shannon entropy is used to measure the uncertainty of the algorithmic hypothesis space. High entropy indicates a broad range of plausible implementations, while low entropy suggests the text strongly constrains the solution. The authors distinguish between lexical entropy (diversity of keywords/method names) and semantic entropy (diversity of structural implementation logic).
   • Mutual Information (MI): Used to quantify the information gain provided by adding text layers (Title $\to$ Abstract $\to$ Full Methods), measuring how much the text reduces uncertainty about the method.
   • Alignment Metrics: Jensen–Shannon Divergence (JSD) and Cosine Distance measure how closely the distribution of LLM-generated solutions aligns with the "Ground Truth" (the original authors' specific implementation).
   • Residual Information ($K_{res}$): The divergence between the LLM's distribution given full text and the Ground Truth, quantifying the "tacit knowledge" gap.

2. Experimental Design:

   • Case Study: The framework is applied to a specific astrophysical problem: probabilistic spectral reconstruction of Trans-Neptunian Objects (TNOs) from sparse photometry. This method involves Principal Component Analysis (PCA), Kernel Density Estimation (KDE), and machine learning (GBDT/AutoGluon).
   • Experiment 1 (Diagnostic Analysis): Local LLMs (DeepSeek-R1 and GPT-oss) generate 200 candidate algorithmic descriptions per information state (Title, Title+Abstract, Full Methods). These are clustered in lexical (TF-IDF) and semantic (embedding) spaces to calculate entropy and alignment metrics.
   • Experiment 2 (Functional Stress Test): Frontier LLMs (Gemini, ChatGPT, Claude) are tasked with generating executable Python code based on the same text hierarchies. The code is evaluated using a "Hierarchy of Scientific Validity" ($V$):
     • $V=0$: Invalid/Non-physical.
     • $V=1$: Unconstrained/Overfitting (mathematically valid but physically unsound).
     • $V=2$: Functional/Overconfident (correct framework but missing implicit constraints).
     • $V=3$: Calibrated/Scientific (fully aligned with ground truth including tacit knowledge).

Key Results

1. The Entropy Floor: Increasing textual detail (from Title to Full Methods) successfully reduces lexical entropy and shifts the distribution toward the correct conceptual framework (macrostate). However, semantic entropy does not collapse to zero. A persistent "entropy floor" remains, indicating that even with full text, multiple structurally diverse implementations are consistent with the description.
2. Decoupling of Information Gain and Compression: While adding text increases Mutual Information regarding the concept (e.g., identifying PCA and KDE), it fails to compress the implementation space. The text reorganizes probability mass toward the correct region but does not eliminate structural variance.
3. The Gap in Scientific Validity:
   • Title Only ($V=0$): Models default to generic priors (e.g., Gaussian Processes) and fail to capture the domain-specific problem structure.
   • Abstract ($V=1$): Models identify the core idea (PCA) but fail to constrain the solution space, leading to unconstrained overfitting and physically impossible outputs (e.g., negative flux).
   • Full Methods ($V=2$): Models successfully reconstruct the functional pipeline (PCA + KDE + ML) and produce executable code. However, they systematically fail to infer tacit expert knowledge, such as the necessity of modeling latent covariance (e.g., via Gaussian Copulas) to produce calibrated uncertainty estimates.
   • Ground Truth ($V=3$): Only the original implementation, which includes these implicit constraints, achieves full scientific calibration.

Significance and Claims
The paper claims that LLMs can be repurposed as "zero-shot diagnostic tools" to audit the transparency of scientific literature. The study demonstrates that:

• Textual Limitations: Published manuscripts often fail to transmit the "integrated physical intuition" required for strict scientific calibration. The gap between $V=2$ (functional) and $V=3$ (calibrated) represents information that is rarely formalized in print.
• Macrostate vs. Microstate: Reproducibility should be viewed as converging to a shared algorithmic macrostate (conceptual framework) rather than an identical microstate (specific code). Text is effective at defining the macrostate but insufficient to uniquely determine the microstate.
• Practical Application: The authors propose a "Zero-Shot Stress Test" where authors use LLMs to attempt blind reconstruction of their own methods prior to submission. This process can identify missing structural constraints and hidden assumptions that human reviewers might overlook due to shared domain intuition.
• Future Standards: The paper suggests that alongside source code, authors should provide "reproducibility prompt sets" that explicitly encode the macrostate and essential tacit constraints, enabling LLMs to act as compilers that translate shared scientific intent into functional, environment-specific implementations.

The study concludes that while LLMs can recover core methodologies, they expose the profound extent to which functional science relies on tacit knowledge, suggesting that current textual descriptions are often insufficient for full computational reproducibility without explicit formalization of implicit expert decisions.