Technical Summary: Distribird

1. Problem Statement

Bayesian calibration of process-based models (e.g., in agriculture, hydrology, or epidemiology) requires specifying a prior distribution for every model parameter. While informative priors—derived from scientific literature—are statistically superior to uninformative uniform priors, they are rarely used in practice. The barrier is not a lack of knowledge, but the prohibitive cost of manually gathering, reading, and synthesizing decades of literature to construct these priors.

Recent Large Language Models (LLMs) offer a potential solution by automating literature search and synthesis. However, direct prompting of LLMs for priors introduces critical flaws for scientific workflows:

• Lack of Traceability: LLMs often generate confident priors without citing sources, making verification impossible.
• Hallucination: Models may fabricate priors for non-empirical parameters (e.g., software tuning constants) or typographical errors.
• Data Privacy: Cloud-based LLMs require transmitting parameter descriptions and unpublished modeling details to third parties.

The core problem is the absence of a scalable, trustworthy, and fully local tool that automates literature-based prior construction while maintaining full provenance and rejecting out-of-scope requests.

2. Methodology: The Distribird System

Distribird is an agentic web application and Python library designed to automate the construction of literature-informed priors. It operates entirely on local hardware using open-weight models, ensuring no data leaves the researcher's machine.

2.1 Core Architecture

The system is built on a LangGraph multi-agent pipeline that orchestrates a stateful workflow with three feedback loops and conditional routing. The pipeline processes a ParameterInput (name, description, unit, domain context, and physical bounds) through the following stages:

1. Enrichment & Validity Check:

   • The system first attempts to recognize the parameter. If the LLM does not recognize the name or determines the quantity is non-empirical (e.g., a latent variable), it triggers an Early-Skip mechanism.
   • This bypasses the expensive search and extraction phases (saving 80–95% of runtime) and classifies the request as Likely_Invalid or Suspicious, preventing the generation of unfounded priors.

2. Progressive Search Strategy:

   • Scope Planning: A node classifies the request's specificity (high, medium, low) to determine an initial search breadth (strict, mixed, or broad).
   • Parallel Search: The system queries Semantic Scholar and OpenAlex in parallel for open-access papers. Optional agents can perform deep web searches, with results cross-verified to prevent hallucinated citations.
   • Domain Broadening (Loop B): If the initial strict search yields insufficient relevant values, the system automatically widens the search scope (e.g., from "maize in Central Europe" to "maize globally") and re-runs the search, accumulating evidence rather than discarding previous results.

3. Full-Text Retrieval & Extraction:

   • Tiered Fetching: The system attempts to retrieve full texts via a cascade of sources (URL resolvers, open-access mirrors, DOI converters) before resorting to a heavy, opt-in stealth browser (Camoufox) for paywalled content.
   • Structure Preservation: PDFs are converted to Markdown to preserve table structures, ensuring numerical values are extracted with their associated units and row/column labels intact.
   • Extraction: The LLM extracts numerical values from the full text.

4. Relevance-Aware Synthesis:

   • Relevance Judging: Each extracted value is scored by an LLM for domain relevance (High, Medium, Low).
   • Quality Gate: The pipeline evaluates the quantity and quality of extracted values. If evidence is thin, it triggers Search Refinement (Loop A) or Extraction Refinement (Loop C) to find better data.
   • Distribution Fitting:
     • ≥5 Values: Fits five candidate distributions (Normal, Truncated Normal, Gamma, Log-Normal, Beta) and selects the best via Akaike Information Criterion (AIC).
     • 2–4 Values: Uses moment matching with a widened standard deviation to prevent overfitting.
     • 1 Value: Centers a Truncated Normal on the value with uncertainty derived from the source.
     • 0 Values: Falls back to a wide, uninformative Truncated Normal.
   • Weighting: Values are weighted by sample size and relevance score (High=1.0, Medium=0.6, Low=0.25).

5. Output & Provenance:

   • The system outputs a PipelineResult containing the fitted distribution, confidence level, and a complete provenance chain (search queries, retrieved papers, extracted values, and fitting rationale).
   • Formats include JSON, executable Python (SciPy), and R scripts.

2.2 Technical Constraints

• Local Execution: All LLM calls use open-weight models (Qwen3.6 27B, Gemma 4 31B, Mistral Small 4 119B) quantized for local GPU execution (e.g., NVIDIA H100).
• Budget Control: An IterationBudget limits total LLM calls (default 30) to ensure termination.
• Reproducibility: Fixed seeds and task-specific temperature settings ensure deterministic outputs for the same input.

3. Key Contributions

1. Automated Literature-Informed Priors: Distribird is a system that fully automates the pipeline from parameter description to a fitted probability distribution using scientific literature, specifically targeting process-based models.
2. Validity Guardrails: Unlike standard LLMs, Distribird explicitly detects and rejects out-of-scope requests (typos, non-empirical parameters) via an early-skip mechanism and a terminal validity classifier, preventing the generation of "confident but unfounded" priors.
3. Full Provenance: Every prior is traceable to specific papers and extracted values. The system exports a complete audit trail, addressing the "black box" nature of LLM-generated science.
4. Local, Privacy-Preserving Operation: The entire workflow runs on local hardware with open-weight models, ensuring that unpublished modeling details are never transmitted to third-party providers.
5. Progressive Search Logic: The system dynamically adjusts search breadth based on initial evidence, balancing the need for specificity with the need to find sufficient data.

4. Evaluation Results

The authors evaluated Distribird on 24 parameters across ten scientific domains using three local open-weight models, comparing the full pipeline against a single-prompt LLM baseline (a direct request for a prior without literature search).

• Prior Quality (Accuracy): The full pipeline matched the single-prompt baseline in terms of point-estimate accuracy but did not exceed it. The authors note this is expected, as both methods ultimately rely on the same underlying knowledge base (the LLM's training data vs. the literature it retrieves).
• Robustness to Hallucination: The full pipeline successfully declined to produce priors for all five out-of-scope test parameters (fabricated or non-empirical). In contrast, the single-prompt baseline generated confident, unfounded priors for these parameters in 11 of 30 model–parameter cases.
• Traceability: Every prior produced by Distribird was linked to specific sources, whereas the baseline returned priors with no traceable basis.
• Local Operation: The system successfully ran entirely on local hardware without transmitting sensitive data.

5. Significance and Claims

The paper argues that for scientific workflows, provenance, robustness against fabrication, and data privacy are more critical than marginal improvements in point-estimate accuracy.

Distribird does not claim to be a "smarter" estimator than a well-prompted LLM. Instead, it claims to be a trustworthy tool. Its significance lies in making literature-based prior construction:

• Scalable: Automating a process that previously took days of manual reading.
• Auditable: Providing a defensible chain of evidence for every prior.
• Safe: Preventing the use of priors for parameters that lack empirical support.
• Private: Enabling the use of proprietary or unpublished model details without exposing them to external AI providers.

The authors conclude that these properties make Distribird suitable for serious scientific work, where the ability to verify and defend a prior is a prerequisite for its use.