Technical Summary: Zero-Shot Active Feature Acquisition via LLM-Elicitation

1. Problem Definition

The paper addresses Active Feature Acquisition (AFA), a sequential decision task where an agent must select which features to observe under a limited budget to reach a classification or ranking decision. The central bottleneck in existing AFA methods is their reliance on large, labeled datasets to train probabilistic models that guide the acquisition process. This requirement fails in regimes where AFA is most valuable: rare diseases, newly characterized conditions, or atypical patient cases where labeled data is scarce or non-existent.

The authors propose Zero-Shot AFA, a framework that operates without task-specific labeled data. They leverage Large Language Models (LLMs) as a source of unsupervised domain knowledge to construct the necessary probabilistic models. However, the paper identifies two fundamental obstacles in using LLMs directly for AFA:

1. Planning Deficiency: LLMs are poor sequential planners. They tend to pattern-match heuristically rather than reasoning about the cascading, non-local implications of feature correlations required for optimal ordering.
2. Calibration Issues: LLMs are poorly calibrated as formal probabilistic objects, making them unreliable for the white-box modeling (conditioning, marginalization, optimization) required by AFA.

The goal is to bridge this gap by strictly separating roles: the LLM provides knowledge (elicitation), while a classical algorithm handles the sequential planning.

2. Methodology

2.1 Discriminative MRF Elicitation

Instead of asking the LLM to generate full class-conditional distributions $p(x|H)$ and $p(x|\neg H)$, which is prone to error and ambiguity, the framework elicits only discriminative statistics sufficient for a Markov Random Field (MRF). Specifically, the LLM is queried for:

• Unary deviations: How a feature $j$ deviates under hypothesis $H=1$ relative to $H=0$ ($\Delta \alpha_j = \alpha_{j,1} - \alpha_{j,0}$).
• Pairwise co-variations: How features $j$ and $l$ co-vary differentially under the two hypotheses ($\Delta \psi_{jl} = \psi_{jl,1} - \psi_{jl,0}$).

These parameters define a discriminative MRF where the score for an entity $v$ is the log-ratio:
$$S_v(x) = \sum \Delta \alpha_j x_j + \sum \Delta \psi_{jl} x_j x_l$$
This approach avoids the need to model absolute class distributions, aligning with the discriminative nature of medical literature.

2.2 Resolving Gauge Ambiguity (Binary Case)

A critical methodological challenge arises in the binary classification setting: the elicited log-ratio $S_v$ fixes the ratio $p_1/p_0$ but leaves the "gauge" (the absolute scale of $p_0$ and $p_1$) undefined. Classical AFA criteria (like Conditional Mutual Information) require per-class predictive distributions, which cannot be computed from the ratio alone.

The authors resolve this via a Maximum-Entropy (MaxEnt) closure. Among all pairs $(p_0, p_1)$ consistent with the elicited log-ratio, the framework selects the pair that maximizes the sum of entropies $H(p_0) + H(p_1)$. This yields a principled, unique reconstruction of the class-conditionals:
$$p_e(x) \propto \exp\left( (-1)^{e-1} S_v(x) / 2 \right), \quad e \in \{0, 1\}$$
This closure allows the application of standard information-theoretic acquisition criteria in the zero-shot setting.

2.3 Acquisition Policies

The framework operates on two settings: Binary Classification and Top-k Identification (selecting the $k$ most supported hypotheses from $N$ candidates).

• Preference Scores: To decide between hypotheses (or entities), the framework uses "dueling" scores.
  • Wald Accumulator: A gauge-free, deterministic sum of observed feature energies. It ignores unobserved features but is exact at full observation.
  • Mean-Field Scores (Chernoff-Wald / Sign-Vote): These integrate over unobserved features using the reconstructed class-conditionals (via MaxEnt for binary, direct elicitation for top-k). They utilize the Expected KL divergence or Sign-Vote (stack-A) to score features, providing a "soft" prediction that accounts for the unobserved subgraph.
• Feature Selection: The framework evaluates features based on Conditional Mutual Information (CMI) or a novel F-target gain (expected improvement in the preference score).
• Top-k Allocation: For ranking $N$ entities, the authors introduce a Priority-Weighted Allocation rule. Instead of evaluating all $\binom{N}{2}$ pairs, entities are clustered in MRF parameter space. The budget is allocated to "duels" (pairwise comparisons) that are:
  1. Near the top-$k$ frontier (high influence on the final ranking).
  2. Not yet resolved (uncertain winner).
     This reduces the combinatorial space from $O(N^2)$ to $O(K)$, where $K$ is the number of clusters.

3. Key Contributions

1. First Zero-Shot AFA Framework: The paper presents the first method for active feature acquisition that requires no task-specific labeled data, RL episodes, or meta-training, relying entirely on LLM-elicited knowledge.
2. Discriminative Elicitation & MaxEnt Closure: A novel protocol for extracting only sufficient statistics from LLMs and a principled MaxEnt closure to resolve the resulting gauge ambiguity, enabling the use of classical AFA criteria.
3. Dual-Track Planning for Chaotic Cases: The framework employs both a deterministic (Wald) and a probabilistic (Mean-Field) scoring track. The authors demonstrate that for "chaotic" patients (where early observations are locally inconsistent with the latent label), the Mean-Field track, anchored by the elicited MRF structure, significantly outperforms the Wald accumulator.
4. Priority-Weighted Top-k Policy: A novel allocation strategy for top-k identification that clusters entities and prioritizes budget toward boundary pairs, significantly improving ranking accuracy over greedy or random baselines.
5. Quantification of the Knowledge-Planning Gap: By comparing a formal planner using LLM-derived models against an in-context LLM planner given the same models, the study isolates the "planning" deficit of LLMs. The formal algorithm achieves the same accuracy with far fewer observations, proving that the bottleneck is planning, not knowledge.

4. Experimental Results

The framework was evaluated on a real-world clinical cohort of 167 Inflammatory Bowel Disease (IBD) patients with 216 gene-expression features and 25 clinical phenotypes. The MRF parameters were elicited by an LLM blind to this specific cohort.

• Binary Classification:
  • Against the full-observation likelihood (the model's own ground truth), the formal planner (using MaxEnt closure) reached 100% accuracy with full observation and significantly outperformed random ordering early on (e.g., +30.6 percentage points at $t=10$).
  • Against clinical labels, the planner saturated at ~65% accuracy, reflecting an irreducible mismatch between the LLM-elicited model and clinical reality (biological error).
  • LLM Baselines: Direct LLM baselines (MRF-LLM and RAG-LLM) given the same elicited parameters performed significantly worse (0.24–0.46 accuracy) than the formal planner, confirming the necessity of separating knowledge from planning.
• Top-k Identification:
  • Sign Accuracy: On "chaotic" patients (those with unstable decision trajectories), the linearity (mean-field) score outperformed the Wald accumulator by 34.1 percentage points at $t=2$, narrowing only as more data was observed.
  • Ranking Accuracy: The priority-weighted allocation rule significantly outperformed random, greedy, and clustered baselines. At $t=150$, it achieved 0.864 sign accuracy, a 4.7 percentage point lift over the best non-priority baseline.
  • Top-5 Ranking: The framework approached the irreducible ceiling set by the MRF-clinical mismatch, significantly outperforming LLM baselines which failed to utilize the elicited structure effectively.

5. Significance and Claims

The paper claims to establish a mathematically rigorous foundation for zero-shot AFA by strictly separating the LLM's role (knowledge elicitation) from the algorithm's role (sequential planning).

• Shift in Bottleneck: The authors argue that the fundamental bottleneck in zero-shot AFA has shifted from data scarcity to knowledge fidelity. The formal planner is capable of saturating the available biological signal; the remaining gap between the model's performance and clinical ground truth is due to the limitations of the LLM's elicited knowledge (hallucinations or mechanistic simplifications), not the planning algorithm.
• Handling Heterogeneity: The framework demonstrates robust adaptability to patient heterogeneity, specifically managing "chaotic" cases where standard biological expectations fail, by leveraging the structural priors of the elicited MRF.
• Practical Utility: The work provides a viable path for automated diagnostic support in rare disease settings where labeled data is unavailable, offering a principled alternative to human expert interrogation or data-hungry machine learning models.

The authors remain modest, acknowledging that the framework relies on the fidelity of LLM elicitation and that current LLMs cannot yet provide calibrated continuous magnitudes, necessitating the discretization of observations into ternary values.