High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuning

1. Problem Statement

Large Language Models (LLMs) are currently trained to respond to almost every prompt, regardless of whether they possess the necessary knowledge or reasoning capabilities. This behavior leads to hallucinations (generating incorrect facts or reasoning steps), which poses significant risks in high-stakes domains like medicine, law, and coding.

Existing methods often attempt to fix this by post-processing outputs or training models to either answer fully or abstain completely. However, there is a lack of methods that allow models to partially abstain—providing only the information they are confident in while explicitly marking or omitting uncertain parts. The core challenge is balancing response completeness (Recall) with response correctness (Precision) without requiring complex inference-time computations.

2. Methodology: HALT

The authors propose HALT (High Accuracy, Less Talk), a post-training paradigm that aligns an LLM's outputs strictly with its pre-trained capabilities. The core philosophy is that LLMs do not acquire new capabilities during fine-tuning; they only learn to better utilize existing knowledge. Therefore, fine-tuning on data the model cannot reliably generate increases hallucinations.

The HALT pipeline consists of four key steps:

A. Preliminary Response Generation

For a given prompt $x_j$ and ground truth $y_j$, the pretrained model $M$ is used to generate an initial response via few-shot prompting. This response serves as a baseline to estimate what the model can currently generate.

B. Fragmentation

The preliminary response is split into factual fragments. The fragmentation strategy depends on the domain:

• Independent Fragments: Used for domains like biographies or lists. Fragments are atomic, independently verifiable statements (e.g., "Obama was born in Hawaii").
• Causally Dependent Fragments: Used for domains like math, coding, or sequential reasoning. Fragments are split at natural boundaries (e.g., new lines, equation steps), where the correctness of a step depends on previous steps.

C. Correctness Assessment

An Evaluator LLM (specifically Llama3-405B in the experiments) assesses the correctness of each fragment against ground truth information (e.g., Wikipedia articles or step-by-step solutions).

• For independent fragments: Each is assessed individually. Incorrect fragments are flagged.
• For causally dependent fragments: The evaluator identifies the first incorrect fragment. All subsequent fragments are considered incorrect because the logical chain is broken.

D. Constructing the Capability-Aligned Dataset

The final training response ($y^H_j$) is constructed based on the assessment:

• Independent Fragments: Incorrect fragments are simply removed.
• Dependent Fragments: The first incorrect fragment and all subsequent fragments are replaced with a single marker: "Unsure from here".

E. Tuning the Trade-off ($\alpha$)

HALT introduces a tunable threshold to control the balance between completeness and correctness. Instead of using a single greedy response, the system samples $N$ preliminary responses. It selects the response at the $\alpha$-percentile of correctness:

• Low $\alpha$ (Conservative): Selects responses with fewer correct fragments, resulting in a model that abstains more often but has very high correctness.
• High $\alpha$ (Aggressive): Selects responses with more correct fragments, increasing completeness but potentially lowering precision.

3. Key Contributions

1. Capability-Aligned Fine-Tuning: A novel method that trains models only on content they are capable of generating, preventing the model from learning to hallucinate on unknown concepts.
2. Partial Abstention Mechanism: Unlike methods that force a binary "answer or silence" choice, HALT allows models to output partial answers, stopping exactly where confidence drops and marking the rest as "Unsure from here."
3. Controllable Trade-off: The $\alpha$ parameter allows practitioners to explicitly tune the model for specific deployment needs (e.g., high precision for medical diagnosis vs. high recall for creative writing).
4. No Inference-Time Overhead: Unlike methods requiring sampling, verification probes, or post-processing at test time, HALT produces a standard fine-tuned model that generates reliable outputs directly.

4. Experimental Results

The authors evaluated HALT on four open-source models (Llama3-8B, Llama3-70B, Gemma2-9B, Mistral-7B) across four domains: Wikipedia Biographies, Mathematics (MATH), Medical QA (MedExQA), and Coding (APPS).

• Correctness vs. Completeness: HALT effectively trades completeness for correctness. By tuning for highest correctness ($\alpha=40\%$), the Llama3-70B model achieved 87% correctness across all four domains, compared to 51% with standard fine-tuning.
• F1 Score: Across all settings, HALT improved the F1 score (harmonic mean of precision and recall) by an average of 4% compared to baselines like FactTune, IDK, and standard fine-tuning.
• Single Reliable Model: A single model trained on a mix of all four domains using HALT achieved 87% accuracy while maintaining 53% response completeness (relative to standard fine-tuning).
• General Instruction Following: Evaluation via AlpacaEval showed that HALT models maintained near-parity with base models in general instruction following, with only modest degradation in tasks where judges penalize abstention (e.g., math reasoning).
• Add-On Variant: The authors demonstrated a variant where uncertain fragments are marked as "Uncertain" rather than omitted. This retains full response length while still informing users of potential errors, achieving similar correctness gains when the uncertain parts are excluded from evaluation.

5. Significance

HALT addresses a critical limitation in current LLM deployment: the inability to distinguish between what a model knows and what it is guessing.

• Safety & Reliability: It provides a practical mechanism for deploying LLMs in high-stakes environments where hallucinations are unacceptable.
• Efficiency: It shifts the cost of reliability to the training phase (data generation) rather than the inference phase, making it scalable for production.
• Theoretical Insight: The results reinforce the hypothesis that fine-tuning does not expand a model's fundamental capabilities but rather refines its ability to utilize pre-trained knowledge. Forcing models to answer beyond their capabilities is counter-productive.

In summary, HALT offers a robust framework for creating "reliable" LLMs that know when to stop talking, significantly reducing hallucinations while maintaining high utility.