"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation

Technical Summary: "I Don't Know" – Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation

Problem Statement
The paper addresses the challenge of establishing "appropriate trust" in AI systems, specifically Large Language Models (LLMs). While LLMs offer human-level communication, they suffer from hallucinations (factually incorrect outputs) and "sycophancy" (over-aligning with user preferences at the expense of accuracy). A critical issue is that LLMs often express over-confidence in their answers, even when the question is non-objective or the context is insufficient. This over-confidence prevents users from accurately judging the truthfulness of the system. The authors argue that the human value of benevolence—defined here as the need for honesty, reliability, and dependability—can be met if an AI system can monitor its own knowledge, reflect on its uncertainty, and communicate this self-reflection to the user.

Methodology
The authors propose CERTA (Certainty Enhanced RAG for Trustworthy Answers), a specialized Retrieval Augmented Generation (RAG) system designed to incorporate self-reflected certainty into its responses.

1. RAG Triad Certainty Modeling:
   Instead of relying on the LLM to self-assess its confidence (which can be biased), CERTA calculates uncertainty based on the semantic similarity between three components of the RAG pipeline, known as the RAG Triad:

   • Question-Context Relevance (QCR): Measures if the retrieved context is semantically aligned with the user's question.
   • Context-Answer Relevance (CAR): Measures if the generated answer is supported by the retrieved context (groundedness).
   • Answer-Question Relevance (AQR): Measures if the generated answer semantically addresses the original question.

   These scores are calculated using cosine similarity on embeddings (OpenAI's text-embedding-3-small). These numerical scores are then translated into textual "certainty instructions" provided to the LLM generator.

2. System Approaches:
   The paper evaluates four distinct approaches using two LLMs (OpenAI's GPT-4.1 and Mistral-Small-3.1):

   • RAG (Baseline): Standard RAG without specific instructions on certainty.
   • CERTA-0: Adds a general instruction to the prompt to be honest about the certainty of the answer.
   • CERTA-1: Incorporates the QCR score and instruction into the prompt, informing the model of the relevance between the question and the retrieved context.
   • CERTA-2: A multi-step approach. It first generates an intermediate answer, calculates all three Triad scores (QCR, CAR, AQR), and then prompts the model again with the question, context, intermediate answer, and all three certainty instructions to produce a final, self-reflective response.

3. Certainty Benchmark:
   To evaluate these methods, the authors created a dataset of 90 question-context pairs derived from the TrustLLM Benchmark. The dataset covers four categories:

   • Factuality: Factual questions (some unanswerable).
   • Personal Preference: Subjective questions (e.g., "What is better?").
   • Sycophancy: Questions designed to test if the model agrees with a user's stated (potentially flawed) viewpoint.
   • Moral Choices: Ethical dilemmas.

   Each question is paired with three types of contexts: relevant (contains the answer), incomplete (lacks the specific answer), and irrelevant (contains no answer).

Key Contributions

1. The CERTA System: A novel extension to RAG that utilizes the RAG Triad to calculate and communicate uncertainty both numerically and verbally. It includes a dashboard allowing users to select behaviors for low-certainty scenarios (e.g., default response, "I don't know," or fallback to LLM knowledge).
2. The Certainty Benchmark: A curated dataset of 90 non-objective question-context pairs across four value-based categories and three context types, designed specifically to test an AI's ability to express uncertainty appropriately.
3. Empirical Evaluation: A comparative study of baseline RAG against three CERTA configurations using two distinct LLMs.

Results
The experiments indicate that CERTA successfully shifts system behavior toward appropriate uncertainty:

• "I Don't Know" Frequency: CERTA-1 and CERTA-2 produced significantly more "I don't know" responses compared to the baseline RAG, particularly when contexts were incomplete or irrelevant.
• Factuality: With relevant contexts, systems answered appropriately. With irrelevant contexts, baseline RAG occasionally hallucinated using internal knowledge, whereas CERTA systems remained strictly objective and expressed uncertainty.
• Personal Preference: Baseline RAG occasionally provided confident answers to subjective questions. CERTA approaches correctly identified the lack of objective answers and opted for "I don't know."
• Sycophancy & Morality: While baseline RAG and CERTA both avoided direct contradictions, CERTA systems were less likely to agree or disagree with opposing viewpoints, instead opting for "I don't know" more frequently. CERTA also demonstrated more caution in moral judgments, avoiding strict moralizing when the context was incomplete.
• Model Differences: The open-source Mistral model exhibited more inherent uncertainty than GPT-4.1, but CERTA improved both.
• Consistency: The addition of certainty instructions did not cause the systems to change correct answers to incorrect ones; it primarily increased the rate of cautious responses where appropriate.

Significance and Claims
The paper claims that conveying self-reflected certainty is a viable mechanism for building appropriate trust in AI, aligning with the human value of benevolence. The authors conclude that:

• CERTA-0 is often sufficient to improve upon baseline RAG by introducing basic honesty regarding certainty.
• CERTA-1 provides a more careful approach by explicitly utilizing the Question-Context Relevance score.
• The method allows systems to be "risk-averse" in high-stakes scenarios by admitting uncertainty rather than hallucinating.
• The approach is retriever-agnostic, assessing only the quality of the provided context rather than the retrieval mechanism itself.

The authors remain modest regarding their claims, noting that the dashboard has not yet been validated with human users to measure actual trust calibration. They acknowledge limitations, such as the use of multiple-choice formats which may mitigate sycophancy artificially, and the reliance on cosine similarity which tends to produce mid-range scores (around 0.5) rather than extreme values. Future work is suggested to include user studies and testing with noisier data sources.