Technical Summary: Deepchecks – Evaluating Retrieval-Augmented Generation (RAG)

Problem Statement

Retrieval-Augmented Generation (RAG) systems, which integrate external knowledge sources with Large Language Models (LLMs), are transforming domains such as healthcare, finance, and customer service by mitigating hallucinations and improving context relevance. However, evaluating these systems remains a significant challenge. The stochastic nature of LLM outputs and the complex interplay between retrieval and generation components make end-to-end assessment difficult. Existing methodologies often lack a unified approach that can simultaneously evaluate retrieval quality, generation fidelity, and safety without relying exclusively on expensive, human-annotated ground truth. Furthermore, generic benchmarks frequently fail to capture the nuances of real-world, domain-specific deployments.

Methodology

The paper introduces Deepchecks, a modular, end-to-end evaluation framework designed to address these gaps. The methodology is structured around three core pillars:

1. Property-Based Decomposition

Deepchecks decomposes the complex RAG task into manageable, independent "properties" to enable granular error tracing:

• Retrieval Relevance: Assesses whether retrieved information addresses the input query.
• Grounded in Context: Evaluates if the generated answer is factually consistent with the retrieved source (faithfulness).
• Completeness: Measures if the answer comprehensively addresses all parts of the question.
• Safety Properties: Detects toxicity, PII leakage, jailbreak attempts, and ethical violations.
• Expected Output Similarity: (Optional) When a reference answer exists, this metric compares the generated output against the ground truth.

2. Novel "Grounded in Context" Pipeline

To specifically address hallucinations, Deepchecks proposes a multi-step pipeline for the "Grounded in Context" property, distinct from standard LLM-as-a-Judge approaches:

1. Extraction: A proprietary Small Language Model (SLM) extracts factual statements from the LLM's output.
2. Retrieval: Relevant passages are retrieved from the context for each specific statement.
3. Classification: A proprietary SLM, trained on production-like data, classifies whether each statement is entailed by the retrieved passages.
4. Aggregation: Entailment scores are aggregated into a unified score.

3. End-to-End (E2E) Holistic Evaluation

Deepchecks employs a learned aggregation mechanism to synthesize scores from the core properties into a single holistic metric. This mechanism classifies interactions into three categories:

• Negative: Any failure in core properties (relevance, groundedness, completeness, safety) triggers a negative classification.
• Positive: The answer must meet high thresholds across all core properties.
• Unknown: Interactions that pass metrics but do not meet the stringent criteria for "Positive."

4. Lifecycle Tools

Beyond static metrics, the framework includes tools for:

• Root Cause Analysis (RCA): Pinpointing specific pipeline components (e.g., retrieval vs. generation) responsible for failures.
• Version Comparison: Side-by-side metric analysis to evaluate system upgrades.
• Production Monitoring: Tracking reference-free metrics over time to detect performance drift and data distribution shifts.

Key Contributions

• Integrated Framework: A unified system that combines retrieval and generation evaluation with safety checks, moving beyond siloed assessments.
• Reference-Free Capability: The ability to evaluate RAG systems effectively without requiring human-annotated ground truth answers, utilizing proprietary SLMs and LLM-based reasoning.
• Holistic Metric Generation: The paper claims to be the first framework to generate a single, holistic metric for RAG pipelines that aligns with human intuition regarding positive or negative interactions.
• Production-Ready Tooling: The inclusion of RCA, version comparison, and drift monitoring tools specifically designed for the iterative improvement and maintenance of deployed RAG systems.

Results

The paper benchmarks Deepchecks against established frameworks, specifically RAGAS and LangSmith, across several datasets:

• Groundedness Benchmarking: On the "Grounded in Context" task, Deepchecks achieved superior ROC-AUC scores compared to RAGAS and LangSmith across multiple datasets:

  • TRUE Benchmark: Deepchecks (0.86) vs. RAGAS (0.65) vs. LangSmith (0.63).
  • SQuAD: Deepchecks (0.92) vs. RAGAS (0.83) vs. LangSmith (0.65).
  • PubmedQA: Deepchecks (0.84) vs. RAGAS (0.80) vs. LangSmith (0.78).

• End-to-End Evaluation: In accuracy tests on balanced binary classification (Positive vs. Negative interactions), Deepchecks demonstrated superior or competitive performance:

  • Public Datasets: Deepchecks achieved high accuracy (e.g., 0.96 on RAG-dataset-12000, 0.94 on SQUAD), generally outperforming RAGAS and matching or exceeding LangSmith.
  • Client Datasets: In real-world production scenarios (Technical Support and Job Profiles), Deepchecks significantly outperformed competitors. For example, on Client 2 (Job Profiles), Deepchecks achieved 0.81 accuracy, compared to 0.46 for RAGAS and 0.54 for LangSmith.

Significance and Claims

The paper positions Deepchecks as a significant advancement in the evaluation of LLM-driven RAG systems. Its primary significance lies in its ability to:

1. Bridge the Gap Between Research and Production: By offering tools for root cause analysis and production monitoring, it addresses the practical needs of deploying RAG systems in dynamic environments.
2. Enhance Reliability in Real-World Scenarios: The framework's superior performance on client datasets suggests it is better calibrated for the complexities of real-world data distributions compared to methods optimized primarily for public benchmarks.
3. Provide Actionable Insights: By decomposing failures into specific properties, the framework enables targeted debugging rather than relying on opaque, aggregate scores.
4. Align with User Expectations: The learned aggregation mechanism is designed to reflect human intuition regarding system quality, ensuring that evaluations align with actual user satisfaction and safety requirements.

The authors conclude that Deepchecks provides a robust foundation for assessing reliability, relevance, and user satisfaction, offering a user-aligned tool to advance the state of the art in generative AI applications.