SAGE: Scalable AI Governance & Evaluation

Technical Summary: SAGE (Scalable AI Governance & Evaluation)

Problem Statement
Industrial-scale search systems, particularly those handling natural language queries with multifaceted intent (e.g., LinkedIn Job Search and People Search), face a critical "governance gap." While semantic models have expanded expressiveness, they complicate the specification and enforcement of relevance standards. Traditional governance relies on two suboptimal proxies:

1. Manual Editorial Review: High-fidelity but unable to cover the combinatorial diversity of natural language queries at scale.
2. Engagement Metrics: Provide delayed feedback and suffer from systematic biases (e.g., popularity amplification), often failing to capture high-impact relevance failures or "silent" regressions.

The core challenge is constructing a Surrogate Judge ($J$) that approximates the latent human utility function ($H$) with minimal alignment divergence ($\Delta$) while maximizing coverage ($C$) across an unbounded semantic space.

Methodology: The SAGE Framework
SAGE operationalizes human product judgment through a triad of constructs: Policy (P), Precedent (E), and an LLM Surrogate Judge (J). The framework is built on three design principles:

1. Bidirectional Calibration: Unlike static evaluation systems, SAGE employs a dynamic loop where Policy, Precedent, and Judge co-evolve.

   • Human $\to$ Policy/Precedent: Experts flag policy intuition gaps and ambiguities.
   • Judge $\to$ Precedent: The Judge acts as an adversarial auditor, surfacing errors in human-curated "golden" datasets (e.g., identifying evidence in verbose documents missed by humans).
   • Judge $\to$ Policy: The Judge exposes policy under-specification when it introduces unsupported constraints or applies inconsistent logic.
   • Result: This loop systematically resolves semantic ambiguities, transforming subjective judgment into executable, multi-dimensional rubrics.

2. Interpretable Multi-Attribute Relevance: To eliminate "black-box" opacity, the Judge decomposes relevance into orthogonal attributes (e.g., Title, Seniority, Company) rather than a monolithic score. This enables targeted failure attribution during calibration. The system uses a graded 5-point scale (0–4) rather than pairwise ranking to enforce explicit quality thresholds (e.g., suppressing grades 0–1).

3. High-Throughput via Distillation: To bridge the gap between frontier model reasoning and industrial inference costs, SAGE utilizes teacher–student distillation.

   • Teacher Judge: A frontier LLM (GPT-o3) calibrated against expert precedent.
   • Student Judge: An 8B parameter model distilled from the Teacher. It is trained on stratified production traffic and aspirational queries, with underrepresented score classes (0–3) rebalanced to prevent collapse toward high-relevance predictions.
   • Online Model: The Student Judge is further distilled into an ultra-compact (0.6B) architecture for real-time serving. This involves structural pruning, reasoning internalization (collapsing Chain-of-Thought into 1-token predictions), and context minimization via offline summarization.

Key Contributions
The paper presents three primary contributions to industry-scale human-centric evaluation:

1. Bidirectional Calibration for Intent Alignment: A mechanism that transforms heterogeneous human judgment into versioned evaluation criteria, allowing the evaluation system to co-evolve with human judgment and surfacing policy ambiguities.
2. Interpretable Multi-Attribute Relevance Evaluation: A decomposition of relevance into orthogonal attributes that eliminates opacity, enabling precise failure attribution and achieving near-human-level agreement.
3. High-Throughput Evaluation via Distillation: A pipeline that compresses frontier LLM reasoning into efficient student surrogates, enabling large-scale offline evaluation ($>10^7$ annotations/day) and low-latency online evaluation ($>10^4$ QPS) at a fraction of the cost.

Results
Deployed within LinkedIn Search ecosystems, SAGE achieved the following:

• Alignment Quality: Through bidirectional calibration, the Teacher Judge achieved a linear weighted Cohen's kappa of 0.77 against expert human judgments, surpassing the 0.7 threshold for substantial agreement and approaching the practical ceiling of expert product owners (0.83).
• Distillation Fidelity: The Student Judge maintained high fidelity, achieving kappa scores of 0.72 (Job Search) and 0.73 (People Search) against human experts.
• Cost Efficiency: The Student Judge operates at 92$\times$ lower cost than the Teacher Judge and 154$\times$ lower cost than minimum-bound human labeling.
• Production Impact: By enabling simulation-driven development and policy-based oversight, SAGE helped detect regressions invisible to engagement metrics. Two sequential online A/B experiments integrating the policy-aligned Online Model resulted in a 0.25% lift in LinkedIn Daily Active Users (DAU), representing a significant step-function change in growth for the Job Search product.

Significance and Claims
The paper claims that SAGE successfully formalizes human-centric governance at a scale required for industrial AI systems. It argues that:

• High-Fidelity Curation > High-Volume Annotation: A small set of expert-curated precedents provides more leverage than large-scale labeling, shifting the human effort from producing labels to resolving ambiguity.
• The Judge Can Audit Ground Truth: Calibrated LLMs can identify errors in human-curated datasets, improving the quality of the supervision itself.
• Policy as a Versioned Artifact: Treating policy as a versioned software specification allows for stable long-term governance, distinguishing metric shifts caused by definition changes from actual model failures.
• Semantic Evaluation Complements Behavioral Proxies: While engagement signals remain standard, they often miss "silent" relevance degradation. SAGE demonstrates that operationalized product judgment provides a more stable and diagnostic indicator of ecosystem health, directly correlating with long-term user growth (DAU).