💬 NLP

JAF: Judge Agent Forest

The paper introduces JAF (Judge Agent Forest), a framework that enhances automated evaluation and agent self-refinement by having a judge agent perform joint inference across a cohort of responses using a novel locality-sensitive hashing method to select diverse, relation-aware exemplars, thereby enabling holistic learning and improved reasoning in complex tasks like cloud misconfiguration triage.

Original authors: Sahil Garg, Brad Cheezum, Sridhar Dutta, Vishal Agarwal

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sahil Garg, Brad Cheezum, Sridhar Dutta, Vishal Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: From a Solo Judge to a "Forest" of Judges

Imagine you are a manager trying to decide which of your employees' reports are good and which need fixing.

The Old Way (Standard AI):
Currently, most AI systems work like a solo judge. You hand them one report at a time. The judge reads it, checks the facts, and gives a grade. They don't know what the other employees wrote. If Employee A makes a small mistake that looks like a typo, and Employee B makes the exact same mistake, the solo judge might miss it on both reports because they are looking at them in isolation. They lack the "big picture."

The New Way (JAF - Judge Agent Forest):
The authors propose a new system called JAF. Instead of looking at one report alone, the judge looks at a small group of related reports all at once.

Think of it like a panel of experts or a forest of trees. When the judge reviews one specific issue (like a security vulnerability in a cloud server), they also glance at 5 or 10 other similar issues from the same company.

If the judge sees that three different servers all have the same weird security setting, they realize, "Oh, this is a pattern, not a one-time mistake."
If one report says "This is safe" but a similar report from a neighbor says "This is dangerous," the judge spots the inconsistency immediately.

By looking at the group together, the judge becomes much smarter, spotting patterns and errors that a solo judge would miss.

The Real-World Problem: Cloud Security Triage

The paper tests this idea on a very difficult job: Cloud Vulnerability Triage.

Imagine a massive cloud network with thousands of computers, apps, and security settings. Every day, scanners find thousands of potential problems (vulnerabilities).

The Challenge: A problem that looks dangerous on paper might be harmless in reality because of how that specific computer is set up. Conversely, a "medium" problem might be critical if it's on a server that talks to the internet.
The Noise: The data is messy. Different sources give different risk scores for the same problem.
The Need: You need to decide which problems to fix right now and which to ignore. Doing this one by one is slow and prone to error because the AI doesn't see how the problems connect.

JAF helps the AI see the connections. It treats a group of related security issues as a single "cohort" (a logical family of problems) rather than a random list.

How It Works: The "Hash Map" Analogy

To make this work efficiently, the system needs a way to pick the right "neighbors" to show the judge. You can't show the judge every other report; that would be too much information.

The paper introduces a clever sorting method called Locality-Sensitive Hashing (LSH).

The Analogy: The Library Sorter
Imagine a massive library where books are sorted not just by title, but by a complex code that considers the author, the genre, the mood of the story, and the type of paper used.

The Code: The system turns every security report into a short "binary code" (like a barcode made of 0s and 1s).
The Buckets: Reports with similar codes get put into the same "bucket."
The Selection: When the judge needs to review a report about a "Web Server," the system doesn't just grab random reports. It grabs reports from the same "bucket" (other web servers) and maybe a few from "neighboring buckets" (databases that talk to web servers).

This ensures the judge is always comparing apples to apples, or at least apples to oranges that grow in the same orchard.

The "Forest" Metaphor:
The paper calls it a "Forest" because:

Each time the judge reviews a report, they see a slightly different set of neighbors (like looking at a tree from a different angle).
By repeating this process many times, the system builds a "knowledge graph" where information flows between related issues, just like roots connecting trees in a forest. If one tree (issue) learns something new, that knowledge spreads to the others.

The Results: Smarter and Faster

The researchers tested this on 315 cloud assets (computers/servers) with various security misconfigurations.

The Solo Judge (Baseline): When the AI tried to fix the reports one by one, it was often unsure. The "confidence" in its answers was all over the place. Some were very confident, others were shaky. It took many rounds of checking to get close to a good answer.
The JAF Judge: When the AI looked at groups of related reports, it became much more confident and accurate much faster.
- It found inconsistencies the solo judge missed.
- It stabilized its answers quickly (reaching high confidence in fewer steps).
- It clearly identified the "hard cases" that were still confusing, flagging them for human experts to review.

Summary

JAF is a framework that stops AI judges from working in silos. Instead of judging one case at a time, it judges a group of related cases together.

Old Way: "Is this one report correct?" (Solo view)
JAF Way: "Is this report correct, and does it match the pattern of these 10 similar reports?" (Group view)

By using a smart sorting system (Hashing) to pick the right neighbors, JAF allows the AI to learn from the collective experience of the whole group, leading to faster, more consistent, and more reliable security decisions.

Technical Summary: Judge Agent Forest (JAF)

1. Problem Statement

Current agentic AI frameworks rely on judge agents to evaluate outputs, enforce guardrails, and enable iterative self-refinement. However, standard implementations operate in an instance-local fashion: a judge evaluates a single query-response pair $(Q_i, R_i)$ in isolation, potentially using static in-context examples that do not reflect the current workload's specific context.

This isolation fails to capture cross-instance dependencies and emergent cohort-level patterns. In complex domains like cloud vulnerability triage, the risk of a specific issue often depends on its relationship to other issues within the same tenant or account (e.g., shared misconfigurations, network topology, or recurring software roles). Instance-local judges cannot systematically exploit these regularities, enforce consistency across similar assets, or propagate insights (such as a discovered misconfiguration) from one ticket to related ones. Furthermore, standard methods for improving reasoning—such as iterative self-refinement on a single instance or multi-chain-of-thought (CoT) exploration—often duplicate compute costs and fail to leverage the diversity of reasoning already present across a workload.

2. Methodology: Judge Agent Forest (JAF)

JAF introduces a paradigm shift where the judge agent conducts joint inference across a cohort of logically related query-response pairs (e.g., all vulnerability-asset pairs for a specific tenant or time window), rather than evaluating them in isolation.

Core Mechanism

Cohort-Level Reasoning: Instead of a static context, the judge's prompt for a focal instance $(Q_i, R_i)$ includes a small, dynamically selected set of peer exemplars $(Q_j, R_j)$ from the same cohort.
Knowledge Graph Induction: These overlapping neighborhoods induce a dynamic, cohort-level knowledge graph where nodes are instances and edges represent co-occurrence in a judge's prompt. Information and critique propagate along these edges, analogous to belief propagation or label propagation in graph-based semi-supervised learning.
Ensemble Behavior: By randomly sampling different neighborhoods for each judge invocation (or across iterative refinement rounds), JAF creates an ensemble effect. Each instance is evaluated under diverse, locally coherent contexts, allowing the system to detect inconsistencies and stabilize judgments without changing the underlying LLM architecture.

Neighborhood Construction: Locality-Sensitive Hashing (LSH)

To efficiently select relevant peers from large, heterogeneous cohorts, JAF employs a flexible Locality-Sensitive Hashing (LSH) framework that goes beyond simple embedding-based k-nearest neighbors (kNN).

Learned Hash Predicates: JAF learns binary hash codes by integrating semantic embeddings, categorical labels, domain metadata, and LLM-defined predicates.
Information-Theoretic Objectives: The hash functions are optimized to maximize mutual information with side information (labels/metadata) while minimizing redundancy (total correlation) among bits. This ensures codes are informative, disentangled, and interpretable.
Hierarchical Structure: The hashing scheme creates a multi-scale partition of the cohort. Coarse buckets capture broad semantic similarities (e.g., asset type), while finer splits capture task-specific nuances (e.g., specific misconfigurations).
OOD and Anomaly Detection: The framework includes mechanisms to detect out-of-distribution (OOD) patterns or rare anomalies within buckets, assigning them dedicated hash bits to ensure they are scrutinized together.
LLM-Guided Predicates: LLMs can propose semantic binary splits (e.g., "external-facing" vs. "internal-only") or identify reference subsets for OOD detection, which are then validated via divergence-based clustering.

Integration with Reasoning Strategies

Iterative Self-Refinement: JAF extends self-refinement to the cohort level. As the judge critiques and refines responses, corrections propagate through the knowledge graph. Instances that stabilize (are "frozen") provide high-quality exemplars for refining more difficult cases.
CoT Exploration: JAF treats reasoning paths (CoTs) as first-class objects. By hashing CoTs, the system can share successful reasoning templates across different instances, amortizing the cost of exploring diverse reasoning paths.
Probabilistic Evaluation: By running multiple randomized evaluations on fixed refined responses, JAF generates an empirical acceptance probability ( $\hat{p}_i$ ). This serves as a stability metric, flagging fragile decisions that are sensitive to context variations for human review.
RL-Guided Exploration (Future): The hash structure provides a state space for Reinforcement Learning (RL) to adaptively allocate test-time compute, directing more resources to under-explored or historically error-prone hash buckets.

3. Key Contributions

Joint Inference Framework: Proposes JAF, a framework that elevates the judge from a local evaluator to a holistic learner by enabling joint inference across related query-response pairs.
LSH-Based Neighborhood Selection: Develops a novel, information-theoretic LSH algorithm that learns informative, interpretable, and relation-aware binary codes to select diverse exemplars, overcoming the limitations of pure embedding geometry.
Unification of Techniques: Demonstrates how JAF unifies iterative self-refinement, multi-CoT exploration, and ensemble learning into a single cohort-aware mechanism, effectively implementing belief propagation and random forest principles via in-context learning.
Probabilistic Stability Metrics: Introduces a method to estimate decision confidence via randomized neighborhood sampling, providing a robust signal for identifying unstable or ambiguous cases.

4. Experimental Results

The authors validate JAF on the demanding task of cloud misconfiguration triage, a domain requiring the synthesis of software roles, network exposure, and IAM policies.

Setup: A cohort of 315 assets with misconfigurations was processed by a primary agent (Llama3.3-Nemotron-Super-49B-v1.5). The baseline used isolated iterative self-refinement; JAF used cohort-aware refinement with simple, objective-based neighborhood sampling (without embeddings or LSH in this initial study).
Metrics: Evaluation used a probabilistic judge to measure the empirical correctness probability ( $\hat{p}_i$ ) of the final answers under diverse peer contexts.
Findings:
- Higher Accuracy & Stability: After just 5 refinement iterations, JAF significantly shifted the distribution of correctness probabilities toward higher values (mean $\hat{p}_i$ increased, variance decreased) compared to the baseline.
- Faster Convergence: JAF achieved high-quality results in 5 iterations that the baseline could not reach even after 10 iterations.
- Robustness: JAF-based analyses were less sensitive to the specific peers seen in context, indicating more sound underlying reasoning.
- Hard Case Identification: Both methods identified a tail of low-confidence cases, but JAF consistently reduced the number of such fragile instances, naturally highlighting the most difficult issues for human review.

5. Significance and Claims

The paper claims that JAF represents a fundamental shift in how agentic systems handle evaluation and refinement. By situating joint inference at the judge layer, JAF maintains agent modularity while promoting information flow across inputs.

Scalability: The framework is designed to handle large, heterogeneous cohorts (hundreds or thousands of long-context cases) where instance-local approaches fail to capture dependencies.
General Applicability: While motivated by cloud security, the authors assert that JAF is broadly applicable to any domain requiring coherent, context-aware refinement of large cohorts of model decisions.
Modest Claims: The authors emphasize that even a "naive" instantiation of JAF (without sophisticated LSH or embeddings) yields substantial improvements. They position the LSH machinery and RL integration as future work to further optimize performance, rather than prerequisites for the framework's core benefits. The paper does not claim to replace supervised fine-tuning (SFT) but suggests JAF can act as a pre-conditioner to produce higher-quality pseudo-labels for SFT.

In summary, JAF bridges belief propagation and ensemble learning principles to create a robust, context-sensitive evaluation layer that allows agentic systems to learn from the collective behavior of a cohort, rather than just individual instances.