🤖 machine learning

Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures: A Multi-Stage Pipeline with Deterministic Reliability Scoring

The paper introduces PROBE, a multi-stage pipeline that combines deterministic PCAP normalization, evidence-grounded ensemble reasoning, and a model-agnostic evaluation framework to achieve high-accuracy, reliable, and bias-free automated diagnosis of 802.11 packet captures, overcoming the hallucination, uncalibrated confidence, and evaluation bias inherent in standard LLM approaches.

Original authors: Jerome Henry, Swadhin Pradhan, Miroslav Popovic

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jerome Henry, Swadhin Pradhan, Miroslav Popovic

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your evidence is a digital recording of a Wi-Fi network conversation (called a "packet capture"). Your goal is to figure out why a user's internet connection failed.

For a long time, this job required a human expert with deep knowledge of network protocols. They would read the recording, spot the tiny clues, and write a report. But humans are slow, expensive, and sometimes disagree with each other.

Recently, people tried using AI (specifically Large Language Models, or LLMs) to do this automatically. The paper argues that while these AI models are fast, they are like overconfident interns who sometimes make things up. If the recording is cut short, the AI might invent a reason for the failure that isn't actually there. If asked how sure it is, it will confidently say "95% sure," even when it's guessing.

The authors of this paper, working at Cisco, built a new system called PROBE to fix these problems. Think of PROBE not as a single detective, but as a highly organized detective agency with a strict workflow.

Here is how PROBE works, using simple analogies:

1. The Translation (Normalizing the Data)

First, the system takes the raw, messy digital recording and translates it into a clean, readable text report. Crucially, this translation is deterministic, meaning it always produces the exact same text for the same recording. It's like turning a chaotic crime scene photo into a numbered list of every item found, so no one can later claim, "I didn't see that item."

2. The "Brainstorming" Session (The Ensemble)

Instead of asking one AI to solve the case, PROBE asks many AIs to look at the same case, but with different instructions:

The "Root Cause" Detective: Looks only for the main reason the connection failed.
The "Sequence" Detective: Checks if the steps happened in the right order.
The "Evidence" Detective: Starts by listing the weird things they see and works backward to a conclusion.

They also ask a different type of AI (a "second opinion") to look at the case independently. This is like bringing in a detective from a different police precinct to ensure they aren't all making the same mistake.

3. The "Reality Check" (Reconciliation)

This is the most important part. In a normal meeting, if 5 out of 9 detectives say "No crime happened" and 4 say "It was a robbery," the group might vote "No crime." The paper found that AI models tend to be cowardly; they often vote "No crime" just to be safe, even when there is a crime.

PROBE doesn't just take a vote. It brings in a Senior Judge (a powerful AI model). The Judge doesn't just listen to the detectives; the Judge has the original, unedited recording in front of them.

The Judge looks at the 9 different reports.
The Judge checks every claim against the actual recording.
If a detective says, "Frame 12 shows a failure," the Judge checks: "Does Frame 12 actually exist? Does it show a failure?"
The Judge picks the best report, even if it was the "minority opinion" that the other detectives ignored.

4. The "Trust Score" (Reliability Scoring)

Instead of asking the AI, "How sure are you?" (which the paper found is useless because they always say "95%"), PROBE calculates a Trust Score based on math:

Did they cite real evidence? (Did they point to frames that actually exist?)
Did the detectives agree? (Did the different AI runs come to the same conclusion?)
Did the second opinion agree? (Did the different police precinct agree?)

If the score is high, the system auto-accepts the diagnosis. If the score is low, it flags the case for a human to review.

What They Found (The Results)

The paper tested this on 104 real-world Wi-Fi failure cases. Here is what happened:

The "Intern" (Single AI): Got it right about 91% of the time but missed critical clues in 35% of cases.
The "Voting Group" (Ensemble without a Judge): Actually got worse (dropping to 84%). Because the AIs were too cautious, they voted "No problem" too often, missing real failures.
The "Senior Judge" (PROBE): When the Judge reviewed the group's work, the accuracy jumped to 96%.
The "Confidence Trap": The paper confirmed that asking an AI "How confident are you?" is a waste of time. They always say they are very confident, even when they are wrong. PROBE's math-based Trust Score is the only reliable way to know if a diagnosis is safe to use.

The Big Takeaway

The paper concludes that for complex diagnostic tasks (like fixing Wi-Fi, but potentially other fields too), you don't need a "super-smart" single AI. You need a system that:

Generates many different theories.
Has a "Judge" that checks those theories against the hard facts (the original data).
Ignores the AI's own "confidence" and instead calculates trust based on how well the evidence matches the facts.

In short: Don't trust the AI's opinion; trust the AI's ability to check its own work against the evidence.

Technical Summary: Evidence-Grounded Ensemble Diagnosis of 802.11 Packet Captures

Problem Statement

Diagnosing connectivity issues from 802.11 packet captures (PCAPs) is a resource-intensive task requiring deep protocol expertise. It is slow, inconsistent among engineers, and lacks scalability. While Large Language Models (LLMs) offer a path toward automation, single-pass LLM analysis exhibits three critical failure modes:

Hallucinated Completion: Models infer non-existent protocol events (e.g., assuming a handshake failure when a capture is truncated mid-exchange) to fill gaps, leading to false positives.
Uncalibrated Confidence: Models generate self-reported confidence scores that cluster artificially high (e.g., 0.95) regardless of diagnostic difficulty, rendering them useless for routing decisions.
Golden Reference Bias: Evaluation frameworks that rely on "golden" references co-produced by the same LLM family introduce circular bias, penalizing stylistic divergence rather than diagnostic error.

Furthermore, naive ensemble methods (majority voting) often degrade performance in diagnostic contexts. Because LLMs exhibit a conservative bias toward "no issue" or "insufficient evidence" when faced with ambiguity, majority voting amplifies these conservative verdicts, causing the ensemble to miss confirmed failures more often than a single-pass model.

Methodology: The PROBE Pipeline

The authors introduce PROBE (Protocol Reasoning Over evidence-Based Ensembles), a multi-stage diagnostic pipeline designed to rectify these failures through deterministic, evidence-grounded reasoning. The system operates in five stages:

PCAP Normalization: Binary PCAP files are converted into a deterministic, structured text representation using tshark and PDML. This preserves frame-level verifiability, including frame numbers, timestamps, protocol types, status codes, and RF metadata. Every factual claim made by the system can be mechanically verified against this text.
Draft Ensemble Generation: The system generates an $N \times M$ $N \times M$ matrix of candidate diagnoses.
- Run Diversity ( $N$ ): Multiple independent runs use different analytical prompts (e.g., "root cause focus," "protocol sequence anomalies," "evidence-first") to test reasoning stability.
- Candidate Diversity ( $M$ ): Each run produces multiple hypotheses to surface internal ambiguity.
- Progressive Obfuscation: Key fields (e.g., RSSI, status codes) are masked in later runs to ensure diagnoses rely on structural protocol logic rather than shallow keyword matching.
- Cross-Model Second Opinion: An independent model from a different architecture family (e.g., Llama 3.3 alongside Claude Sonnet) provides an uncorrelated perspective.
Verdict Taxonomy: Candidates must select from a strict four-value taxonomy: CONFIRMED_ISSUE, PLAUSIBLE_ISSUE, INSUFFICIENT_EVIDENCE (critical for truncated captures), and NO_ISSUE_FOUND.
Evidence-Grounded Reconciliation: Instead of majority voting, a dedicated reconciler model (e.g., Claude Opus) reviews all candidates against the raw PCAP text and optional human expert (SME) annotations. The reconciler synthesizes the best-supported diagnosis, explicitly comparing model claims against packet evidence and overriding conservative or hallucinated conclusions.
Deterministic Reliability Scoring: A composite confidence score ( $C$ $C$ ) is computed without relying on LLM self-assessment. It is derived from three measurable signals:
- Evidence Validity ( $E$ ): Verifies that cited frames exist in the PCAP and that evidence items logically support the verdict (including "contributing" evidence for INSUFFICIENT_EVIDENCE verdicts).
- Run-to-Run Stability ( $S$ ): Measures agreement on key frames and verdicts across independent runs.
- Cross-Model Agreement ( $A$ ): Measures convergence between the primary ensemble and the second-opinion model.

Key Contributions

A Multi-Stage Ensemble Pipeline: PROBE generates diverse hypotheses and employs a formal reconciliation step that evaluates candidates against packet evidence rather than relying on vote counts. This includes a verdict-aware evidence framework where the definition of "supporting evidence" adapts to the diagnostic conclusion (e.g., the absence of a frame supports an INSUFFICIENT_EVIDENCE verdict).
Deterministic Reliability Scoring: The system introduces a composite score derived from evidence validity, stability, and cross-model agreement. This eliminates the need for uncalibrated LLM self-reported confidence and enables principled escalation to human review.
Model-Agnostic Evaluation Framework: The authors propose an evaluation method based on per-field assertion matching. This separates verifiable facts (frame numbers, protocol types, root cause assertions) from narrative style, eliminating the circular bias inherent in golden references co-produced by specific models.

Experimental Results

Evaluated on 104 capture-reviewer pairs from 87 enterprise Wi-Fi captures:

Performance: Single-pass LLM analysis (Config B) achieved a weighted F1 of 0.912, outperforming the human SME baseline (0.871) in frame coverage but missing critical diagnostic frames in 35% of cases.
Ensemble Failure: Naive majority voting (Config D) caused a performance drop to 0.842, as the ensemble converged on conservative "no issue" verdicts, missing 50% of confirmed failures.
Reconciliation Success: Incorporating the reconciliation step (Config F, Full PROBE) elevated performance to 0.957, achieving a 96% auto-accept rate. The reconciler successfully recovered correct minority diagnoses that the ensemble had outvoted.
Confidence Calibration: LLM self-reported confidence was uninformative (71% of values were exactly 0.95). The deterministic composite score spread across a wider range but was not predictive of correctness in the current dataset because the reconciler was so effective (4% error rate) that there was insufficient variance in outcomes to predict. Instead, low confidence served as an indicator of capture difficulty/ambiguity.
Cost-Efficiency: The reconciliation step dominates the cost (approx. $0.345/capture), while ensemble diversity adds minimal cost. Config C (one-shot + reconcile) offered the best cost-efficiency ratio (0.964 Wt F1), while Config F provided robustness and reliability signals at a 23% cost premium.

Significance and Claims

The paper claims that reconciliation against source evidence is the decisive quality lever, not the size of the ensemble or the diversity of models. The core architectural insight is that a judge model with access to the original data can select the correct hypothesis from a set of independently generated candidates, even if the majority of those candidates are wrong.

The authors highlight several broader implications:

Diagnostic Reasoning vs. Self-Consistency: Standard self-consistency (majority voting) is counterproductive for diagnostic tasks due to conservative-verdict bias. Evidence-grounded reconciliation is a necessary alternative.
Absence as Evidence: A framework where the absence of expected data is treated as valid evidence is crucial for handling incomplete or truncated traces.
Evaluation Integrity: Golden references co-produced by LLMs require assertion-based evaluation to avoid circular bias.
Confidence Reframing: In highly effective diagnostic pipelines, confidence scores should be reframed as indicators of data difficulty (for dataset enrichment) rather than quality gates for output filtering.

The authors conclude that reliable automated diagnosis is achievable not through a single superior model, but through a principled architecture that separates hypothesis generation from evidence-grounded evaluation. They note that while the system is demonstrated on 802.11, the methodology generalizes to any domain with structured, deterministic ground truth (e.g., medical imaging, financial audits, legal case analysis).