Technical Summary: LogDx-CI

Problem Statement

Continuous Integration (CI) failure logs are inherently large (median 5k lines, up to 200k+ in the corpus) and noisy. While coding agents (e.g., Claude Code, Codex, Cursor) rely on upstream tools to reduce these logs into a manageable context for Large Language Models (LLMs), the field lacks public empirical evidence regarding which reduction strategies preserve sufficient evidence for downstream root-cause diagnosis. Existing benchmarks like SWE-bench or Terminal-Bench focus on end-to-end task success or repository editing, treating the context-reduction step as opaque infrastructure. LogDx-CI addresses this gap by isolating and evaluating the "context-reduction" step specifically for its ability to preserve failure signals for LLM diagnosers.

Methodology

Corpus and Ground Truth

The benchmark utilizes a corpus of 35 real GitHub Actions failure cases spanning 8 failure categories (e.g., test assertions, compile errors, timeouts) and 7+ ecosystems (pytest, cargo, Maven, Docker, etc.). Log sizes range from 27 lines to over 200k lines.

• Ground Truth: Root causes and required signals were drafted by AI (Claude Opus 4.7) and verified by a single author. Each case includes metadata, a "must-mention" checklist, and forbidden claims.
• Privacy: All logs underwent a privacy audit (redacting URLs, API keys, tokens) with zero hits across the 35 cases.

Evaluation Framework

The pipeline consists of: Raw Log → Context Reduction Tool → LLM Diagnoser → Diagnosis Score.

• Diagnosers: Four variants were tested:
  1. real-debugger-v1: Claude Haiku 4.5 (single-shot).
  2. real-debugger-v2: Claude Sonnet 4.6 (single-shot).
  3. real-debugger-v3: OpenAI gpt-5-mini (single-shot).
  4. real-agent-v1: Claude Sonnet 4.6 with a 5-turn agent loop using deterministic tools (grep, read_file, tail, view_log_lines) operating on the raw log.
• Metric: The primary metric is diagnosis_score_v1_1, a calibrated linear combination of category accuracy, signal recall, relevant file/test recall, and evidence quote rates. It penalizes "confident errors" (confidence ≥0.70 but demonstrably wrong). A secondary metric, confident_error_rate, tracks safety.

Baselines (11 Context Providers)

The study compares 11 reduction strategies:

1. Raw: Full log.
2. Tail: Last 200 lines.
3. Grep: Regex-filtered failure patterns + context.
4. RTK Modes: rtk-read, rtk-log, rtk-err-cat (various modes of the RTK tool).
5. LLM Summarizers: Map-reduce summarizers using Claude Haiku 4.5 and OpenAI gpt-5-mini.
6. Hybrids: Logic-based routers (e.g., hybrid-grep-120k-tail: use grep if ≤120k tokens, else tail-200).

Key Results

1. Hybrid Routers Dominate the Pareto Frontier

In single-shot diagnosis, hybrid grep+tail routers achieve the best cost-quality trade-off.

• Top Performers: hybrid-grep-120k-rtk-tail (Score: 0.670) and hybrid-grep-120k-tail (Score: 0.666).
• Efficiency: These methods achieve quality comparable to standalone grep (0.639) but at 4.5× fewer tokens (20k vs. ~88k tokens per case) and significantly lower cost ($0.03 vs. ~$0.13 per case).
• Safety: Top hybrids produced zero confident errors, whereas rtk-log had a 13.3% confident error rate, actively misleading the LLM in ~1 in 8 cases.

2. Agent-Loop Regime Collapses Quality Variance

When an agent is allowed to make follow-up tool calls (5-turn limit) to recover from insufficient context:

• Quality Convergence: The performance spread across reduction tools collapsed 7× (from a single-shot spread of 0.42 to an agent-loop spread of 0.059). Weak contexts (e.g., rtk-log, raw) were rescued by the agent, gaining +0.3 to +0.4 in score.
• Persistent Cost Differences: While quality converged, costs did not. Weak contexts forced the agent to issue 2–4× more tool calls to recover the diagnosis compared to strong contexts.
• Top Agent Method: Surprisingly, llm-summary-v1-gpt-5-mini became the #1 agent-loop method (Score: 0.749) with the lowest tool call count (0.37/case), as the summary front-loaded the failure signal effectively.

3. Cross-Family Summary Outperforms Self-Call

The study tested the "self-call bias" hypothesis (that a summarizer and debugger from the same vendor family might inflate scores due to shared priors).

• Finding: A cross-family pair (gpt-5-mini summarizer feeding a Claude Haiku debugger) outperformed the same-family pair by +0.071 on average across four diagnoser variants.
• Cost: The gpt-5-mini summarizer was also 10× cheaper ($0.18 vs. $1.75 per case) than the Haiku summarizer, largely due to avoiding Claude-Code-CLI nested invocation overhead.

Significance and Claims

LogDx-CI claims to be the first public benchmark specifically optimizing log reduction tools for downstream LLM diagnostic accuracy across model families. Its significance lies in three "load-bearing" findings:

1. Hybrid dominance: Simple hybrid routers (grep + tail) provide the optimal balance of cost and evidence preservation, outperforming complex LLM summarizers and raw logs in single-shot scenarios.
2. Agent resilience: While agent loops can recover from poor context, they do so at a higher operational cost (more tool calls), reinforcing the value of high-quality upstream reduction.
3. Bias falsification: The benchmark provides empirical evidence against the self-call bias hypothesis in this specific task, showing that cross-family summarization is both more effective and more cost-efficient.

The authors conclude that for production coding-agent stacks, the cost of implementing the correct context-reduction strategy (approx. $0.03/case for single-shot, $0.18/case for agent-loop with cross-family summary) is at least 10× less than the cost of getting it wrong (e.g., using rtk-log or raw logs which lead to misdiagnosis or excessive tool usage).

Limitations

The paper notes several limitations:

• Corpus Size: 35 cases is a prototype; absolute score magnitudes may shift with expansion, though top-3 rankings appear robust.
• Ground Truth: Verified by a single author and AI-drafted, not independently human-annotated.
• Model Scope: Tested only on Anthropic (Haiku/Sonnet) and OpenAI (gpt-5-mini) families.
• Reproducibility: gpt-5-mini exhibits run-to-run variance even at temperature=0, meaning specific cell numbers may shift by ±0.02, though aggregate rankings remain stable.
• Scope: Does not evaluate general coding ability, multi-step planning, or non-CI log shapes (e.g., build-system streams).