Technical Summary: K-BROWSECOMP

Problem Statement

While frontier large language models (LLMs) are shifting evaluation focus from foundational capabilities (e.g., instruction following, reasoning) toward compositional, agentic tasks, the Korean AI community lacks standardized benchmarks for web-browsing agents. Existing Korean benchmarks primarily focus on static language understanding, factual knowledge, and reasoning (e.g., KorQuAD, KMMLU, CLIcK), failing to assess the ability to retrieve region-specific information, maintain evidence states, and synthesize information across multiple web pages. Furthermore, leading global models often saturate existing English-centric benchmarks, blurring the distinction between in-distribution and out-of-distribution evaluation. There is a critical need for a benchmark grounded in Korean contexts to measure progress in agentic browsing and address structural disadvantages Korean developers face regarding local cultural and institutional knowledge.

Methodology

The authors introduce K-BROWSECOMP, a web-browsing agent benchmark consisting of 400 problems designed to require multi-step retrieval and reasoning within the Korean web ecosystem. The benchmark is divided into two distinct subsets:

1. K-BROWSECOMP-VERIFIED (300 Problems)

This subset consists of manually crafted questions validated by native Korean speakers.

• Construction: Annotators followed strict guidelines requiring questions to be grounded in Korean contexts, supported by public textual web evidence, and difficult to answer via direct search but easy to verify once the answer path is known.
• Reasoning Formats: Questions require either multi-hop reasoning (sequentially traversing websites to find intermediate entities) or parallel-branching (intersecting multiple independent constraints to identify a unique answer).
• Constraints: Each item must have a unique, temporally stable answer. Annotators were prohibited from using LLMs to generate questions or relying on private/paid sources.
• Validation: All items were manually verified to ensure the gold answer, intermediate entities, and cited sources were recoverable from public web evidence. Items with inaccessible or inconsistent evidence were rejected.

2. K-BROWSECOMP-SYNTHETIC (100 Problems)

This subset addresses the difficulty and cost of manual construction by leveraging LLMs to generate adversarial problems based on identified failure modes.

• Motivation: Exploiting the information asymmetry where solving a browsing problem is hard, but verifying an answer is easier once the evidence path is known. The authors hypothesized that generating hard problems is also difficult.
• Generation Pipeline:
  • Seed Data: Used gold source pages from the Verified set and a taxonomy of nine failure modes (F1–F8) identified from model errors on the Verified set (e.g., ineffective search direction, cross-source hopping failure, constraint-tracking failure).
  • Agent Proposer: A web-browsing LLM agent (Claude Code) was tasked with drafting adversarial questions backwards from seed pages, targeting specific failure modes.
  • Filtering: Generated questions underwent three sequential filters:
    1. Searchability: Ensured the gold answer did not appear directly in search results.
    2. Well-formedness: Confirmed the answer was uniquely extractable from the source page.
    3. Adversarial Difficulty: Verified that target models (GPT-5.4-mini, Gemini-3.1-Flash-Lite) failed to solve the question.
• Outcome: Only 100 of 268 generated candidates passed all filters, resulting in a diagnostic split that specifically targets semi-structured parsing and constraint tracking.

Key Contributions

1. First Korean Agentic Benchmark: K-BROWSECOMP provides the first standardized benchmark for evaluating web-browsing agents in Korean contexts, covering diverse domains such as entertainment, education, local geography, and institutional knowledge.
2. Failure-Mode-Targeted Generation: The paper introduces a methodology for constructing synthetic diagnostic splits by using a browsing agent to generate problems that specifically exploit recurring failure modes identified in human-verified data.
3. Trajectory-Level Failure Taxonomy: The authors define a detailed taxonomy of 9 failure modes (F0–F8) occurring at the trajectory level, ranging from incomplete outputs and ineffective search directions to cross-source hopping and constraint-tracking failures.
4. Diagnostic Stress Test: The SYNTHETIC split serves as a targeted stress test, demonstrating that even models capable of solving verified problems struggle with adversarially generated variations that require precise state maintenance.

Experimental Results

The benchmark was evaluated on a diverse set of proprietary and open-weight models, including global frontier models (GPT-5.5, DeepSeek-V4-Pro, GLM-5.1) and Korean-specific models (K-EXAONE, A.X-4.0, HyperCLOVAX).

• Performance on Verified Subset:
  • Frontier models achieved low accuracy, ranging from 30.00% to 45.67%. GPT-5.5 led with 45.67%, a substantial drop from its performance on the original English BrowseComp benchmark (84.4%).
  • Korean LLMs released through the government's "Proprietary AI Foundation Model Project" performed significantly worse, with scores ranging from 0.00% to 10.33%.
• Performance on Synthetic Subset:
  • Accuracy dropped further on the SYNTHETIC split, with the strongest model (GPT-5.5) reaching only 26.00%. No model exceeded 30.00% on this split.
• Failure Analysis:
  • Search Effort: Incorrect trials often consumed more search calls than correct ones, suggesting that failures are not due to insufficient retrieval budgets but rather to the inability to merge constraints or preserve entity roles after retrieval.
  • State Maintenance: The primary bottleneck identified is the failure to maintain candidates, constraints, and role bindings across multiple turns. Models frequently retrieve relevant evidence but fail to synthesize it into a final answer due to premature candidate commitment or unmerged evidence branches.
  • Korean Models: Despite being Korean-centric, open-weight Korean models showed no clear advantage over global counterparts in this setting, often failing at specific stages such as tool-call protocols, post-retrieval state tracking, or answer finalization.

Significance and Claims

The paper claims that K-BROWSECOMP serves as a critical diagnostic target for the Korean AI ecosystem. The results indicate that current models, even state-of-the-art frontier LLMs, struggle with the compositional demands of web browsing in Korean contexts. The significant performance gap between global and Korean models highlights the need for specialized development in agentic capabilities rather than just static knowledge or language coverage.

The authors assert that progress in Korean browsing agents requires stronger mechanisms for trajectory-level state maintenance—specifically, the ability to preserve candidate sets, constraints, and source pointers across multiple search turns. The SYNTHETIC split demonstrates that failure-mode-targeted generation can produce useful diagnostic items that expose persistent weaknesses not captured by standard benchmarks. The authors position K-BROWSECOMP as a shared resource to encourage the prioritization of browsing-agent capabilities alongside static benchmark performance, ultimately aiming to build reliable web-browsing agents for Korean web environments.

The paper concludes with modest limitations, noting that the verified set is modest in scale (300 items) and uneven in domain coverage, and that the benchmark relies on a single browsing harness and search backend, which may conflate model capability with retrieval coverage and tool-interface reliability. Continued revalidation is necessary due to the dynamic nature of web evidence.