Agentic Framework for Political Biography Extraction

1. Problem Statement

The paper addresses a critical bottleneck in political science research: the scalable extraction of structured political facts (specifically elite biographies) from vast, unstructured, and noisy web sources.

• The Bottleneck: Traditional data production relies on manual coding by human experts or research assistants. This process is prohibitively expensive, time-consuming, and difficult to scale. It leads to static datasets that are often outdated, limited to high-profile elites, and biased toward regions with dense English-language documentation.
• The Limitation of Current AI: While Large Language Models (LLMs) have excelled at classification tasks (assigning fixed labels to text), they struggle with extraction tasks. Extraction requires:
  1. Open-domain search: Finding information across a non-exhaustive set of sources.
  2. High complexity: Reconstructing multi-field, temporal career histories rather than single labels.
  3. Context dependency: Information retrieval is path-dependent (e.g., finding a past role requires knowing a specific organization name discovered in a previous step).
  4. The Long-Context Problem: Relevant information for a single individual is often dispersed across thousands of documents, exceeding the context windows of even the largest models and leading to "lost-in-the-middle" errors where models fail to attend to critical details.

2. Methodology: The "Synthesis-Coding" Agentic Framework

The authors propose a two-stage framework that decomposes the extraction task into Synthesis (evidence acquisition and refinement) and Coding (structured mapping).

A. The Agentic Architecture (Upstream Synthesis)

Instead of a static "Retrieval-Augmented Generation" (RAG) approach that retrieves a fixed set of documents, the authors employ a recursive agentic loop (inspired by ReAct):

• Supervisor Agent: Maintains the global state, identifies knowledge gaps (e.g., "Missing early career dates"), and delegates tasks.
• Searcher Agents: Execute iterative, targeted web searches based on the Supervisor's instructions. They use tools to search, browse, and retrieve specific documents.
• Iterative Refinement: The system does not stop after one search. It analyzes retrieved documents, identifies new entities or dates, and generates new search queries to fill gaps.
• Synthetic Report Generation: The agent consolidates verified findings from heterogeneous sources (government sites, news, NGOs, local language archives) into a single, compressed, "signal-dense" synthetic report (wiki-like summary). This report serves as the input for the coding stage, avoiding the noise of raw long-context inputs.

B. The Coding Stage (Downstream Extraction)

• A separate LLM coder takes the curated synthetic report (or a curated Wikipedia page) and maps the information into a structured dataframe based on a strict codebook (e.g., Organization, Role, Start Date, End Date).
• This stage is treated as a standard extraction task where the input is already filtered and organized.

C. Evaluation Design

The authors validate the framework through three experiments using data from China, the US, and OECD countries:

1. Experiment 1 (Coding Validity): Tests if LLMs can match human coders when given curated inputs (Baidu Baike/Wikipedia).
2. Experiment 2 (Synthesis Superiority): Compares Agentic Synthesis (full web search) vs. Human Collective Synthesis (Wikipedia) vs. Non-Wiki Agentic Synthesis.
3. Experiment 3 (The Necessity of Synthesis): Diagnoses whether raw long-context concatenation (feeding all retrieved docs directly to the LLM) is sufficient, or if explicit synthesis (compression) is required.

3. Key Results

Experiment 1: LLMs Match/Exceed Humans in Coding

• When provided with identical, curated biographical texts (Baidu Baike), contemporary LLMs (Grok-4.1, Gemini-2.5, Qwen) matched or exceeded the accuracy of trained human research assistants.
• Recall: LLMs recovered 10–16 percentage points more true career events than humans, likely due to the absence of human fatigue and selective reading.
• Cost: LLM coding cost was approximately $0.13 per official, compared to $6.25 for human coding (a ~50x reduction).

Experiment 2: Agentic Synthesis Outperforms Wikipedia

• In open-web environments (US and OECD), the agentic system significantly outperformed the "Human Collective" baseline (Wikipedia).
• Performance Gains: Agentic synthesis increased the F1 score by ~14.7 percentage points (from ~0.76 to ~0.89).
• Recall vs. Precision: The primary gain was in Recall (+31.4 points), recovering "long-tail" information (early careers, minor roles, non-English sources) that Wikipedia omits. Precision dropped slightly (-5.2 points) but remained robust (~0.82–0.85).
• Non-Wiki Capability: Even when blocked from accessing Wikipedia, the agent achieved substantial gains (+11.7 F1), proving it can reconstruct biographies from raw government and news sources.

Experiment 3: Synthesis is Essential (The "Refinement Premium")

• Raw Concatenation Failure: Simply feeding all retrieved documents (raw context, ~300k tokens) to the LLM resulted in poor performance.
• Refined Representation: Using the synthetic report (compressed, ~30k tokens) derived from the same evidence resulted in a 10.4 point F1 increase over the raw approach.
• Mechanism: Long-context models suffer from "quantity penalties" (attention dilution) and "quality penalties" (noise/multilingual confusion). Explicit synthesis compresses evidence into a high-signal format, mitigating these failures.

4. Key Contributions

1. Formal Distinction: The paper rigorously distinguishes Extraction (open-domain, multi-step, stateful) from Classification (closed-set, single-step), arguing that standard LLM benchmarks fail to capture the complexity of political fact extraction.
2. Synthesis-then-Coding Framework: It proposes that evidence synthesis is a prerequisite for valid extraction. Skipping synthesis (e.g., naive context stuffing) creates a quantity-quality trade-off that degrades performance.
3. Scalable Agentic Package: The authors open-source a scalable framework that automates the "research assistant" workflow, capable of processing thousands of targets in parallel with full source traceability.
4. Empirical Validation: They demonstrate that automated systems can not only match human validity but also expand the informational scope of political science, recovering data on mid-level officials and non-English contexts that were previously too costly to collect.

5. Significance and Implications

• Democratizing Data Production: The framework lowers the barrier to creating high-quality, dynamic datasets for "information-poor" environments (smaller nations, lower-tier officials), reducing the "information structure bias" where research is limited to well-documented elites.
• Cost Efficiency: It reduces the cost of data production by roughly 97% compared to manual collection while improving coverage.
• Methodological Shift: It challenges the field to move beyond treating human coding as the only gold standard. While LLMs introduce new technical biases (e.g., sensitivity to context length), these are traceable and correctable, unlike the opaque cognitive biases of human coders.
• Future Research: The generated datasets enable new network analyses of elite politics (co-service ties, overlapping tenures) at a scale previously impossible, potentially reshaping theories of regime stability and policy coalitions.

In conclusion, the paper argues that automation in political science requires an architectural shift: moving from passive text processing to active, agentic research workflows that synthesize evidence before coding, thereby unlocking the potential for large-scale, transparent, and expansive political data.