💻 computer science

How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests

This registered report outlines a large-scale empirical study that investigates how humans, bots, and coding agents communicate about vulnerabilities in pull requests by analyzing both explicit identifiers and implicit security language across various project artifacts to understand their impact on review outcomes.

Original authors: Pien Rooijendijk, Christoph Treude, Mairieli Wessel

Published 2026-06-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pien Rooijendijk, Christoph Treude, Mairieli Wessel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, global construction site where thousands of teams are constantly building and repairing digital skyscrapers (software). In this world, there are three types of workers: Humans (the original architects), Bots (automated robots that handle routine tasks like updating materials), and Coding Agents (smart AI assistants that can write new blueprints on their own).

This paper is a "registered report," which means it's a detailed plan for a study the researchers are about to conduct. They want to understand how these three types of workers talk about safety hazards (vulnerabilities) when they submit changes to the building plans (Pull Requests).

Here is the breakdown of their plan using simple analogies:

1. The Two Ways Workers Talk About Danger

When a worker finds a crack in the wall or a weak beam, they need to flag it. The researchers noticed workers do this in two very different ways:

The "Official ID Card" Method (Explicit References):
Sometimes, a worker points to a specific danger and says, "This is CVE-2024-1234." It's like showing a barcode or a serial number. Everyone knows exactly what that specific number means because it's listed in a giant, official database.
- The Paper's Observation: Previous studies mostly looked only at these "ID cards."
The "Casual Warning" Method (Implicit Signals):
Other times, a worker just says, "Hey, this door might let unauthorized people in," or "This pipe looks like it could leak SQL data." They describe the problem in plain English without using a specific ID number.
- The Paper's Observation: The researchers suspect that by ignoring these "casual warnings," we are missing a huge chunk of safety conversations.

2. The Big Question: Who Says What?

The researchers want to know: Do Humans, Bots, and AI Agents talk about safety differently?

Do AI Agents (the new kids on the block) use more "Official ID Cards" because they are programmed to be precise?
Do Humans rely more on "Casual Warnings" because they understand the context better?
Do Bots stick to the rules and only use the IDs?

3. The Three Things They Plan to Investigate

The study is built around three main questions (Research Questions):

RQ1: The Distribution (Who is talking?)
They will count how often each type of worker uses "Official IDs" vs. "Casual Warnings" in different parts of the conversation (the title of the request, the description, or the comments).
- Analogy: It's like counting how many times the Architect, the Robot, and the AI Assistant use a formal safety code versus just shouting "Watch out!" in the breakroom.
RQ2: The Reality Check (Is the danger real?)
This is the most critical part. Just because someone says there is a danger, doesn't mean there is one.
- The researchers will use a "Safety Scanner" (a tool called Semgrep) to look at the actual code changes.
- They want to see: If an AI says "I fixed a security hole," did it actually fix a hole? Or did it just make a claim that wasn't true?
- Analogy: If a worker says, "I reinforced the bridge," the researchers will physically inspect the bridge to see if it's actually stronger, or if the worker just painted over a crack.
RQ3: The Reaction (How do people respond?)
How does the way the danger is described affect how the team reacts?
- If a worker uses a formal "Official ID," do the reviewers trust it faster and merge the change quicker?
- If a worker uses a "Casual Warning," does it cause a longer debate, more questions, or get rejected because it's harder to verify?
- Analogy: Does the construction foreman listen immediately when you show a barcode, or do they make you wait while they argue about whether your description of the leak is accurate?

4. How They Will Do It (The Toolkit)

The Dataset: They are using a massive collection of over 33,000 pull requests from popular open-source projects. This includes work done by famous AI agents like GitHub Copilot, Devin, and Cursor.
The Detection:
- For "Official IDs," they will use a digital magnifying glass (Regular Expressions) to find patterns like "CVE-..." or "GHSA-...".
- For "Casual Warnings," they will use a list of safety keywords (like "insecure," "hack," "bypass") to find natural language discussions.
The Verification: Since computers can make mistakes, they will use a smart AI (Gemini) and human experts to double-check a sample of the findings to make sure the "danger signals" are real.

5. The Limits (What This Study Won't Tell You)

The researchers are very honest about the boundaries of their study:

They are only looking at popular projects (those with many "stars" or followers). Small, quiet projects might have different rules.
They are only looking at GitHub. They aren't checking private emails, chat rooms, or other websites where safety issues might be discussed.
They are looking at the conversation and the code, but they can't guarantee that their "Safety Scanner" will catch every single possible flaw in the world.

Summary

In short, this paper is a blueprint for a study that asks: In the digital construction site of the future, how do humans, robots, and AI talk about safety? Do they use formal codes or plain language? And does the way they talk change whether their safety claims are believed or acted upon? The goal is to make sure we aren't ignoring the "casual warnings" just because they don't have a barcode attached.

Technical Summary: How Humans, Bots, and Agents Communicate About Vulnerabilities in Pull Requests

Problem Statement
Software vulnerabilities are typically communicated through standardized identifiers (e.g., CVE, CWE, GHSA) or implicit natural language (e.g., "SQL injection," "unauthorized access"). While prior research has extensively utilized explicit identifiers to study security-related development activities, this approach potentially overlooks discussions that rely solely on implicit language. Furthermore, the increasing prevalence of bots and autonomous coding agents in pull request (PR) workflows introduces new dynamics in how vulnerabilities are referenced. Existing studies have largely focused on how these actors use explicit identifiers but have not sufficiently investigated implicit security signals, whether such references correlate with actual vulnerabilities in the code, or how they influence PR review outcomes. This registered report addresses these gaps by investigating vulnerability communication across humans, bots, and coding agents.

Methodology
The study employs a large-scale empirical analysis based on the AIDev-pop dataset, a curated subset of the AIDev dataset containing over 33,000 pull requests from 2,807 popular open-source repositories (those with >500 stars). The dataset includes PRs authored by five specific coding agents (OpenAI Codex, Devin, GitHub Copilot, Cursor, Claude Code) as well as human and bot contributors.

The execution plan involves three primary phases:

Vulnerability Signal Detection:
- Explicit References: Identified using regular expression matching against standardized patterns for identifiers such as CVE, CWE, GHSA, GO, RUSTSEC, OSV, MAL, and USN (detailed in Table II of the paper).
- Implicit Signals: Detected using a validated keyword-based approach (Zhou and Sharma [11]) applied to PR titles, descriptions, review comments, commit messages, and timeline discussions.
- Validation: Detected signals are validated using the Gemini-2.0-flash LLM. To ensure reliability, two representative samples (360 PRs each for explicit and implicit signals) are manually coded by human annotators to establish ground truth, with inter-rater reliability measured via Cohen's $\kappa$ .
Account Classification:
- Coding Agents: Identified via specific user accounts annotated in the original AIDev-pop dataset.
- Bots: Identified via GitHub tags, username heuristics (containing "bot"), and a curated list of known bot accounts, with manual verification on a stratified sample.
- Humans: All remaining accounts not classified as agents or bots.
Data Analysis & Static Analysis:
- RQ1 (Distribution): Analyzes the frequency and distribution of explicit vs. implicit references across actor types and PR components using chi-square tests.
- RQ2 (Code Quality): Uses Semgrep (configured with default rule sets and four severity levels) to perform static analysis on the code before (base commit) and after (head commit) the PR changes. This determines if a vulnerability signal corresponds to an actual vulnerability being introduced, fixed, or remaining unchanged. Regression models control for PR characteristics (e.g., lines changed, file count).
- RQ3 (Social Response): Analyzes review activity (comment counts, reviewer counts), response times, and PR outcomes (merged, closed, open) using logistic and count-based regression models, controlling for repository and PR complexity.

Key Contributions

Comprehensive Signal Analysis: The study extends beyond explicit identifiers to include implicit security-related signals, providing a more complete picture of vulnerability communication.
Actor-Centric Investigation: It specifically compares how humans, bots, and coding agents differ in their communication patterns regarding vulnerabilities.
Empirical Validation: The research links communication signals to actual code changes via static analysis, addressing the question of whether references (especially those from agents) correspond to real security issues.
Dataset and Pipeline: The authors provide a dataset and analysis pipeline specifically designed for studying vulnerability communication in PR discussions, facilitating future research in this domain.

Results
Note: As this is a registered report describing a planned study, the paper does not present final empirical results or statistical findings. The text outlines the hypotheses and analytical framework rather than reporting outcomes.

The paper details the plan to answer three research questions:

RQ1: How vulnerability references are distributed among implicit/explicit types, actor types, and PR components.
RQ2: The association between reference types and static-analysis-detected security changes (introduced vs. fixed).
RQ3: The association between reference types and social response metrics (review activity and PR outcomes).

Significance and Scope
The study contributes a large-scale empirical investigation into vulnerability communication practices in modern software development. By distinguishing between explicit identifiers and implicit language, and by accounting for the role of autonomous agents, the research aims to uncover patterns that might be missed by studies relying solely on formal identifiers.

The authors acknowledge specific limitations regarding validity:

Internal Validity: Reliance on regex for explicit IDs may miss malformed identifiers; keyword-based detection for implicit signals may yield false positives/negatives, though mitigated by LLM and human validation. Static analysis tools (Semgrep) may also produce false positives or miss context-dependent issues.
External Validity: Findings are specific to popular open-source repositories (>500 stars) that actively use coding agents. The results characterize communication practices within these specific ecosystems and may not generalize to smaller, private, or non-GitHub projects.

The paper concludes by positioning this work as a foundational step toward understanding how different actors communicate about security, with future work suggested to investigate incorrect or unsupported security claims, particularly in AI-generated PRs.