XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants

1. Problem Statement

AI coding assistants (e.g., GitHub Copilot) improve developer productivity by automatically gathering context from a project's codebase (surrounding code, related files, dependencies) to generate code recommendations. However, these assistants typically merge code from multiple sources into a single prompt sent to a Large Language Model (LLM) without distinguishing the origin or trustworthiness of that code.

This architecture creates a novel attack surface: Cross-Origin Context Poisoning (XOXO).

• The Threat: A malicious contributor with commit privileges can subtly modify shared code in a repository.
• The Mechanism: The attacker applies semantics-preserving transformations (e.g., renaming variables, reordering independent statements) that keep the code functionally correct and benign.
• The Impact: When a victim developer triggers a code generation request, the assistant automatically includes the poisoned context. The subtle changes "steer" the LLM to generate buggy or vulnerable code (e.g., SQL injection, XSS) that bypasses the model's safety guardrails, even though the input context appears harmless.

2. Methodology

A. The Attack Vector: XOXO

The attack relies on the observation that LLMs are sensitive to code structure and naming conventions, even when semantics remain identical.

• Semantics-Preserving Transformations: The attacker modifies identifiers (variables, functions) or reorders non-dependent statements. The code compiles and runs correctly, but the transformation alters the statistical patterns the LLM relies on for generation.
• Real-World Demonstration: The authors demonstrated an attack on GitHub Copilot. By renaming a shared variable from USE_RAW_QUERIES to RAW_QUERIES, they caused Copilot to generate SQL injection-vulnerable code for a database search feature, whereas the original context led to safe, parameterized queries.

B. The Algorithm: Greedy Cayley Graph Search (GCGS)

To systematically identify effective transformations without manual trial-and-error, the authors propose GCGS, a black-box algorithm.

• Core Insight (Confidence Monotonicity): The authors discovered that correct LLM outputs are correlated with high model confidence. Furthermore, combining multiple confidence-reducing transformations tends to reduce confidence monotonically (i.e., the combined effect is worse than the individual parts).
• Search Space: The space of transformations is modeled as a Cayley Graph (an infinite tree) where nodes are composite transformation sequences and edges are atomic transformations (e.g., renaming foo to bar).
• Algorithm Steps:
  1. Shallow Exploration: Sample atomic transformations and measure the resulting drop in model confidence (using log-probabilities or class probabilities).
  2. Deep Greedy Composition: Iteratively compose transformations that yield the lowest confidence scores, descending the tree toward "failure points" where the model generates incorrect or vulnerable code.
  3. Efficiency: This approach avoids exhaustive search by greedily following the path of steepest confidence decline.

3. Key Contributions

1. XOXO Attack: Introduction of a stealthy, inference-time attack exploiting automatic context inclusion in AI coding assistants.
2. GCGS Algorithm: Development of an efficient black-box search algorithm leveraging the confidence monotonicity property to find adversarial semantics-preserving transformations.
3. Empirical Validation: Comprehensive evaluation against 8 state-of-the-art (SoTA) LLMs (including GPT-4.1, Claude 3.5 Sonnet v2, and various open-source models) across bug injection and vulnerability injection tasks.
4. Real-World Impact: A successful end-to-end attack on GitHub Copilot, demonstrating that current safety mechanisms are insufficient against context poisoning.

4. Evaluation Results

The authors evaluated XOXO on three benchmarks: HumanEval+, MBPP+ (bug injection), and CWEval-Python (vulnerability injection).

• Attack Success Rate (ASR):
  • Bug Injection: Achieved an average ASR of 83.67% across all models. For example, GCGS achieved 92.00% ASR on Claude 3.5 Sonnet v2 and 97.11% on Llama 3.1 8B.
  • Vulnerability Injection: Achieved an average ASR of 52.26%. Notably, the attack triggered 17 distinct vulnerability types (CWEs), including SQL Injection and Cross-Site Scripting, despite the models' safety alignment.
• Stealth and Naturalness:
  • The generated adversarial code maintained high CodeBLEU scores (>96), indicating it looked natural to human reviewers.
  • The bugs were often subtle (e.g., failing only 1-2 test cases or missing a specific boundary check), making them difficult to detect during code review.
• Efficiency: GCGS significantly outperformed unguided search baselines, often requiring fewer queries to succeed while achieving higher success rates.
• Robustness: The attack remained effective even at temperature 0 (greedy decoding), which is the standard setting for production coding assistants to ensure determinism.

5. Significance and Implications

• Architectural Flaw: XOXO reveals a fundamental limitation in current LLM architectures: they struggle to maintain consistent security and correctness when processing semantically equivalent code variations. This is not a "jailbreak" but a failure of reasoning under specific contextual perturbations.
• Ineffectiveness of Current Defenses:
  • Adversarial Fine-tuning: The authors found that fine-tuning models on adversarial examples did not significantly reduce vulnerability (ASR remained >87%).
  • Static Analysis & Guardrails: Since the attack works by omitting defensive checks rather than inserting malicious code, signature-based tools and current AI guardrails (like GitHub Copilot's safety filter) failed to detect the vulnerability.
• Supply Chain Risk: The attack highlights a critical risk in software supply chains. A single malicious commit can poison the context for all downstream developers using AI assistants, leading to widespread generation of insecure code.
• Future Directions: The paper suggests that effective defenses require provenance tracking (logging context sources), origin separation (processing contexts independently), and improved LLM interpretability to understand how specific context snippets influence outputs.

In conclusion, XOXO demonstrates that AI coding assistants are vulnerable to subtle, automated attacks that exploit the very mechanism (automatic context gathering) designed to make them useful, posing a significant threat to software security.