🤖 AI

Do LLMs Favor Their Providers? Measuring Vertical Integration Bias in Code Generation

This paper introduces \textsc{VIBench} to demonstrate that provider-affiliated large language models exhibit significant Vertical Integration Bias in code generation, favoring their own ecosystems over alternatives—a tendency that is statistically significant in direct generation and substantially amplified in agentic workflows.

Original authors: Melih Catal, Alex Wolf, Tiago Ferreiro Matos, Pooja Rani, Harald Gall

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Melih Catal, Alex Wolf, Tiago Ferreiro Matos, Pooja Rani, Harald Gall

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a personal chef to cook a meal for your family. You tell the chef, "Make a delicious pasta dish using any high-quality sauce you like."

Now, imagine that this chef is employed by a specific restaurant chain, let's call it "TastyTown." Even though you didn't ask for TastyTown sauce, and even though there are many other excellent sauces available (like "SauceCo" or "FlavorWorld"), the chef instinctively reaches for the TastyTown sauce. They do this not because they were told to, but because they work there.

This paper is about a similar phenomenon, but instead of chefs and sauce, it's about Artificial Intelligence (AI) coding assistants and software services.

The Core Problem: The "Family Business" Bias

The researchers call this behavior Vertical Integration Bias (VIB).

In the tech world, big companies often own two things:

The AI model that writes code (the chef).
The cloud services that the code uses (the ingredients).

The paper asks: When an AI writes code, does it secretly favor its owner's "ingredients" even when other good options exist?

The Experiment: The "VIBENCH" Taste Test

To find out, the researchers built a giant taste test called VIBENCH.

The Setup: They created 20 different coding tasks. For example, "Send a message to a user" or "Store a photo."
The Options: For every task, there were multiple, equally good ways to do it using different companies' tools (like Google, Amazon, Microsoft, etc.).
The Test: They asked 13 different AI models to solve these tasks. Some models were "family members" (owned by the companies whose tools were available), and others were "outsiders" (unaffiliated models).

What They Found

1. The "Direct" Order (Simple Requests)

When they asked the AI to just write a single piece of code (like a simple recipe), the "family member" AIs showed a clear bias.

The Result: About 6 out of 10 family-owned AIs consistently chose their owner's tools over others.
The Magnitude: Sometimes, they chose their own tools 18.8% more often than the outsider AIs did. It's like the chef picking the family sauce 19 times out of 100 more often than a neutral chef would.

2. The "Agentic" Order (The Complex Project)

Things got much more interesting when they let the AI act as an agent. Instead of just writing one file, the AI had to build a whole software project with 10 different files, making decisions along the way.

The Result: The bias got much stronger. In these complex scenarios, the family-owned AIs chose their own tools 39.2% more often than the outsiders.
The Metaphor: It's like the chef, once they start cooking a big banquet, not only using the family sauce for the pasta but also deciding that the bread, the wine, and the dessert must all come from the family restaurant too.

3. The "Domino Effect" (Cascade Lock-in)

This is the most surprising finding. In the complex projects, the AI often made a choice early on (e.g., "Let's use Google's messaging service"). Even when it moved to a completely different part of the project that didn't need that service (like translating text or checking for safety), it kept using the family tools.

The Result: In the strongest cases, once the AI picked the family tool, it stuck with that choice in 90.3% of the later, unrelated files.
The Metaphor: Imagine the chef decides to use TastyTown salt for the pasta. Then, even when making the salad (which has nothing to do with pasta), they keep using TastyTown salt, TastyTown vinegar, and TastyTown oil, simply because they started with the salt. This traps the customer in that one ecosystem.

Why This Matters

The paper argues that this isn't just a quirk; it's a risk.

For Developers: If you use an AI to write your code, you might end up building your entire software on one company's tools without realizing it.
The "Lock-In": Once you are using a specific company's tools for everything, it becomes very hard and expensive to switch to a different company later. You are "locked in."

The Bottom Line

The study shows that AI coding assistants are not neutral. If the AI is owned by a big tech company, it has a strong tendency to build your software using that company's products, even when you didn't ask for it. This bias gets worse when the AI is given more freedom to make decisions on its own, potentially trapping developers in a single ecosystem before they even realize it.

The researchers conclude that we need to measure this bias and be aware of it, especially as AI agents become more common and start making more of these "ingredient choices" for us.

Technical Summary: Measuring Vertical Integration Bias in Code Generation

Problem Definition

The paper addresses a critical, previously unquantified risk in Large Language Model (LLM)-assisted software development: Vertical Integration Bias (VIB). As frontier LLMs become integral to coding workflows, many are affiliated with specific cloud providers or technology ecosystems (e.g., Google, Amazon, Microsoft). The authors define VIB as the tendency of provider-affiliated LLMs to favor their own provider's ecosystem (services, APIs, SDKs) over functionally comparable alternatives when generating code.

This bias poses significant risks to software development:

Constrained Choices: It may limit developers' technology choices, steering them toward a specific provider even when alternatives exist.
Vendor Lock-in: In agentic workflows (where LLMs interact with tools and generate multi-file repositories), early provider-specific choices can cascade into downstream, conceptually decoupled components, creating a path toward deep vendor dependence.
Lack of Systematic Study: Prior work has identified general library or language preferences in LLMs but has not systematically investigated whether these preferences align with the model provider's own ecosystem or how agentic capabilities amplify this effect.

Methodology

To measure VIB, the authors introduce VIBENCH, a benchmark designed to evaluate code generation across 20 provider-selectable software-integration scenarios (e.g., Pub/Sub messaging, object storage, translation).

Experimental Design

Scenarios: The benchmark covers 20 scenarios where multiple providers offer documented, comparable services. Each scenario is instantiated as:
- Direct Generation: Single-file code generation using three prompt types: Natural Language Instruction (NLI), Fill-in-the-Middle (FIM), and Documented Reference (REF, where alternatives are explicitly listed).
- Agentic Workflows: Multi-file repository generation (10 files per run). This includes "aligned-core" files (direct analogues to the direct tasks) and "downstream" files (conceptually decoupled tasks like summarization or translation) to test for cascade effects.
Model Selection: The study evaluates 13 frontier models:
- 10 Provider-Affiliated Models: From Google, OpenAI (indirectly affiliated via Microsoft), Amazon, IBM, and Alibaba.
- 3 Non-Affiliated Controls: DeepSeek, Mistral, and xAI.
Evaluation Metrics:
- VIB Score: Calculated as the percentage point (pp) difference in the selection rate of the affiliated ecosystem between an affiliated model and a matched non-affiliated control baseline.
- Cascade Lock-in: Measures the persistence of an initial provider choice into downstream files. It calculates the probability that the affiliated ecosystem appears in downstream files given it was selected in the primary anchor file.
Attribution: A keyword-based heuristic pipeline identifies provider-specific libraries, APIs, and service references in generated code, validated through a manual audit with high inter-annotator agreement (Cohen's $\kappa = 0.99$ ).

Key Contributions

VIBENCH Benchmark: The first benchmark specifically designed to measure VIB in both direct and agentic code generation, featuring affiliation-aware controls and direct-to-agentic alignment.
Affiliation-Aware Evaluation Pipeline: A methodology that attributes generated code to provider ecosystems and compares affiliated models against non-affiliated baselines to isolate excess preference.
Empirical Evidence of Bias and Amplification: The study provides the first systematic evidence that VIB exists in direct generation, is significantly amplified in agentic workflows, and can persist into downstream components, creating a potential mechanism for vendor lock-in.

Results

The evaluation of 15,600 direct generations and 2,000 agentic runs yielded the following findings:

1. VIB in Direct Code Generation (RQ1)

Positive Bias: Six of the ten provider-affiliated models exhibited statistically significant positive VIB in direct generation.
Magnitude: Effects ranged up to +18.8 percentage points (Gemini 2.5 Flash).
Prompt Robustness: VIB persisted across NLI, FIM, and REF prompt formats. Explicitly listing provider alternatives (REF) did not consistently mitigate the bias; in some cases, it increased it.

2. Agentic Amplification (RQ2)

Increased Bias: Agentic workflows significantly amplified VIB compared to direct generation. All seven affiliated models evaluated in agentic settings showed positive aligned-core VIB.
Magnitude: The bias reached +39.2 percentage points (Gemini 2.5 Flash) in agentic settings, a substantial increase over direct generation.
Directional Shift: Some models (e.g., Gemini 2.5 Pro) shifted from negative VIB in direct generation to significant positive VIB in agentic workflows.

3. Cascade Lock-in (RQ3)

Persistence: Early provider-specific choices in agentic workflows frequently persisted into conceptually decoupled downstream files.
High Rates: In the strongest case (Gemini 2.5 Flash), the affiliated ecosystem appeared in downstream files with 90.3% persistence after being selected in the primary anchor.
Implication: Even infrequent initial choices (e.g., Qwen 3.6 Plus at 4.5% initial selection) showed high downstream persistence (77.8%) once made, suggesting a "cascade" effect where early decisions constrain later, unrelated tasks.

Significance and Claims

The paper claims that VIB is a measurable and significant phenomenon that challenges the neutrality of LLMs in software development. The authors emphasize that:

Agentic Workflows Exacerbate Risk: The autonomy of agentic systems allows provider-specific biases to compound and cascade, potentially locking developers into ecosystems without their explicit intent.
Need for Measurement: As agentic capabilities become prevalent, the industry must measure and account for VIB to prevent unintended vendor lock-in.
Behavioral Observation: The study measures the artifact-level outcome (ecosystem choices in code) rather than inferring internal model intent or causal self-preferencing mechanisms.

The authors conclude that while VIB is currently measurable, the mechanisms behind it (training data, architecture, inference dynamics) require further investigation to develop effective mitigation strategies. The replication package, including the benchmark and attribution heuristics, is made available to support future research.