💻 computer science

LLM-Based Robustness Testing of Microservice Applications: An Empirical Study

This empirical study demonstrates that prompt strategy significantly influences the diversity and coverage of LLM-generated robustness tests for microservice APIs, revealing that a taxonomy-guided few-shot approach outperforms both larger model ensembles and fixed prompts in exposing distinct failure modes.

Original authors: Hrushitha Goud Tigulla, Marco Vieira

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hrushitha Goud Tigulla, Marco Vieira

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you own a busy restaurant with a kitchen (the main app) and several specialized stations: a salad bar, a grill, a drink station, and a cashier. Each station is a "microservice." They talk to each other to get your order done.

Now, imagine you want to make sure your restaurant doesn't crash if a customer does something weird. Maybe they try to order a salad with a negative number of items, or they try to pay without a credit card, or they send a message that is too long to read. This is called Robustness Testing: intentionally trying to break the system with "bad" inputs to see where it fails.

The problem is, humans are tired. We can't think of every weird thing a customer might do. So, the researchers in this paper asked: Can we use AI (specifically Large Language Models or LLMs) to come up with these weird tests for us?

Here is what they found, explained simply:

1. The Setup: The AI Chefs

The researchers hired three different "AI Chefs" (AI models of different sizes and specialties) to write these tests. They gave them the restaurant's menu (the API specifications) and asked them to generate tests.

They tried 7 different ways of asking (called "Prompt Strategies"):

The Blank Slate: Just saying "Write some tests."
The Strict Manager: Giving them a checklist of exactly what to test.
The Teacher: Showing them examples of bad tests first.
The Thinker: Asking them to think step-by-step before writing.
The Expert Guide: Giving them a rulebook of how things can break, plus examples of specific tricky situations.

2. The Big Discovery: How You Ask Matters More Than Who You Ask

The most surprising finding was that the way you ask the question matters more than which AI you use.

The "Strict Manager" Trap: When they gave the AI a strict checklist (the "Structured" prompt), all three AIs wrote the exact same tests. It was like giving three different chefs the exact same recipe card; they all made the exact same dish. This is bad because if the recipe has a blind spot, you miss it.
The "Expert Guide" Success: When they gave the AI a rulebook plus clear examples of tricky situations (like the difference between "missing a key" and "having an empty key"), the AIs started thinking differently. They found unique bugs that the others missed.

The Analogy: Imagine you are looking for lost keys in a house.

If you tell three different people, "Look in the kitchen," they will all look in the kitchen. If the keys aren't there, you find nothing.
If you tell them, "Look in the kitchen, but also check the fridge, the toaster, and the cat's bed," they will spread out and find more places.
The paper found that changing how you tell the AI to look (the prompt) was more effective than hiring a "better" AI.

3. The "Code Specialist" Paradox

One of the AIs was a "Code Specialist" (trained specifically to write code). You might think this would be the best at finding bugs.

The Problem: When asked to just "critique and improve" its own work (a strategy called Self-Refine), this specialist wrote perfect code that didn't actually check for errors. It was like a chef who made a beautiful cake but forgot to taste it to see if it was burnt.
The Fix: When the researchers gave this specialist the "Expert Guide" (the rulebook with examples), it suddenly became the best performer, finding more bugs than any other combination. The rulebook gave it the "adversarial intent"—the mindset to try to break things, which its code training didn't give it on its own.

4. The "Zero-Shot" Surprise

There was one strategy where they gave the AI no instructions at all, just the menu.

The Result: This AI found a specific type of bug that the others missed: State-based bugs.
The Analogy: The other AIs were focused on "Is the ingredient fresh?" (checking the data). The "Zero-Shot" AI was thinking, "Wait, did the customer try to order dessert before they ordered a main course?" (checking the flow).
Lesson: Even a "dumb" or unguided AI can find weird, logical mistakes that a highly guided AI misses because the guided AI is too focused on the rules.

5. The "Key-Absent" vs. "Value-Empty" Confusion

The paper highlights a specific confusion the AIs had.

The Rule: "If a value is missing, set it to null."
The AI's Mistake: The AIs interpreted this as "Set the value to an empty string" (like name="").
The Reality: In computer systems, name="" (empty) and name (missing entirely) are two totally different things that break the system in different ways.
The Solution: The AIs couldn't tell the difference until the researchers showed them concrete examples of both. Once they saw the difference, they could test both scenarios.

Summary of the Takeaways

Don't just hire a bigger AI: A smaller AI with a better prompt can beat a giant AI with a bad prompt.
Don't be too strict: If you give the AI a rigid checklist, they will all do the exact same thing. You need to give them rules but let them be creative.
Show, don't just tell: If you want the AI to understand a subtle difference (like "missing" vs. "empty"), you have to show them an example.
Mix your strategies: To find the most bugs, you shouldn't just run one test. You should run a mix: some strict tests, some guided tests, and even some "wildcard" tests with no instructions.

In short, the paper proves that how you talk to the AI is the secret sauce for finding software bugs, not just the size of the AI itself.

Technical Summary: LLM-Based Robustness Testing of Microservice Applications

Problem Statement
Microservice architectures are susceptible to reliability threats when malformed, missing, or boundary-value inputs cascade across dependent services. While robustness testing systematically exercises such inputs to expose server-side failures, generating diverse and effective test cases remains a significant challenge. Although Large Language Models (LLMs) can generate tests from API specifications, it is unclear whether different model architectures and prompt strategies produce diverse sets of failure modes or if they converge on the same bugs. Existing research focuses on code correctness and compilation rates but lacks empirical evidence on the diversity of failure modes discovered across model-prompt configurations in running microservice systems.

Methodology
The authors conducted a controlled empirical study applying LLM-based test generation to two architecturally distinct open-source microservice systems:

TeaStore: A Java-based monolingual application with 6 services communicating via form-encoded HTTP.
OTel Astronomy Shop: A polyglot application with 27 services (Go, C++, Rust, JS, .NET, etc.) communicating via JSON APIs and gRPC.

The study utilized 3 open-source LLMs (14B, 32B code-specialized, and 70B parameters) and 7 prompt strategies grouped into three categories:

Baselines: ZeroShot (API spec only) and Structured (pre-built violation table mapping endpoints to mutations).
Established Techniques: FewShot (examples of failure patterns), Chain-of-Thought (CoT), and Self-Refine.
Domain-Knowledge Strategies: Guided (embedding the mutation taxonomy from Vieira et al. [1] as context) and GuidedFewShot (adding CRASH failure classification and concrete examples distinguishing key-absent vs. value-empty mutations).

The experimental pipeline involved generating Java/JUnit 5 test code, cleaning outputs, executing them against live Docker deployments, and classifying robustness failures based on HTTP status codes $\ge$ 500 (the "Abort" class). The study defined Failure Modes (FMs) as unique (endpoint, parameter, mutation-type) triples verified against source code, identifying 9 FMs for TeaStore and 14 for OTel. Diversity was measured using pairwise Jaccard similarity of triggered FM sets.

Key Contributions

Prompt Strategy Dominance: Empirical evidence demonstrating that prompt strategy explains more variation in test diversity than model size. Changing the prompt for a fixed model yields greater failure-mode set variation than changing the model under a fixed prompt.
Structured Collapse: Identification of a phenomenon where a prescriptive prompt (Structured) causes all models to produce identical failure-mode sets (Jaccard similarity = 1.00), effectively eliminating diversity.
New Prompt Strategies: Introduction of Guided and GuidedFewShot strategies that embed mutation taxonomies as domain context. GuidedFewShot specifically addresses the ambiguity of mutation rules (e.g., distinguishing between removing a key vs. setting a value to empty) through concrete examples.
Performance Transformation: Evidence that domain knowledge can transform a zero-coverage configuration into the highest-performing one. The code-specialized model (qwen2.5-coder:32b) triggered zero failures under Self-Refine but achieved the highest single-run coverage (56% on TeaStore, 57% on OTel) under GuidedFewShot.
Single-Model vs. Multi-Model: Demonstration that varying prompt strategies for a single model outperforms using multiple models with a single prompt. For instance, qwen2.5-coder:32b achieved 100% FM coverage on OTel by varying prompts across three strategies, outperforming any multi-model ensemble under a fixed prompt.

Results

Coverage: No single run achieved complete coverage on either system (maximum single-run coverage was 57%). However, union coverage varied significantly by strategy. FewShot achieved 89% union coverage on TeaStore, while GuidedFewShot achieved 64% on OTel.
Diversity: Structured prompting resulted in zero diversity (J=1.00). Self-Refine on OTel produced the highest diversity (J=0.00) but with low coverage. GuidedFewShot maintained low similarity (high diversity) while achieving high coverage, particularly on OTel (J=0.12).
The "Key-Absent" Limitation: The study found that LLMs given only taxonomy rules (in Guided) consistently failed to generate "key-absent" mutations (omitting a parameter key), interpreting "replace by null" as setting a value to an empty string. GuidedFewShot resolved this by providing explicit examples of both mutation types.
State-Based Failures: ZeroShot was the only strategy to discover state-based failure modes (e.g., removing a cart item without authentication), which parameter-focused strategies missed.

Significance and Claims
The paper claims that for robustness testing in microservices, prompt engineering is more critical than model selection. The authors argue that practitioners should not rely on a single model-prompt configuration, as this provides an incomplete view of the system's failure surface. Instead, they recommend:

Using GuidedFewShot as a strong baseline for single-model, single-prompt scenarios due to its high single-run coverage and maintained diversity.
Employing multi-configuration testing (varying prompts for a single model) as a more efficient path to high coverage than running multiple models with a single prompt.
Including at least one ZeroShot run to capture state-based failures that mutation-focused strategies overlook.
Recognizing that domain knowledge (mutation taxonomies) must be accompanied by concrete examples to disambiguate protocol-level implementations (e.g., HTTP parameter handling) that LLMs may misinterpret based on training data distributions.

The study concludes that while LLMs can generate robustness tests, their effectiveness is heavily dependent on how domain knowledge is framed in the prompt, and that a combination of strategies is necessary to maximize failure discovery.