💻 computer science

MIRAGE: Online LLM Simulation for Microservice Dependency Testing

MIRAGE introduces an online LLM-based simulation approach that dynamically generates microservice dependency responses at runtime by analyzing source code and traces, achieving near-perfect fidelity in status codes and response shapes while outperforming traditional static methods in complex, stateful testing scenarios.

Original authors: XinRan Zhang

Published 2026-04-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: XinRan Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to perfect a new recipe for a complex dish, like a multi-layered lasagna. To test your recipe, you need to know exactly how the ingredients from your neighbor's garden (the microservices) will react. But your neighbor is away, or their garden is too expensive to visit every time you want to cook.

Traditionally, chefs would try to solve this by recording what the neighbor did last time they visited. They'd write down: "If I ask for tomatoes, they give me 3 red ones." This is called Record-Replay.

The Problem: Life is messy. What if you ask for tomatoes on a Tuesday instead of a Monday? What if you ask for 5 tomatoes instead of 3? What if the neighbor's garden is currently out of stock? The old "recorded" notes don't work because they are static. They can't handle new situations, leading to failed dishes (broken software tests).

Enter MIRAGE: The "Acting" Chef

The paper introduces MIRAGE, a new way to test software. Instead of writing down static notes, MIRAGE uses a super-smart AI (an LLM) to act as the neighbor's garden in real-time.

Here is how it works, using simple analogies:

1. The "Live Actor" vs. The "Scripted Robot"

Old Way (Record-Replay): Imagine a robot that only knows a script. If you ask for "tomatoes," it gives you "3 red tomatoes" because that's what it saw once. If you ask for "tomatoes" in a weird way, the robot freezes or gives you the wrong thing.
MIRAGE (Online LLM Simulation): Imagine a brilliant method actor. You don't give them a script. Instead, you show them the neighbor's blueprints (source code) and a diary of past visits (traces). When you ask, "Can I have 5 tomatoes?" the actor thinks, "Hmm, looking at the blueprints, we have plenty. But wait, the diary says we ran out last Tuesday. Let me check the current stock." Then, they give you the correct answer right then and there.

2. The "Memory" Trick

The biggest challenge in software testing is state.

Scenario: You buy a ticket (Request 1). Then you try to buy a second ticket for the same seat (Request 2).
Old Way: The robot forgets you bought the first ticket. It lets you buy the second one, causing a crash.
MIRAGE: The AI actor has a memory. It remembers, "Oh, I just sold that seat to you in the last message." It keeps track of the whole conversation, ensuring the simulation feels real and consistent.

3. The Three "Modes" of Operation

The paper tested MIRAGE in three different "costumes":

White-Box Mode (The Insider): The AI gets the source code (the secret recipe) of the neighbor's garden. It knows exactly how the garden works. Result: It was perfect (99% accuracy). It was like having the neighbor's head gardener on the phone.
Black-Box Mode (The Outsider): The AI doesn't have the source code (maybe the garden belongs to a different company). It only has the diary of past visits (traces) and the caller's notes. Result: It was still very good at guessing the right outcome (94% accuracy), though it sometimes guessed the wrong shape of the answer (like giving you a basket instead of a bag).
The "Scripted" Alternative: The researchers tried making the AI write a strict rulebook (Intermediate Representation) first. Result: This failed on complex tasks. It's like trying to write a rulebook for a chaotic jazz concert; the rules are too rigid, and the AI gets confused. Letting the AI "improvise" (online simulation) worked much better.

Why Does This Matter?

It's Cheaper and Faster: Setting up a real "neighbor's garden" (a real database and server) for testing takes hours and costs money. MIRAGE runs in the cloud, costs pennies per test, and needs no heavy equipment.
It Handles the Unexpected: Because the AI understands the logic (the code) rather than just memorizing history (the logs), it can handle questions it has never seen before.
It's Reliable: In the tests, when the real software would fail, MIRAGE failed too. When the real software worked, MIRAGE worked. It mimics the "pass/fail" decision of the real world perfectly.

The Bottom Line

Think of MIRAGE as hiring a method actor to play the role of your software's dependencies during testing. Instead of forcing them to read a rigid script (which breaks when the plot changes), you give them the character's backstory and let them improvise.

The result? You get a simulation so realistic that your software tests pass and fail exactly as they would in the real world, but without the cost and hassle of setting up the real world every time. It turns software testing from a game of "matching static cards" into a dynamic, intelligent conversation.

1. Problem Statement

Microservice integration testing requires simulating downstream dependencies to verify system behavior without deploying the actual services. Existing approaches rely on static artifact generation performed before test execution:

Record-Replay: Captures production request-response pairs and replays the closest match.
Pattern-Mining: Extracts stateful rules from traces to build mock servers.
Specification-Driven Stubs: Generates mocks from API schemas (e.g., OpenAPI).

Limitations: These methods fail when test scenarios deviate from the pre-generated artifacts (e.g., unseen parameters, complex error conditions, or cross-request state dependencies). The paper notes that record-replay achieves only 62% status-code fidelity and 16% response-shape fidelity on held-out scenarios, as they cannot dynamically reason about implicit state or code logic not captured in the initial artifact.

2. Methodology: MIRAGE

The authors propose Online LLM Simulation, a runtime approach where a Large Language Model (LLM) remains "in the loop" during testing. Instead of generating a static mock, the LLM answers each dependency request as it arrives, maintaining cross-request interaction state throughout the test scenario.

System Architecture:
MIRAGE consists of three core components:

Context Builder: Assembles a system prompt containing:
- Dependency Source Code ( $S_D$ ): (White-box mode) Truncated implementation of the target service.
- Caller Source Code ( $S_A$ ): Context on how the caller interacts with the dependency.
- Trace Summaries: Aggregated production traces grouped by endpoint and status code distribution.
Per-Request LLM Server: A FastAPI server intercepting HTTP requests.
- It serializes the incoming request (method, path, body, headers) into a JSON user message.
- It appends this to a conversation history (bounded to ~20 exchanges) to allow the LLM to track state (e.g., cart items, tokens, reservations) across multiple requests in a single scenario.
- The LLM returns a JSON object with status, body, and optional headers.
Scenario Context: Before a test, the harness informs the LLM of the sequence of calls (e.g., "reserve then confirm") without revealing expected outcomes, allowing the model to anticipate multi-step flows.

Operating Modes:

White-Box: LLM has access to dependency source code, caller code, and traces.
Black-Box: LLM has access only to caller code and traces (simulating third-party dependencies).

3. Key Contributions

Novel Paradigm: Introduces "Online LLM Simulation" as a primitive for dependency testing, shifting from compile-time artifact generation to runtime, stateful reasoning.
MIRAGE Implementation: A fully functional system that integrates LLMs into the CI/CD loop for microservice testing.
Comprehensive Evaluation: Validated across 110 test scenarios spanning 14 caller-dependency pairs in three distinct systems:
- Demo: A custom system with complex patterns (optimistic locking, saga orchestration).
- Online Boutique: Google's microservice demo.
- Sock Shop: Weaveworks' benchmark.
Ablation Studies: Analyzed the impact of input signals (source code vs. traces) and output constraints (free-form vs. typed Intermediate Representations).

4. Key Results

Performance Metrics:

White-Box Mode: Achieved 99% status-code fidelity (109/110 scenarios) and 99% response-shape fidelity.
Black-Box Mode: Achieved 94% status-code fidelity but dropped to 75% response-shape fidelity (indicating the model infers error codes well but struggles to reconstruct exact JSON structures without source code).
Comparison: MIRAGE significantly outperformed baselines:
- Record-Replay: 62% status / 16% shape.
- Pattern-Mining: 61% status (Demo only).
- Contract IR (Typed Generation): 55% status (Demo only).

Key Findings:

Source Code Dominance: Dependency source code alone yields 100% fidelity. Without it, the model infers correct error codes (94%) but loses structural accuracy.
Structured vs. Free-Form: Constraining the LLM to generate a typed Intermediate Representation (IR) hurt performance on complex stateful services (dropping to 55%) because typed match conditions cannot express implicit cross-request state. However, IR worked adequately (86%) on simple stateless APIs.
End-to-End Equivalence: In 8 end-to-end integration tests on the Demo system, MIRAGE produced the exact same pass/fail outcomes as real dependencies, validating its utility for CI pipelines.
Cost & Latency:
- Cost: $0.16–$ 0.82 per dependency (using models like Claude Sonnet/Opus).
- Latency: ~3s per request (vs. ~50ms for real services). However, MIRAGE eliminates the 2–5 minute setup time required for Docker provisioning and database initialization, making the net CI time comparable or faster for complex setups.
- Stability: Results were consistent across three different LLM families (within 3% variance).

5. Significance and Implications

Solving the "Stateful Mock" Problem: MIRAGE addresses the fundamental limitation of static mocks: the inability to handle unseen scenarios and complex state transitions. By keeping the LLM in the loop, it dynamically reasons about state.
Practical CI/CD Integration: It offers a "no-infrastructure" testing approach. Teams can run integration tests without provisioning heavy dependency environments (Docker, databases), reducing CI bottlenecks.
Guidance for Practitioners:
- Use White-Box mode when source code is available for maximum fidelity.
- Use Black-Box mode for third-party services; it is sufficient for control-flow testing (status codes) but less reliable for value-level assertions.
- Avoid rigid Intermediate Representations for complex, stateful services; free-form generation with implicit state tracking is superior.
Future Work: The authors plan to extend validation to value-level correctness (not just shape), support gRPC/event-driven protocols, and explore hybrid approaches combining runtime simulation with structural validation.

In conclusion, MIRAGE demonstrates that LLMs can serve as high-fidelity, runtime simulators for microservice dependencies, outperforming traditional static mocking techniques, particularly in scenarios involving complex state and error handling.