MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs
MASTOR is a multi-agent framework that leverages source code analysis and a challenger-agent review process to generate semantic test oracles for RESTful APIs, significantly outperforming existing baselines in detecting business logic violations and achieving a 75.4% mutation score.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a team of inspectors to check a massive, complex factory that produces digital products (APIs). The factory has hundreds of different machines (endpoints) that take inputs and spit out products.
Traditionally, when testing these machines, inspectors only check the shipping label. They ask: "Did the machine turn on? Did it return a 'Success' sticker (HTTP 200)? Is the box the right shape?" If the sticker is green and the box looks right, the inspector says, "All good!"
The Problem:
The paper argues that this is dangerous. A machine could be broken inside, producing the wrong product, but still slap a perfect "Success" sticker on the box and keep the right shape. The label doesn't tell you if the product inside is actually what the customer ordered. This is a "semantic" failure—the logic is wrong, even if the surface looks fine.
The Solution: MASTOR
The authors built a new system called MASTOR (Multi-Agent Approach to Semantic Test Oracle Generation). Instead of just looking at the shipping label, MASTOR sends a team of specialized agents to walk through the factory floor, read the blueprints (source code), and understand exactly how each machine should work.
Here is how MASTOR works, broken down into simple steps:
1. The Blueprint Reader (Source Analysis)
Before testing, MASTOR sends a Source Extraction Agent to the factory.
- What it does: It doesn't just look at one machine; it traces every wire, pipe, and instruction manual connected to that machine (a "transitive import closure").
- The Result: It creates a detailed "Source Context" for every machine. This is like a cheat sheet that says: "If you put in a number smaller than 2, the machine must return a 'Bad Request' error. If you put in a valid name, it must return the specific capital city."
2. The Two-Track Inspection Team (Oracle Generation)
Once the cheat sheets are ready, MASTOR splits the work into two parallel teams:
- Team A (Single-Operation Path): These agents look at one machine at a time. They ask: "If I give this machine a broken input, does it correctly crash with the right error code? If I give it a good input, does it return the exact right data fields?" They use four different strategies (like checking boundaries and reversing logic) to make sure they don't miss anything.
- Team B (Multi-Operation Path): These agents look at how machines talk to each other. For example, Machine A creates a user and gives them an ID. Machine B needs that ID to fetch the user's profile. Team B checks: "Did Machine A actually save the ID correctly so Machine B can find it later?" This catches errors that happen when machines work together.
3. The Strict Editor (Challenger Agent)
This is the secret sauce. After the teams write their inspection rules (oracles), they don't just hand them in.
- The Review: A dedicated Challenger Agent (a strict editor) reads every rule. It asks: "Are you sure about this? Did you actually read the blueprint, or are you guessing?"
- The Fix: If the editor finds a weak rule or a guess, it sends it back to the original team with a note: "Go back and fix this specific part." The team rewrites only that part. This ensures the final rules are rock-solid and based on real evidence, not hallucinations.
4. The Final Report (Normalization & Output)
Finally, the system cleans up the rules, throws away any that don't make sense, and turns them into three useful formats:
- Executable Code: Ready to run automatically in a CI/CD pipeline (like a robot checking the factory every night).
- Postman Scripts: Ready for developers to test manually.
- Human Readable: A plain English description so a human can read and understand why a test exists.
The Results (The Scoreboard)
The authors tested this on 13 real-world factory projects (containing over 250,000 lines of code and 296 different machines).
- The Score: MASTOR caught 75.4% of the hidden bugs (mutations) that were planted in the code.
- Comparison:
- Compared to just asking a smart AI to guess the rules based on the shipping label (Direct Prompting), MASTOR was 30% better.
- Compared to a tool that only reads the official manual (SATORI), MASTOR was 49% better.
- Why? Because SATORI and Direct Prompting rely on what is written down or guessed. MASTOR relies on what is actually coded. If the manual says "Return a list," but the code says "Return a list only if the user is an admin," MASTOR knows the truth because it read the code.
The Cost
The system is a bit more expensive to run than a simple guess (it costs about $0.56 per API on average), but the authors argue it's worth it because it finds the deep, hidden logic errors that other tools miss.
In Summary:
MASTOR is like hiring a team of expert detectives who don't just check the packaging of a product; they read the factory's internal wiring diagrams to ensure the product inside is exactly what it's supposed to be. They double-check each other to make sure no mistakes slip through, resulting in a much higher quality safety net for software.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.