🤖 AI

Just-in-Time Catching Test Generation at Meta

This paper presents a scalable, Just-in-Time catching test generation system at Meta that uses code-change-aware methods and AI-assessed filtering to significantly reduce false positives while successfully identifying and preventing serious bugs from reaching production in large-scale backend systems.

Original authors: Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman, Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Ru

Published 2026-02-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman, Zachary Haluza, Mehrdad Honarkhah, Herve Robert, Jiacheng Liu, Weini Liu, Sreeja Thummala, Xiaoning Yang, Rui Xin, Sophie Zeng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef running a massive, high-speed restaurant kitchen (Meta's codebase) that serves billions of meals every day. Every few minutes, a sous-chef (a developer) submits a new recipe change to the head chef. Usually, these changes are just tweaks to make the food taste better. But sometimes, a change accidentally turns the soup into poison.

Traditionally, the kitchen has a safety net called "Hardening Tests." Think of these as taste-tests done before the new recipe is even written. The goal is to make sure the new recipe works perfectly so it can be added to the permanent menu. If the test passes, the recipe is safe. If it fails, the chef fixes the recipe and tries again. These tests are designed to pass.

The New Idea: "Catching Tests"
This paper introduces a different kind of safety net called "Just-in-Time Catching Tests." Instead of trying to prove the new recipe is good, these tests are designed to fail.

Here is the analogy:

Hardening Test: "Let's taste this new soup. If it tastes good, we keep it." (Goal: Pass)
Catching Test: "Let's taste this new soup. If it tastes bad (or different from the old soup in a weird way), we stop the chef immediately." (Goal: Fail)

The goal isn't to write a perfect test; the goal is to find a test that screams, "Hey! Something changed here that shouldn't have!" before the bad soup reaches the customers.

The Big Problem: The "False Alarm" Noise

The problem with this approach is False Positives. Imagine the test screams "POISON!" but the soup is actually fine. The chef just changed the garnish, and the test got confused.

If the test screams "POISON!" every time a chef changes a spoon, the kitchen grinds to a halt. The chefs get annoyed, stop trusting the tests, and the whole system slows down. The paper calls this "development drag." The challenge was: How do we find the real poison without screaming about every single garnish change?

How They Solved It: The "Diff-Aware" Detectives

The researchers built two types of automated detectives to look at the changes:

The "Dodgy Diff" Detective: This detective looks at the new recipe and assumes, "This looks suspicious, like a mutant version of the old recipe." It tries to break the new recipe to see if it fails. It's like a security guard who assumes everyone is a thief until proven otherwise.
The "Intent-Aware" Detective: This is the smarter detective. It reads the chef's notes (the "diff intent") to understand why the recipe changed. It asks, "If the chef tried to do this specific thing, what could go wrong?" It then creates a test specifically designed to catch that specific mistake.

The Results:

The "Intent-Aware" detective was 20 times better at finding these "weak catches" (tests that fail on the new code) compared to just guessing.
It found 4 times more useful alerts than the traditional "Hardening" tests.

The Filter: The "LLM Judges"

Even with smart detectives, there are still too many false alarms. So, the team added a second layer of filters: Automated Assessors.

Think of these as a panel of expert food critics (using AI and strict rulebooks) who look at the "Poison!" alarm and decide: Is this a real emergency, or just a false alarm?

The Rule-Based Judge: Looks for specific patterns. "If the test failed because the kitchen's oven broke (infrastructure issue), ignore it."
The AI Judge (LLM-as-Judge): Reads the code and the error message to understand the context. "The chef changed a boolean from True to False. Is that a bug, or did they mean to do that?"

The Magic Number:
These judges were able to filter out 70% of the false alarms automatically. This meant human chefs only had to look at the most suspicious 30% of alerts. This kept the kitchen moving fast while still catching the real problems.

Did It Actually Save the Day?

Yes. The team sent 41 alerts to human engineers.

8 of them were confirmed to be real bugs.
4 of those 8 were severe failures that would have caused major crashes in production (serving poisoned soup to millions).
Because of these tests, those 4 disasters were stopped before they happened.

The Bottom Line

The paper shows that you can catch serious bugs just before they go live by:

Generating tests that are designed to fail on new code.
Using AI to understand what the code change was trying to do.
Using smart filters to ignore the noise so humans don't get overwhelmed.

The result is a system that catches critical bugs without slowing down the developers, acting like a highly efficient security guard who only stops you if you're actually trying to steal something, not just because you're wearing a different hat.

Technical Summary: Just-in-Time Catching Test Generation at Meta

1. Problem Statement

The paper addresses the challenge of detecting severe regressions in large-scale backend systems (hundreds of millions of lines of code) before code changes ("diffs") land in production. Traditional automated test generation focuses on "hardening" tests: tests generated to pass at creation time to prevent future regressions. In contrast, this work targets "catching" tests, which are designed to fail at generation time on the proposed change but pass on the parent revision.

The core difficulty lies in the Oracle Problem. Unlike hardening tests where the expected behavior is often implicit (e.g., no crash), catching tests must detect deviations from a "general oracle" (the true, often vague, expected behavior). This makes distinguishing between true positives (genuine bugs revealed by the test) and false positives (tests failing due to oracle misalignment or test bugs) extremely difficult. High false positive rates create "development drag," impeding engineer velocity. The goal is to generate catching tests with a high enough true positive rate to justify the cost of investigation without halting development.

2. Methodology

The authors deployed a Just-in-Time (JIT) catching test generation system at Meta, triggered when high-risk diffs are submitted to the Continuous Integration (CI) system. The methodology comprises three main components:

2.1 Diff-Aware Workflows

Unlike traditional hardening workflows that treat the code as a static target, the proposed workflows are "diff-aware," utilizing information about the specific code change to generate failing tests. Two specific approaches were implemented:

Dodgy Diff Workflow: Treats the diff as a "mutant" (a buggy version of the parent) without inferring intent. It uses a mutation-guided LLM-based test generator to create tests that distinguish the diff's behavior from the parent.
Intent-Aware Workflow: Attempts to approximate the diff's intention using the diff's title, summary, and code. It infers "risks" associated with the implementation intent, constructs mutants representing these risks, and generates tests to kill these mutants. Tests that fail on the diff are harvested as "weak catches."

2.2 Baselines

To evaluate effectiveness, the authors compared their diff-aware workflows against three baselines:

Coincidental Catches: Tests generated for hardening that happen to fail on the diff but pass on the parent.
Hardening Workflows (TestGen-LLM & ACH): Standard coverage-guided and mutation-guided test generators applied to the parent revision, then checked against the diff.

2.3 Assessor Mechanisms (True/False Positive Filtering)

To mitigate false positives and reduce human review load, the system employs automated assessors to score "weak catches" (candidate tests) on a scale from -1.0 (highly likely false positive) to 1.0 (highly likely true positive).

LLM-as-Judge: An ensemble of state-of-the-art LLMs (Llama3.3-70B, Gemini 3 Pro, Claude Sonnet 4) analyzes the test code, execution trace, diff, and inferred intent. They provide normalized token probability scores and categorical likelihoods (High/Medium/Low).
Rule-Based Assessor (RubFake): A deterministic system that identifies specific patterns indicative of false positives (e.g., test runner failures, brittle reflection usage, incorrect mocks) or true positives (e.g., unexpected boolean flips, empty container exceptions).

2.4 Human-in-the-Loop

The system targets a "sense-check" interaction with engineers. Instead of asking engineers to debug test code, the system asks a simple question: "Is this behavior change (e.g., a boolean flipping from true to false) expected?" This minimizes friction. If the engineer confirms the change is expected, the signal is dismissed; if surprised, they investigate further.

3. Key Contributions and Results

3.1 Effectiveness of Diff-Awareness

Analysis of 22,126 generated tests revealed that diff-aware methods significantly outperform hardening baselines:

Diff-aware workflows produced 4x as many weak catches as hardening workflows.
They produced 20x as many weak catches as the "coincidental" baseline.
The Intent-Aware workflow generated weak catches for 7.9% of diffs, compared to 4.0% for the Dodgy Diff workflow and 3.6% for the best hardening baseline.

3.2 Reduction of Human Review Load

The combination of LLM and rule-based assessors reduced the human review load by approximately 70%. The assessors achieved over 98% precision in identifying false positives (specifically, discarding tests with low scores), allowing the system to scale evaluation capacity to roughly 4x that of manual review alone.

3.3 Statistical Correlation with Human Labels

Inferential statistical analysis compared the assessors' scores against human-labeled diffs ("Good" = accepted/landed, "Bad" = reverted/abandoned).

Good diffs (accepted) showed significantly more false positive assessments.
Bad diffs (rejected) showed significantly more true positive assessments.
While inter-rater agreement between assessors was modest (Spearman $\rho \le 0.535$ ), the results suggest the assessors provide complementary signals rather than redundant noise.

3.4 Real-World Impact

From 41 engineer reach-outs based on failing test signals:

8 were immediately confirmed as strong catches (true positives), yielding an engineer-experienced true positive rate of 19.5%.
4 of these 8 strong catches were confirmed to be severe failures that would have caused production outages if not intercepted.
Dismissing false positives took engineers only a few minutes, resulting in negligible drag on developer velocity.

4. Significance and Claims

The paper claims that Just-in-Time Catching is a scalable and industrially applicable approach for preventing severe failures in massive codebases. Its significance lies in several areas:

Shift from Hardening to Catching: It demonstrates that generating tests specifically to fail on proposed changes is feasible and effective, provided that diff-awareness and intent approximation are utilized.
Scalability via Automation: The use of LLM and rule-based assessors successfully manages the high false-positive rate inherent in automated catching, enabling the system to operate at scale without overwhelming human reviewers.
High Value of Targeting: By focusing on high-risk diffs, the system achieves a high return on investment; nearly half of the confirmed bugs were severe, suggesting that targeted catching is more efficient than random bug hunting.
Developer Experience: The "sense-check" interface minimizes friction, allowing engineers to validate behavior changes without engaging deeply with test code, which is particularly valuable in an era of increasing machine-generated code.

The authors conclude that while the current system is not perfect and future optimization is needed, the results provide a strong baseline for the "Catching JiTTest Challenge," proving that automated catching tests can prevent serious production failures with minimal impact on engineering velocity.