🤖 machine learning

Responsiveness Verification: Will Predictions Change? How Much? How Often?

This paper introduces a framework and algorithms to measure "responsiveness"—the probability that a machine learning model's output changes when its inputs are altered by interactions, noise, or manipulation—providing statistical guarantees to enhance safety and reliability across domains like recidivism prediction, content moderation, and LLM benchmarking.

Original authors: Harry Cheon, Meredith Stewart, Bogdan Kulynych, Tsui-Wei Weng, Berk Ustun

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Harry Cheon, Meredith Stewart, Bogdan Kulynych, Tsui-Wei Weng, Berk Ustun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a video game where the rules change every time you press a button. If you jump, the floor might move; if you shoot, the enemy might teleport. In the real world, the "players" are people, and the "game" is the machine learning models that decide things like who gets a loan, who gets flagged as a bot, or which AI chatbot is safe to use. These models are usually trained on a static snapshot of the world, like a photograph. But the real world is a movie; people change their behavior, sometimes to game the system, sometimes just by living their lives. The big question scientists are asking is: If a person changes their inputs (like their income, their social media posts, or how they type a prompt), does the model's answer change in a way we can predict? Or does the model get stuck, giving the same wrong answer no matter what the person does? This is the problem of "responsiveness." If a model isn't responsive, it can be dangerous because it might permanently lock someone out of opportunities or let bad actors slip through the cracks without us noticing.

This paper introduces a new way to test these models, not by trying to build a perfect one from scratch, but by acting like a stress-test engineer. The authors, Harry Cheon and their team, propose a method to measure "responsiveness"—the probability that a model's prediction will actually change when a person interacts with it. Think of it like a "shake test" for a bridge. Instead of just looking at the blueprints, you send a truck across it to see if the bridge sways, holds, or collapses. The team developed a toolkit that lets anyone define how a person might try to change their data (like "I can increase my savings but I can't change my age") and then uses math to simulate thousands of these changes. They don't just guess; they use strict statistical rules to say, "We are 95% sure that this model will change its mind 88% of the time," or conversely, "This model is stubborn and won't change its mind at all, even if you try."

The researchers found that many models are surprisingly stubborn. In one test involving recidivism prediction (guessing if someone will re-offend), they discovered that about 17.3% of people predicted to re-offend were "precluded"—meaning no matter how they cleaned up their criminal record in the simulation, the model still said "yes, they will re-offend." The model was essentially ignoring their efforts to improve. In another test with content moderation (catching fake bot accounts), they showed that some models looked great on paper but were actually easy to trick. When they tested Large Language Models (the smart chatbots) with harmless prompts, some seemed safe. But when they "gamed" the prompts by adding misspellings or translating them into other languages, the safety of the models dropped significantly. For example, a model named OLMo 2 7B looked very safe initially, but under these interactive tests, its compliance with harmful requests jumped up, making it much less safe than it appeared.

The paper argues against the idea that we need to redesign all our models to be "predictable" before we can use them. Instead, they suggest we should first test the models we have to see where they break. They built a Python library to help people do this testing. Their main finding is that by measuring responsiveness, we can catch these hidden failures before they cause real-world harm. They didn't just suggest this might work; they provided the algorithms, the statistical guarantees, and real-world examples showing exactly how much the predictions changed (or didn't) under different scenarios. They proved that without this kind of testing, we might be trusting models that are actually quite fragile, or worse, models that are unchangeable and unfair. The paper concludes that while we can't control how people interact with our systems, we can definitely build better tools to see how our systems react to those interactions, ensuring they remain safe and fair even when the world around them shifts.

Technical Summary: Responsiveness Verification

Problem Statement

Machine learning models are frequently deployed in dynamic environments where inputs change due to routine user interactions, strategic manipulation (gaming), or noise. In these settings, models may undermine safety by predicting over regions of the input space they have not seen, leading to unanticipated output changes. Existing approaches often attempt to mitigate these risks by learning models with predictable behaviors under interactions (e.g., strategic classification or recourse). However, this is challenging due to a lack of data to specify interaction behaviors and the difficulty of defining the language for input changes. Furthermore, current tools often compromise model fidelity (e.g., by dropping mutable features) to achieve safety.

The authors argue that instead of forcing models to behave predictably during training, a more practical approach is to treat models as given and rigorously test how their outputs change under realistic interactions. This paper addresses the gap in reliable, model-agnostic testing for "responsiveness"—the probability that a model's output changes when its inputs change under a specified interaction model.

Methodology

1. Formal Framework

The paper introduces a formal definition of an Interaction Model $I(x) = (A(x), P_{x,\cdot})$ for an individual with features $x$ :

Action Set $A(x)$ : Defines feasible actions $a$ subject to feature-level constraints (e.g., monotonicity), joint inequality constraints ( $G(x,a) \leq 0$ ), and joint equality constraints ( $H(x,a) = 0$ ).
Downstream-Effect Distributions $P_{x,\cdot}$ : Models stochastic effects resulting from an action. This includes random effects (e.g., measurement noise) and causal effects where an action on one feature influences others (e.g., increasing posts increases URL counts).
Reachable Point: A new state $x' = x + a + \epsilon$ , where $\epsilon$ is sampled from the downstream distribution.

Responsiveness ( $\rho(x)$ ) is defined as the probability that a reachable point $x'$ attains a target output set $\hat{Y}_x$ :
$\rho(x) := \mathbb{E}_{x'} [\mathbb{I}(f(x') \in \hat{Y}_x)]$

2. Validation Tasks

The framework supports three primary validation tasks:

Falsification: Determining if an output can change (e.g., detecting "preclusion" in lending where a user cannot change a denial to an approval).
Estimation: Measuring the frequency of output changes (e.g., estimating the prevalence of gaming in content moderation).
Testing: Hypothesis testing to determine if responsiveness exceeds a threshold $\tau$ (e.g., ensuring LLM safety benchmarks are robust to prompt variations).

3. Algorithms and Statistical Guarantees

Sampling Reachable Points: The authors propose Algorithm 1 (Rejection Sampling) to generate feasible points. The algorithm partitions the feature space based on conditional independencies in the interaction model. It samples actions for each partition and checks feasibility using Mixed-Integer Programming (MIP) solvers locally, which is more efficient than solving a global MIP.
Statistical Certification: The paper provides formal statistical guarantees for testing responsiveness. Using a sample of $n$ reachable points, it constructs a confidence interval based on the Beta distribution to test the null hypothesis $H_0: \rho(x) \geq \tau$ . Proposition 3 and Remark 4 provide formulas to determine the minimum sample size $n^*$ required to reject $H_0$ with a specific confidence level and power, ensuring practical decision-making.
Sensitivity Analysis: To address uncertainty in the interaction model specification, the authors propose analyzing responsiveness across a family of interaction models ( $\rho_{all}$ and $\rho_{any}$ ) and performing ablation studies to identify which constraints drive unresponsiveness.

Key Contributions

New Validation Approach: Introduces responsiveness measurement as a model-agnostic method to detect safety violations (preclusion, gaming, lack of robustness) without requiring model retraining.
Flexible Interaction Modeling: Develops machinery to define feasible interactions using complex actionability constraints and downstream effects, allowing practitioners to model realistic scenarios.
Efficient Estimation Algorithms: Designs fast algorithms to estimate responsiveness for any model, dataset, and interaction model, paired with formal statistical guarantees (sample size bounds and confidence intervals).
Empirical Demonstrations: Validates the framework across three domains:
- Recidivism Prediction: Detects "preclusion" where defendants cannot alter their risk scores by clearing criminal history, identifying demographic features as root causes.
- Content Moderation: Evaluates bot detection models, showing that models appearing robust under convex relaxations may actually be vulnerable to gaming.
- LLM Safety: Tests the robustness of safety benchmarks for Large Language Models against prompt interventions (misspellings, roleplay, translation), revealing that models with low base compliance rates may become highly compliant under specific interactions.

Results

Recidivism: The study found that 17.3% of defendants predicted to recidivate were assigned "fixed predictions" (unresponsive to changes in criminal history). Ablation analysis revealed that relaxing immutability constraints on demographic features (e.g., age, drug abuse status) significantly increased responsiveness, suggesting the models relied on these static features rather than criminal history.
Content Moderation: On the TwitterBot dataset, a model selected via a "Convex" relaxation procedure (perceived as robust) had a test AUC of 0.754 but was actually vulnerable to gaming; specifically, 57.1% of points were gameable under the Exact procedure for this Convex-selected model. In contrast, a model selected via "Exact" responsiveness testing had a lower AUC (0.727) but was truly robust, with 9.9% of points gameable for the model selected under the Exact procedure.
LLM Benchmarks: Testing LLMs on the sorrybench dataset revealed that while OLMo 2 7B had the lowest base compliance rate (22.7%), its compliance rate increased substantially under interaction models (e.g., misspelling), failing to meet safety thresholds under responsiveness testing.

Significance

The paper claims that responsiveness verification offers a rigorous, versatile, and actionable framework for promoting safety and reliability in machine learning systems. By shifting focus from learning "safe" models to testing existing models against realistic interaction models, the authors provide practitioners with tools to:

Detect failure modes like preclusion and gaming that are often invisible to static evaluation.
Make principled decisions regarding model selection and deployment based on statistical guarantees.
Identify root causes of unresponsiveness to guide policy changes or model revisions.

The authors emphasize that their approach does not replace model training but complements it by providing a "stress test" for model behavior throughout the lifecycle (training, procurement, deployment). They acknowledge that the validity of the results depends on the practitioner's ability to specify accurate interaction models, recommending sensitivity analysis to guard against misspecified assumptions.