💻 computer science

AI Safety Evaluations Need To Consider Cascading Effects

This paper advocates for a paradigm shift in AI safety evaluations from a model-centric focus to a holistic, systems-oriented approach that analyzes "cascades"—the compounding interactions and cumulative downstream effects across the socio-technical algorithmic supply chain—to better address transparency, accountability, security, and safety gaps.

Original authors: Anna Neumann, Jatinder Singh

Published 2026-03-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna Neumann, Jatinder Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not Just the Engine; It's the Whole Car

Imagine you buy a brand-new, high-performance sports car (the Foundation Model or AI). You take it to a mechanic who runs a test on the engine alone. The engine passes with flying colors! It's fast, efficient, and safe.

But then, you drive that car off the lot. You add a custom turbo kit (a Developer's Prompt), you install a GPS that reroutes you through dangerous neighborhoods (Retrieval System), you put on a tint that makes it hard to see at night (Safety Filter), and you let your teenager drive it while they are distracted by music (User Personalization).

Suddenly, the car crashes.

The mechanic says, "But the engine was fine!" And the teenager says, "But the GPS told me to go there!"

This paper argues that current AI safety checks are like the mechanic only testing the engine. They look at the AI model in isolation or look at the final app as a "black box." They miss the messy, complicated reality of how all the different parts of the car interact with each other to cause a crash.

The authors call this interaction a "Cascade."

What is a "Cascade"?

Think of a Cascade like a game of "Telephone" played in a factory, or a row of dominoes falling.

The Chain Reaction: One small change at the beginning (like a user typing a specific word) gets passed to a database, then to an AI, then to a safety filter, then to a human reviewer, and finally back to the user.
The Compounding Effect: At every step, the message gets slightly changed, amplified, or twisted.
- Example: A user says, "I'm stressed."
- Step 1 (Database): The system sees they worked late last night.
- Step 2 (AI): The AI thinks, "This is a crisis!"
- Step 3 (Safety Filter): The filter thinks, "This is too sensitive, I must block it."
- Step 4 (Result): The user gets a generic, unhelpful message, or worse, the system escalates it to a manager who panics.
The Problem: No single person in the chain sees the whole picture. The database manager didn't know the AI would overreact. The AI developer didn't know the filter would block the help. The user didn't know their work history was being used.

The "Cascade" is the invisible chain of events that turns a small input into a big, unexpected, and potentially harmful output.

The Three Big Problems with Current Safety Checks

The paper identifies three main reasons why we are missing these cascades:

1. The "Blended Soup" Problem (Non-Modularity)

Imagine making a soup.

Old Way (Salad): You have lettuce, tomatoes, and cucumbers. If the salad tastes bad, you can easily take out the tomato and say, "The tomato is the problem."
AI Way (Soup): You blend everything together. If the soup tastes bad, you can't easily say, "It's the salt." Maybe it's the salt plus the heat plus the specific type of carrot.
The Reality: In AI, components (like safety filters and user prompts) are blended together. They interact in weird ways. A safety filter might work perfectly alone, but when combined with a specific user prompt, it might accidentally help the AI say something dangerous. Current tests don't taste the whole soup; they just taste the ingredients separately.

2. The "Blindfolded Chef" Problem (Opacity)

Imagine a restaurant kitchen with three chefs:

Chef A (The Model Provider) cooks the base soup.
Chef B (The App Developer) adds spices.
Chef C (The User) adds their own secret sauce.

Chef A doesn't know what Chef B added. Chef B doesn't know what Chef C added. If the soup makes a customer sick, who is to blame?

The Reality: In AI, different companies and people control different parts of the chain. They are often "blind" to what the others are doing. If a safety rule is broken, it might be because of a tiny change made by a user that the model provider never saw.

3. The "Moving Target" Problem (Dynamism)

Imagine a construction site where the blueprint changes every time you walk through the door.

The Reality: AI systems are becoming "agentic," meaning they can make their own decisions on the fly. They might decide to call a different database or use a different tool depending on the situation.
The Risk: A safety audit done on Monday might be useless by Tuesday because the AI decided to take a completely different path. The "chain" of components isn't fixed; it's built in real-time.

Why Does This Matter?

If we keep testing AI like we test a single engine, we will miss the crashes.

Accountability: If an AI hurts someone, we won't know who to blame. Was it the model? The app developer? The user? The "Cascade" makes it impossible to point a finger at just one person.
Safety: Harmful things happen not because the AI is "evil," but because of a weird combination of settings, filters, and user inputs that no one predicted.

The Solution: A New Way to Audit

The authors propose we need to change how we check AI safety. Instead of just looking at the Model or the App, we need to look at the Supply Chain.

New Analogy: Instead of just testing the engine, we need to test the entire car on a track, with the specific driver, the specific road conditions, and the specific cargo.
The Goal: We need to trace the "Cascade." We need to understand how a change in one part ripples through the whole system. We need to ask: "How does this safety filter interact with this specific user prompt?"

Summary

AI is not a single tool; it is a complex ecosystem.
Current safety checks are too narrow. They look at the pieces in isolation. This paper argues that we must start looking at the connections between the pieces. We need to understand the "Cascading Effects"—how small changes in one part of the system can snowball into big problems down the line.

To make AI safe, we need to stop testing the ingredients and start tasting the soup, while keeping an eye on who added what spice, and how the heat changed the flavor.

1. Problem Statement

Current AI safety and security evaluations suffer from a critical fragmentation. They predominantly focus on two isolated levels:

Foundation Model (FM) Audits: Evaluating the raw capabilities, biases, or safety of the base model (e.g., LLMs) in controlled, reproducible environments, often treating the model as a standalone entity.
Application Audits: Evaluating the final deployed system as a "black box," focusing on user experience and final outputs without inspecting the internal mechanics.

The Gap: These approaches fail to account for the algorithmic supply chain. Modern AI systems are socio-technical assemblages where a base model is integrated with numerous intermediate components (e.g., safety guardrails, retrieval systems, quantization, system prompts, personalization layers, and user preferences) managed by different stakeholders (providers, developers, users).

The Core Issue: These components interact in complex, non-linear ways. The paper argues that safety risks often do not originate from the model itself but emerge from the cascading effects of interactions between these components. Current audits miss these "flow-on effects," leading to blind spots in transparency, accountability, and safety.

2. Methodology and Conceptual Framework

The paper does not present a specific empirical dataset or a new algorithm but rather introduces a conceptual framework and a lens for analysis called the Cascade.

Definition of "Cascade": A process where components in an AI supply chain interact through a series of inputs/outputs, creating cumulative effects on data processing, adaptation, and mediation. These effects can amplify, dampen, or nullify intended behaviors.
Two Intersecting Modalities: The authors analyze cascades through two distinct but overlapping viewpoints:
1. Component Cascade: Focuses on how technical transformations compound. The authors categorize transformations into five types:
  - Augmentative: Adding capabilities (e.g., RAG, tool use).
  - Behavioral: Modifying processing (e.g., system prompts, fine-tuning).
  - Contextual: Incorporating user data (e.g., memory, preferences).
  - Restrictive: Suppressing outputs (e.g., safety filters, moderation).
  - Presentational: Altering format/structure.
  - Mechanism: These layers interact, often creating emergent behaviors that are unpredictable when components are tested in isolation.
2. Stakeholder Cascade: Focuses on the distribution of control and visibility. Different actors (model providers, app developers, users) control different parts of the chain.
  - Visibility Gaps: No single entity sees the entire chain.
  - Control Gaps: A safety measure implemented by one actor can be overridden or undermined by a component managed by another (e.g., a developer's prompt overriding a model provider's safety filter).
Case Study: The framework is applied specifically to Large Language Models (LLMs) due to their ubiquity and complex supply chains, though the authors argue it applies to broader AI systems.

3. Key Contributions

The paper makes three primary contributions to the field of AI safety and auditing:

Introduction of the "Cascade" Concept: It formalizes the idea that AI safety is not a static property of a model but a dynamic outcome of a supply chain. It shifts the unit of analysis from the "model" to the "interaction of components."
Identification of Systemic Auditing Gaps: The authors identify three specific characteristics of AI supply chains that current audits fail to address:
- Non-modularity: Components blend like a "soup" rather than a "salad." Interactions create emergent effects that cannot be predicted by testing parts individually.
- Opacity: Stakeholders have limited visibility into upstream/downstream components, making it impossible to trace the root cause of a failure.
- Dynamism: Supply chains are not static; they change at runtime (especially with Agentic AI), creating new pathways and interactions that static audits cannot capture.
Proposed Research Directions: The paper outlines a roadmap for "Cascade-Aware Auditing," moving beyond model-centric evaluations to systems-oriented audits.

4. Results and Analysis

While the paper is theoretical, its analysis yields specific findings regarding the state of current auditing:

Current Audits are Insufficient:
- Capability Audits (e.g., MMLU benchmarks) test the model in a vacuum, ignoring how system prompts or retrieval tools alter behavior in deployment.
- Application Audits often treat the underlying FM as a black box, missing why a specific failure occurred (e.g., was it the model, the RAG retrieval, or the user's personalized settings?).
Emergent Harms: The paper illustrates how harms arise from cascades. For example, a "simple expression of stress" from a user might be escalated into a "high-priority crisis" due to a cascade of: user input $\rightarrow$ proprietary database retrieval $\rightarrow$ AIaaS crisis detection $\rightarrow$ open-weight model generation $\rightarrow$ corporate server reporting. The harm is not in the model, but in the chain of interactions.
Accountability Dislocation: Because control is distributed, it is difficult to assign responsibility. If a safety filter fails, is it the model provider, the developer who configured the prompt, or the user who customized the settings? The cascade obscures the "chain of command."

5. Significance and Future Directions

The paper argues for a paradigm shift in AI governance:

From Model-Centric to Systems-Oriented: Audits must evolve to examine the entire algorithmic supply chain, not just the foundation model or the final app.
Technical Research Directions:
- Developing methods for lineage tracking to trace data flow and transformations from model release to end-user interaction.
- Creating tools to evaluate component hierarchies and how higher-level transformations override lower-level ones.
- Assessing robustness at interaction points (e.g., how does a safety filter behave when combined with a specific RAG system?).
Human-Centered Research:
- Investigating how different stakeholders interpret and configure these chains.
- Developing participatory audit frameworks that combine technical analysis with stakeholder expertise.
Policy Implications: The paper highlights the tension between transparency (needed for auditing) and security (revealing system prompts might enable jailbreaks). It suggests "selective transparency" and controlled disclosures are necessary.

Conclusion:
The paper concludes that without addressing cascading effects, AI safety evaluations will remain incomplete. As AI systems become more agentic and dynamic, the complexity of these supply chains will increase, making the "cascade" lens essential for ensuring transparency, accountability, and safety in the next generation of AI deployments.