Technical Summary: SETA – Statistical Fault Attribution for Compound AI Systems

1. Problem Statement

Modern AI systems are increasingly architected as Compound AI Systems (CAS): complex pipelines comprising specialized models, retrievers, and software components rather than monolithic networks. While this modularity aids development, it introduces significant challenges in testing, debugging, and ensuring safety. Small, localized faults in upstream components (e.g., data drift, model miscalibration) can propagate through the pipeline, leading to catastrophic system-level failures.

Current robustness testing techniques face two primary limitations:

1. End-to-End (E2E) Testing: Treats the system as a black box. While it can identify that a failure occurred, it cannot attribute the fault to specific internal components.
2. Isolated Component Testing (ICT): Tests modules in isolation but fails to detect emergent faults arising from dynamic interactions between modules or to analyze how errors propagate through the pipeline.

Existing tools (e.g., adversarial attacks like FGSM, formal verification tools like Reluplex, or coverage-based tools like DeepXplore) are largely designed for single-network models. They lack the capability to perform fine-grained fault localization within composed pipelines or to explain why a failure occurred within the execution context. Furthermore, the "test oracle problem"—the difficulty of defining exact correct outputs for intermediate components without ground truth—remains a barrier.

2. Methodology: The SETA Framework

The authors propose SETA (Statistical Fault Attribution), a modular framework that combines Metamorphic Testing (MT) with Execution Trace Analysis. SETA operates without requiring explicit ground-truth oracles by defining behavioral specifications through metamorphic relations.

2.1 Metamorphic Relations (MRs) as Pseudo-Oracles

SETA addresses the oracle problem by verifying the preservation of user-specified properties (Metamorphic Relations) across input transformations, rather than checking single input-output pairs.

• Definition: An MR specifies the expected relationship between a system's output for a source input $x$ and a perturbed input $\tilde{x} = g(x)$.
• Generic Formulation: Relations are defined using Kronecker delta ($\delta$) for equality constraints and Heaviside step functions ($H$) for inequality constraints.
• Task-Specific MRs: The framework defines MRs for four key vision tasks:
  • Image Classification: Invariance of labels or bounded $L_\infty$-norm differences in probability distributions under spatial transformations.
  • Object Localization: Constraints on Intersection over Union (IoU) between predicted and ground-truth bounding boxes.
  • Object Detection: A combination of classification and localization constraints applied to sets of detected objects.
  • Image Segmentation: Pixel-level equivalence or relaxed IoU-based overlap constraints for semantic masks.
• Composite Metamorphic Relations (CMRs): For complex components, multiple individual MRs are combined via logical conjunction (product operator) to form a single composite score ($S_i$). A component is considered correct only if all constituent relations hold.

2.2 State-Based Execution Trace Analysis

SETA models the compound AI system as a state-transition system where states represent computational modules.

• Execution Trace Tree ($T(x)$): During inference, the framework instruments the system to record a dynamic execution graph. For each input $x$ and its perturbed variant $\tilde{x}$, it constructs a tree capturing:
  • Activated modules.
  • Input/output tensors.
  • Routing decisions (which downstream modules are invoked).
  • Scoring function results ($S_q$) comparing reference and perturbed runs.
• Trace Alignment: The framework aligns the trace of the perturbed input against the reference trace to detect deviations in activation sets (e.g., a module triggered in one but not the other) or output inconsistencies.

2.3 Statistical Fault Attribution

To distinguish between the first point of failure and the root cause (which may be an upstream error that propagated), SETA employs a statistical attribution mechanism.

• Failure Contribution (FC) Score: The framework defines a random variable $Z_i$ representing the deviation of module $i$. The FC score for module $i$ is the expected deviation conditioned on a system-level failure:
  $$FC_i = E_{(x, \tilde{x}) \sim D} [Z_i \cdot \mathbb{I}(S(x, \tilde{x}) = 0)]$$
  This measures how frequently a component violates its specification given that the entire system failed.
• Attribution Weights: Scores are normalized across all modules to derive interpretable attribution weights ($\alpha_i$), quantifying each component's relative contribution to system-level unreliability.
• Algorithm: The process involves iterating through a dataset and perturbation set, recording traces, computing system-level scores, and aggregating deviations to produce a normalized attribution vector.

3. Key Contributions

1. Modular Framework: Proposes SETA, integrating Metamorphic Testing with Execution Trace Analysis to localize faults in complex AI pipelines by empirically tracing perturbation propagation.
2. Oracle-Free Specifications: Supports scalable, testable abstractions for black-box models by defining behavioral specifications via metamorphic relations.
3. Extensibility: Due to its modular design, users can define and plug in different classes of metamorphic relations, supporting interpretability in opaque systems.
4. Fine-Grained Analysis: Demonstrates the ability to pinpoint failure origins and surface hidden vulnerabilities in multi-stage pipelines where traditional tools fail.

4. Experimental Results

The framework was evaluated on two distinct systems:

1. Autonomous Rail-Inspection Vision System: A real-world system with 6 modules (1 object detector, 5 classifiers) processing railway imagery.
   • Setup: Tested against 15 image corruption techniques (e.g., fog, snow, motion blur) from the imagecorruptions library.
   • Findings: SETA successfully identified that the object detection module ($f_0$) was highly sensitive to fog and motion blur, while downstream classifiers showed varying robustness. It computed attribution scores ($\alpha_i$) that highlighted specific modules (e.g., $f_1$ and $f_3$) as primary contributors to system failures under specific perturbations.
2. Ensemble Classification Model (CIFAR-10): A sanity check using an ensemble of CNNs (ResNet18, VGG16, Custom CNN).
   • Findings: SETA's predicted accuracies for individual models were within 2-3% of actual accuracies, correctly identifying the "Custom CNN" as the worst-performing model.
3. OCR System: A text-based pipeline using EasyOCR.
   • Findings: The framework was adapted using Levenshtein distance as the metamorphic relation. It successfully quantified robustness and attributed failures to specific modules based on text sequence deviations, demonstrating applicability beyond vision pipelines.

5. Significance and Claims

The paper positions SETA as a proof-of-concept for diagnosing emergent failures in compound AI systems. Its significance lies in bridging the gap between end-to-end testing and isolated component testing.

• Diagnostic Precision: Unlike tools that only detect that a failure occurred, SETA provides a statistical attribution of which module is most responsible, guiding engineers to vulnerable submodules.
• Scalability: By using metamorphic relations, it bypasses the need for ground-truth oracles, making it applicable to complex, real-world pipelines where intermediate ground truth is unavailable.
• Limitations and Future Work: The authors modestly acknowledge that current attribution is correlational, not causal. High scores indicate an association with failure but do not definitively prove causality. The precision depends on the completeness of the hand-crafted metamorphic relations. Future work aims to integrate causal inference, program synthesis for automatic MR generation, and extension to multimodal and reinforcement learning systems.

In summary, SETA establishes a foundation for compositional testing and explainable fault localization, offering a systematic approach to ensure the reliability of complex, multi-component AI systems.