Stress-testing post-market medical-device reporting signals: an empirical study of temporal, entity-label and comparator robustness in FDA MAUDE

Technical Summary: Stress-testing Post-Market Medical-Device Reporting Signals

Problem Statement
Post-market surveillance of medical devices often relies on spontaneous reporting databases, such as the FDA Manufacturer and User Facility Device Experience (MAUDE). A common analytical approach involves calculating a Reporting Odds Ratio (ROR) over a broad time period to identify signals of disproportionate reporting. However, the authors argue that a signal remaining positive over a broad period does not necessarily indicate analytical robustness. Such signals may be artifacts of specific analytical choices, including:

• Temporal instability: Fluctuations masked by long time windows.
• Entity-label dependence: Signals driven by a single manufacturer or model rather than a class-wide issue.
• Comparator sensitivity: Results dependent on a narrow definition of the comparator group.
• Cumulative lock-in: Historical evidence persisting even after local support weakens.

The study addresses the lack of empirical quantification regarding how perturbing time scales, entity labels, comparator specifications, and accumulation strategies affects signal classification.

Methodology
This retrospective empirical methodological study utilized FDA MAUDE records received between January 1, 2020, and December 31, 2025. The primary unit of analysis was the unique MDR_REPORT_KEY.

• Study Universe: The analysis focused on 61 product-code/mechanism pairs. From these, 19 pairs met formal statistical criteria (ROR > 1, Lower Confidence Limit [LCL] > 1, and Benjamini–Hochberg FDR < 0.05) in both an "earlier" period (2020–2023) and a "later" period (2024–2025).
• Stress-Testing Framework: The study applied several perturbation tests to these 19 signals:
  1. Temporal Sensitivity: Calculated quarterly RORs (2024Q1–2025Q4) and performed "leave-one-year-out" (LOO) analyses (2020–2025).
  2. Entity-Label Removal: Systematically removed the leading normalized manufacturer name and the leading normalized model name from the target signal reports to test for single-entity dependence.
  3. Comparator Perturbation: Performed "leave-one-comparator-code-out" (LOCO) analyses, removing individual codes from the comparator set to test for dependence on specific comparator definitions.
  4. Historical Replay: Re-evaluated signals at 24 cumulative cutoffs (2020Q1–2025Q4) and assessed local persistence using 21 rolling four-quarter windows.
• Classification Tiers: Signals were categorized into tiers (A, B, C, U) based on their performance across these stress tests. Tier A represented the highest robustness (passing all temporal, entity, and comparator criteria).
• Contextual Linking: Recall events were linked to signals to provide regulatory context, though not as a gold standard for validation.

Key Contributions
The paper provides an empirical demonstration of how analytical specifications influence signal stability in the MAUDE database. Its primary contributions include:

• Differentiation of Robustness: It establishes that broad-period consistency is insufficient evidence of methodological stability. The study quantifies how many signals that pass standard thresholds fail under specific perturbations.
• Multi-Dimensional Stress Testing: It introduces a framework for separating temporal fragility, entity-label dependence, and comparator sensitivity, arguing these should be inspected separately rather than collapsed into a single score.
• Cumulative vs. Rolling Analysis: It highlights the distinction between cumulative evidence (which can retain historical "lock-in") and rolling local analysis (which reveals current instability).
• Tiered Classification: It proposes a transparent operational hierarchy (Tier A–U) for classifying signal robustness based on specific failure modes.

Results

• Signal Stability: Of the 19 signals meeting broad-period criteria in both time windows, only 5 (Tier A) met all temporal and entity-label robustness criteria.
• Comparator Sensitivity: Under comparator perturbation, 10 signals remained robust (Comparator-A), 4 retained direction but lost precision (Comparator-B), and 5 reversed direction (Comparator-C).
• Temporal Persistence: The five most robust signals maintained an LCL > 1 in all 21 rolling four-quarter windows. In contrast, the predefined "fragile" group (13 signals) had a median rolling persistence of 0.857. This difference was statistically significant (rank-biserial = 0.846; two-sided P = 0.006).
• Cumulative vs. Local: All 19 signals eventually reached cumulative confirmation, illustrating that cumulative analysis can obscure local weakening. For example, the MKJ signal maintained an ROR > 1 in all rolling windows but had an LCL > 1 in only 6/21 windows, demonstrating a loss of precision masked by the point estimate.
• Illustrative Cases: The study highlighted four archetypes (DTZ, DWF, LOX, MKJ). DTZ, DWF, and LOX showed high robustness across all tests, while MKJ demonstrated fragility regarding manufacturer dependence and rolling precision despite meeting broad-period criteria.

Significance and Claims
The authors position this work strictly as an empirical methodological demonstration, not a validated framework, prediction tool, or causal safety assessment.

• Methodological Scope: The findings describe reporting disproportionality within the MAUDE database, not incidence, absolute risk, comparative safety, causality, or prediction.
• Practical Implication: The study argues that broad-period consistency alone should not be treated as a stable finding. Instead, temporal, entity-label, and comparator stress tests provide necessary complementary sensitivity information.
• Limitations: The authors explicitly note that the study definitions were frozen after reviewing the full 2020–2025 record (not truly prospective), the sample size for robustness comparisons was small, and MAUDE lacks an exposure denominator. They emphasize that external surveillance datasets with exposure denominators are required to test the generalizability of these findings.

In conclusion, the paper asserts that treating post-market reporting signals as stable findings requires testing them across diverse analytical specifications before acceptance, as broad-period positivity can conceal materially different levels of analytical robustness.