AGREEMENT AND ERROR RATES IN ANTIMICROBIAL SUSCEPTIBILITY TESTING FOR THREE COMMERCIAL AUTOMATED SYSTEMS: A SYSTEMATIC LITERATURE REVIEW AND META-ANALYSIS

This systematic review and meta-analysis of 39 studies found that while the BD Phoenix, Vitek 2, and MicroScan automated antimicrobial susceptibility testing systems generally demonstrate comparable overall performance, the Vitek 2 system exhibits a significantly higher rate of very major errors compared to MicroScan, with error patterns varying by organism type and susceptibility criteria.

The Gist

Imagine your body is a fortress under attack by invisible invaders: bacteria. Sometimes, these bacteria are tough and have learned to ignore the weapons (antibiotics) doctors usually use to fight them. This is called Antimicrobial Resistance (AMR), and it's a global emergency.

To win this war, doctors need a "scout" to tell them exactly which weapon will work. That scout is a lab test called Antimicrobial Susceptibility Testing (AST). It checks if a specific bacteria is scared of a specific antibiotic.

In the past, this scouting was slow and done by hand. Today, we have automated machines that do the job faster. This paper is a massive report card comparing the three biggest, most popular machines in the world:

1. Phoenix (made by BD)
2. Vitek 2 (made by bioMérieux)
3. MicroScan (made by Beckman Coulter)

The authors of this paper didn't just test one sample; they acted like super-sleuths. They dug through thousands of old research papers (like searching through a giant library) to find 39 studies that compared these machines against the "Gold Standard" (a perfect, slow, manual method). They wanted to see: Which machine makes the fewest mistakes?

The Four Ways Machines Can Mess Up

To understand the results, you have to know the four ways a machine can get a test wrong:

1. Categorical Agreement (CA): The machine and the Gold Standard agree on the answer (e.g., both say "Susceptible"). This is the "Pass" grade.
2. Essential Agreement (EA): The machine is almost right. It might say the bacteria needs a slightly higher dose of the drug, but it's close enough to be useful.
3. Major Error (ME): The machine says the bacteria is "Resistant" (the drug won't work), but the Gold Standard says it actually does work.
   • Analogy: The machine tells the doctor, "Don't use this key, it's broken!" but the key actually fits the lock. The doctor might throw away a good key and pick a worse one.
4. Very Major Error (VME): This is the nightmare scenario. The machine says the bacteria is "Susceptible" (the drug works), but the Gold Standard says it's actually "Resistant" (the drug won't work).
   • Analogy: The machine tells the doctor, "This key is perfect!" but the lock is actually jammed. The doctor gives the patient the key, the door doesn't open, the infection spreads, and the patient gets sicker. This is the most dangerous mistake.

The Big Findings: Who Won the Race?

The researchers looked at the data and found some interesting patterns:

1. The "Good News": They are all pretty good.
For the most part, all three machines agreed with the Gold Standard about 90-95% of the time. They are all reliable scouts. If you use any of them, you are likely to get a correct answer.

2. The "Bad News": Vitek 2 had more "Nightmare Scenarios" (VMEs).
When looking specifically at the dangerous "Very Major Errors," Vitek 2 made more mistakes than the other two.

• The Math: Vitek 2 was about 74% more likely to make this specific type of error compared to MicroScan.
• The Context: This didn't happen with every single bacteria or every single drug. It was mostly a problem with Gram-negative bacteria (a specific family of tough bugs) and certain types of antibiotics (like aminoglycosides).
• The Analogy: Imagine three cars driving on a highway. They all arrive at the destination safely 95% of the time. But, the Vitek 2 car has a slightly higher chance of taking a wrong turn into a ditch when driving through a specific type of fog (Gram-negative bacteria).

3. The "It Depends" Factor:

• Gram-Positive vs. Gram-Negative: The machines behave differently depending on the type of bacteria. Vitek 2 was actually better (or at least equal) for Gram-positive bugs, but struggled more with Gram-negative ones.
• The Rules Change: The "Gold Standard" rules for what counts as "Resistant" or "Susceptible" change every year (like updating the rules of a video game). The paper found that as these rules changed over the years, the error rates of the machines shifted. Sometimes a machine that was perfect yesterday might make more mistakes today if its software isn't updated to match the new rules.

Why Should You Care?

If a machine makes a Very Major Error, a doctor might prescribe an antibiotic that doesn't work.

• The patient stays sick longer.
• They might need to stay in the hospital (ICU) longer.
• The infection can spread to others.
• In severe cases (like sepsis), it can be fatal.

The study found that while all three machines are generally safe, MicroScan and Phoenix were slightly more consistent at avoiding these dangerous errors, especially with the tricky Gram-negative bacteria.

The Bottom Line

Think of these machines as high-tech GPS systems for doctors.

• All three usually get you to the right destination.
• Vitek 2 is a great GPS, but it occasionally gets confused in specific neighborhoods (certain bacteria) and might send you down a dead end.
• MicroScan and Phoenix were slightly more reliable in those tricky neighborhoods.

The Takeaway: No machine is perfect. Because bacteria are constantly evolving and the rules keep changing, doctors and labs can't just set the machine and forget it. They need to keep checking the machines, updating their software, and double-checking results when things look suspicious. It's a reminder that in the fight against superbugs, even our best tools need constant maintenance.

Technical Summary

1. Problem Statement

Antimicrobial resistance (AMR) is a critical global public health threat, causing significant mortality and complicating the treatment of infections. Accurate and timely Antimicrobial Susceptibility Testing (AST) is essential for guiding appropriate therapy and stewardship. While automated commercial AST systems (specifically BD Phoenix™, bioMérieux Vitek® 2, and Beckman Coulter DxM MicroScan WalkAway) are widely used to improve efficiency, there is a lack of comprehensive, head-to-head comparative data regarding their performance against common reference methodologies.

The study addresses the need to quantify:

• Categorical Agreement (CA) and Essential Agreement (EA) between these systems and reference methods.
• Error Rates, specifically Very Major Errors (VMEs) (false susceptibility, leading to treatment failure) and Major Errors (MEs) (false resistance, leading to unnecessary broad-spectrum use).
• How these metrics vary by organism type (Gram-positive vs. Gram-negative), drug class, and evolving interpretive guidelines (CLSI vs. EUCAST).

2. Methodology

The study followed PRISMA guidelines and the Cochrane Collaboration Diagnostic Test Accuracy Working Group standards.

• Search Strategy: A systematic search of MEDLINE and Embase (via Ovid) up to October 17, 2024, combined with hand-searching, yielded 319 unique records.
• Inclusion Criteria: Studies comparing at least two of the three target systems (Phoenix, Vitek 2, MicroScan) against a common reference method (e.g., broth microdilution) with reported CA, EA, VME, or ME data.
• Data Extraction: 39 studies met the inclusion criteria, encompassing over 20 genera, 35 species, and 25 drug classes.
• Quality Assessment: A modified Newcastle-Ottawa Scale and a modified GRADE methodology were used to assess risk of bias (detection, performance, spectrum) and overall quality of evidence.
• Statistical Analysis:
  • A random-effects meta-regression model was used to calculate rate ratios (RR) and 95% confidence intervals (CI).
  • Subgroup analyses were performed based on Gram stain, guideline (CLSI vs. EUCAST), drug class, and CLSI guideline year.
  • Sensitivity analyses excluded studies using agar dilution or specific high-impact studies (e.g., Nielsen et al., 2015) to test robustness.

3. Key Contributions

• First Comprehensive Meta-Analysis: This is the first systematic review to simultaneously compare CA, EA, VME, and ME across all three major commercial AST platforms using a unified statistical framework.
• Granular Subgroup Analysis: The study moves beyond aggregate data to analyze performance nuances, specifically isolating the impact of drug classes (e.g., aminoglycosides) and organism types on error rates.
• Temporal Analysis: The authors analyzed how VME rates have evolved over time in relation to updates in CLSI breakpoints, highlighting the dynamic nature of AST accuracy.
• Clinical Threshold Assessment: The study evaluated the proportion of antibiotic/organism combinations falling below the clinically accepted 3% VME threshold, providing actionable data for laboratory quality assurance.

4. Key Results

Overall Performance

• Agreement: No statistically significant differences were found between the three systems for Categorical Agreement (CA) (all ≥90%) or Essential Agreement (EA) (all >92%).
• Major Errors (ME): No significant differences were observed in ME rates across the systems.
• Very Major Errors (VME): This was the primary differentiator.
  • Vitek 2 demonstrated a significantly higher overall VME rate compared to MicroScan (Rate Ratio [RR] = 1.74; p=0.045).
  • Vitek 2 showed a marginally higher VME rate compared to Phoenix (RR = 1.44; p=0.062).
  • Phoenix and MicroScan showed no significant difference in overall VME rates.

Subgroup Findings

• Gram-Negative Organisms: Phoenix had significantly fewer VMEs than Vitek 2 (RR = 0.53; p<0.001). Vitek 2 had a trend toward higher VMEs than MicroScan (p=0.076).
• Gram-Positive Organisms: No significant differences were found, though Phoenix showed a marginally higher VME rate than MicroScan.
• Drug Class Impact: Vitek 2 exhibited notably higher VME rates for aminoglycosides and monobactams. When aminoglycosides were excluded from the analysis, the performance gap between Vitek 2 and MicroScan narrowed significantly (p=0.099), suggesting drug-specific algorithmic issues drive the overall discrepancy.
• Guideline Impact:
  • Under CLSI criteria, Vitek 2 had a marginally higher VME rate than MicroScan (p=0.064).
  • Under EUCAST criteria, no significant differences were observed between any systems.
• Acceptability Threshold: When assessing the percentage of antibiotic/organism combinations with VME rates <3% (the CLSI acceptable limit):
  • MicroScan: 77.5% (100/129)
  • Phoenix: 76.1% (118/155)
  • Vitek 2: 65.1% (127/195)
  • A Chi-square test indicated a significant association between platform and acceptable VME rates (p=0.02).

5. Significance and Implications

• Clinical Safety: The higher VME rates associated with Vitek 2, particularly for Gram-negative organisms and specific drug classes (aminoglycosides, monobactams), pose a risk of inappropriate therapy (e.g., treating resistant infections with ineffective drugs). This is critical in high-mortality scenarios like sepsis.
• Dynamic Accuracy: The study underscores that AST accuracy is not static. Performance varies based on the specific organism-drug combination and the version of interpretive criteria (breakpoints) applied. As breakpoints evolve (e.g., CLSI updates), systems must be continuously validated.
• Laboratory Practice: The findings suggest that clinical laboratories cannot rely solely on manufacturer claims of equivalence. They must perform rigorous, ongoing verification, particularly for high-risk drug/organism combinations where specific systems may exceed the 3% VME threshold.
• Future Directions: The authors call for continued post-market surveillance and more granular, real-world monitoring of AST systems to ensure they remain aligned with evolving resistance patterns and regulatory standards.

In conclusion, while all three systems generally perform well, Vitek 2 exhibited a statistically higher propensity for Very Major Errors compared to MicroScan and Phoenix, driven largely by specific drug classes and Gram-negative organisms. This highlights the necessity for system-specific validation and continuous monitoring in clinical microbiology.