Assessment of accuracy of detection dog signaling behavior for the diagnosis of SARS-CoV-2 infection: A Canadian study

This Canadian study evaluated the accuracy of detection dogs in identifying SARS-CoV-2 via sweat samples, finding moderate sensitivity and specificity while demonstrating that failing to account for imperfect RT-PCR reference standards and repeated sample sniffing can significantly overestimate performance.

The Gist

Imagine you are trying to find a specific person in a crowded room, but instead of looking for their face, you are looking for their unique "scent signature." That is essentially what this study tried to do, but with a twist: they used dogs to sniff out people infected with the coronavirus (SARS-CoV-2).

Here is the story of the study, broken down into simple parts with some helpful analogies.

1. The Big Idea: Dogs as "Living Breathalyzers"

During the pandemic, we relied heavily on swab tests (RT-PCR) to find the virus. But those tests are like high-end, expensive lab equipment: they take time, cost money, and require a doctor to stick a long stick up your nose.

The researchers asked: Can we use dogs instead?
Just like a bloodhound can smell a specific scent in a forest, these dogs were trained to smell a specific "chemical fingerprint" (volatile organic compounds) that sick people leave behind in their sweat. Think of the dogs as living, breathing breathalyzers that can instantly tell if someone is sick just by smelling a sample.

2. The Experiment: The "Blind Taste Test"

The researchers set up a study in Toronto with three dogs:

• Dog 1: A "green" dog (new to scent work, like a rookie police officer).
• Dog 2 & 3: Experienced dogs (veterans who used to sniff out bedbugs).

They collected sweat samples from people visiting a clinic. Then, they put the samples in a special machine (a "sniffing station") that looked like a rack of holes. The dogs would sniff the holes. If they smelled the "sick" scent, they were trained to sit or look at their handler to signal, "I found it!"

The Golden Rule: The people handling the dogs and the people watching the videos didn't know which samples were from sick people and which were from healthy people. It was a double-blind taste test to make sure no one was cheating.

3. The Problem: The "Imperfect Ruler"

Here is the tricky part. To see if the dogs were doing a good job, the researchers had to compare the dogs' answers to the "Gold Standard" test (the RT-PCR swab).

But the researchers realized something important: The Gold Standard isn't actually perfect.
Imagine you are trying to measure the height of a building using a ruler that is slightly bent. Even if you measure perfectly, your result will be slightly wrong.

• The RT-PCR test sometimes misses the virus (false negatives) or gets confused (false positives).
• Most previous studies treated the RT-PCR test as if it were a perfect, unbreakable ruler.
• This study said, "Wait a minute! Let's use a mathematical model (Bayesian statistics) to account for the fact that our ruler is a little bent."

4. The Findings: What the Dogs Actually Did

When the researchers did the math correctly (accounting for the "bent ruler"):

• The Dogs Were Good, But Not Perfect: The dogs correctly identified sick people about 67% to 78% of the time. They correctly identified healthy people about 67% to 77% of the time.
• The "Bent Ruler" Effect: When they ignored the fact that the RT-PCR test was imperfect, they actually underestimated how good the dogs were. It's like blaming the dog for missing a scent when the "ruler" (the PCR test) was actually the one that failed to see the virus in the first place.
• The "Practice Makes Perfect" Trap: The study also looked at what happens if a dog sniffs the same sample twice. If you let a dog sniff the same "sick" sample over and over, it gets better at recognizing that specific bottle, not necessarily the disease. This made the dogs look smarter than they actually were (inflating their success rate).

5. The Takeaway: A Promising Tool, But Needs Polishing

The Good News:
Dogs have a real potential to be a fast, cheap, and non-invasive way to screen huge crowds (like at airports or schools). They don't need needles, and they can work quickly.

The Bad News:
We can't just grab any dog and say "Go sniff!"

• Training matters: The study showed that how you train the dog and how you test them changes the results.
• Statistics matter: We have to use smart math to account for the fact that our other tests (like PCR) aren't perfect.
• Standardization is key: Right now, every study does it differently. We need a universal "rulebook" so we know if a dog in Canada is as good as a dog in Germany.

In a Nutshell

This study is like a quality control check for a new kind of security scanner. It proved that the "dog scanners" work and are promising, but it also warned us that if we don't measure them carefully (accounting for imperfect reference tests and repeated sniffing), we might get the wrong idea about how good they really are.

The goal? To one day have a team of dogs at an airport entrance, sniffing samples and saying, "You're clear!" or "Stop, this person needs a closer look," saving time and money for everyone.

Technical Summary

1. Problem Statement

During the COVID-19 pandemic, the need for rapid, non-invasive, and scalable screening tools led to the investigation of Detection Dog Signaling Behavior (DDSB) as a diagnostic method. Dogs can detect specific Volatile Organic Compounds (VOCs) associated with SARS-CoV-2 infection in biological samples (e.g., sweat, breath). However, previous studies evaluating DDSB suffered from significant methodological flaws that likely distorted performance estimates. Key issues identified in the literature include:

• Imperfect Reference Standard: Most studies treated Reverse Transcription-Polymerase Chain Reaction (RT-PCR) as a perfect "gold standard," ignoring its known sensitivity and specificity limitations.
• Data Independence Violations: Many studies allowed dogs to sniff the same samples multiple times or reused training samples during validation without statistical adjustment, leading to data dependencies and potential overfitting.
• Methodological Heterogeneity: Lack of standardized protocols for training, sample handling, and validation made cross-study comparisons difficult.

This study aimed to estimate the diagnostic accuracy of DDSB in a Canadian population while rigorously accounting for the imperfection of RT-PCR and the bias introduced by repeated sample sniffing.

2. Methodology

Study Design and Population

• Setting: Conducted at St. Joseph's Health Centre, Toronto, between October and December 2021 (during the Omicron variant emergence).
• Participants: 2,358 individuals (mean age 34.7 years; 55.7% female) recruited from a COVID-19 assessment clinic.
• Samples: Participants provided nasopharyngeal swabs for RT-PCR and sweat samples (collected via gauze from the face, neck, and axillae) for canine detection.
• Training vs. Validation:
  • Training: 437 participants (72 RT-PCR positive, 365 negative) were used to train three dogs.
  • Validation: 146 unique participants (25 RT-PCR positive, 121 negative) were used for the primary validation.

Canine Subjects and Protocol

• Dogs: Three dogs were used: two Labrador Retrievers (one "green" dog with no prior scent detection experience, one with bed bug detection experience) and one German Shepherd (bed bug experience).
• Training: A six-step protocol over 35 days using sweat samples collected 4–6 months prior. Dogs were trained to sit or look at a handler when detecting a positive sample and to sit on a mat for negative samples.
• Validation Setup:
  • Samples were presented in a blinded, randomized manner using an aluminum sniffing station.
  • Sessions were video-recorded and reviewed by two independent, blinded observers.
  • Bias Assessment: The study specifically analyzed the impact of repeated sniffing (where the same sample was presented to a dog more than once) and the imperfection of RT-PCR.

Statistical Analysis

• Primary Analysis: Compared DDSB results against RT-PCR results.
• Bayesian Latent Class Models (BLCM): To address the imperfect nature of RT-PCR, the authors used BLCM. They applied informative priors for RT-PCR sensitivity (median 94.5%) and specificity (median 98.4%).
• Sensitivity Analyses:
  1. Reference Standard: Compared models assuming RT-PCR was perfect vs. imperfect.
  2. Repeated Samples: Compared analyses excluding repeated samples (unique participants) vs. including them (repeated sniffing of the same samples).

3. Key Contributions

• Statistical Rigor: This is one of the first studies to explicitly model the imperfection of RT-PCR when evaluating canine detection, using Bayesian Latent Class Models to derive more accurate sensitivity and specificity estimates.
• Bias Quantification: The study quantified the specific magnitude of bias introduced by two common methodological errors: treating RT-PCR as perfect and failing to adjust for repeated sample sniffing.
• Standardization: The study utilized a detailed Standard Operating Procedure (SOP) for sample collection, storage, and the use of a specialized sniffing station to minimize contamination and handler bias.

4. Results

Diagnostic Performance (Unique Participants, RT-PCR Imperfect)
When accounting for the imperfect accuracy of RT-PCR, the posterior median sensitivity and specificity for the three dogs were:

• Sensitivity: Ranged from 67% to 78% (95% Credible Interval [CrI]: 29%–99%).
• Specificity: Ranged from 67% to 77% (95% CrI: 53%–87%).

Impact of Assumptions on Accuracy

• Assuming RT-PCR is Perfect: When the model assumed RT-PCR was perfect, sensitivity estimates decreased by 7.9% to 9.0%, while specificity remained largely unchanged. This indicates that treating RT-PCR as a perfect gold standard leads to an underestimation of the dog's true sensitivity.
• Impact of Repeated Sniffing: Including repeated positive samples in the analysis without statistical adjustment overestimated sensitivity by 7.9% to 13.4% (assuming imperfect RT-PCR) and 11.4% to 18.3% (assuming perfect RT-PCR). Specificity estimates were unaffected by the inclusion of repeated samples.

Dog Performance

• The "green" dog (Dog 1) showed slightly lower sensitivity estimates compared to the experienced dogs (Dogs 2 and 3), but the differences were not statistically significant due to wide credible intervals.

5. Significance and Conclusion

Significance

• Methodological Correction: The study demonstrates that previous high-performance estimates in canine detection literature may be skewed by uncorrected biases. Specifically, ignoring RT-PCR imperfections underestimates sensitivity, while ignoring repeated measurements overestimates it.
• Clinical Potential: Despite the moderate sensitivity (67–78%), DDSB shows promise as a non-invasive, rapid, and low-cost screening tool, particularly for high-throughput settings like airports or mass gatherings where RT-PCR is logistically challenging.
• Future Directions: The authors emphasize that for DDSB to be deployed effectively, there is an urgent need for:
  1. Standardized training and validation protocols.
  2. Statistical methods that account for reference standard imperfections and data dependencies.
  3. Further research to determine if the observed accuracy translates to cost-effective, real-world clinical utility.

Limitations
The study noted several limitations, including a delay between sample collection and training (4–6 months), which may have affected VOC stability; an unexpectedly early termination of the validation phase; and the inability to fully estimate between-dog and within-dog variability due to the unbalanced design of repeated samples.

In conclusion, while detection dogs show potential as a screening tool, their reported accuracy is highly sensitive to study design and statistical modeling. Rigorous protocols and advanced statistical corrections are essential to validate their utility in public health.