Concordance of Automated ACMG Variant Classification withExpert-Curated Assertions: A Systematic Evaluation Using theClinGen Evidence Repository

Imagine your DNA as a massive, ancient instruction manual for building a human being. It's written in a four-letter code (A, C, G, T) and stretches for billions of characters. Most of the time, this manual works perfectly. But sometimes, a single letter gets swapped, deleted, or added—a "typo" in the code. In the world of clinical genetics, these typos are called variants. Some are harmless, like a typo in a footnote that doesn't change the story. Others are dangerous, like a typo that turns "stop" into "go," causing the body's machinery to run wild and leading to disease.

The big challenge for doctors and scientists isn't finding these typos; it's figuring out which ones are dangerous and which are just harmless noise. To do this, they use a strict rulebook created by medical experts. This rulebook acts like a giant scoring system. If a typo has certain "bad" features (like breaking a gene completely), it gets points toward being "Pathogenic" (dangerous). If it has "good" features (like being common in healthy people), it gets points toward being "Benign" (safe). The goal is to sort millions of these typos into five categories: Pathogenic, Likely Pathogenic, Uncertain, Likely Benign, and Benign. Getting this right is a matter of life and death, but doing it by hand is slow, expensive, and prone to human disagreement. So, scientists have been building computer programs to do the sorting automatically. But how good are these robots really?

---

The Robot Judge vs. The Human Panel

This paper puts a new computer program, called VarTriage, to the ultimate test. The authors wanted to see if this automated robot could match the decisions of the "Gold Standard": a team of human experts who have spent years curating the ClinGen Evidence Repository. Think of the ClinGen repository as the "Hall of Fame" for genetic typos, where the most trusted human experts have already decided, with high confidence, whether a specific variant is dangerous or safe.

The researchers fed 15,334 of these expert-approved variants into VarTriage. They then compared the robot's answers against the human experts' answers to see how often they agreed.

The Big Findings:

• The Robot is a Great Detective for "Bad" Typos: When it came to spotting dangerous variants, VarTriage was quite impressive. It correctly identified 68.9% of the truly pathogenic variants. More importantly, when it did say a variant was dangerous, it was right 80.5% of the time.
• The "Binary" Superpower: If you ignore the "Uncertain" middle ground and just ask, "Is this bad or is it good?", the robot and the humans agreed almost perfectly. Their agreement score (called Cohen's kappa) was 0.98, which is basically a perfect handshake. This means the robot is excellent at keeping the "bad" and "good" piles separate, even if it sometimes isn't sure exactly how bad or good something is.
• The "Uncertain" Pile Problem: The robot's main weakness was that it tended to dump too many variants into the "Uncertain" (VUS) pile. While humans might have enough extra clues to say, "This is definitely safe," the robot, lacking those specific clues, played it safe and said, "I don't know." This resulted in a much lower success rate for spotting "safe" variants (only 16.9% compared to the human experts' 80.2%).

What Drives the Robot's Brain?

The study dug deep to see why the robot made the choices it did. They found two main engines powering its decisions:

1. The "Two-Point" Rule: The researchers discovered that the robot's biggest boost in accuracy came from a specific math trick called the "Bayesian relaxed combining rule." Imagine you have two pieces of evidence that are each "moderately suspicious." The old rules said you needed a "very strong" piece of evidence to convict. The new rule says, "Hey, two moderate pieces of evidence are actually enough to be pretty sure." Using this rule alone boosted the robot's ability to find bad variants by 36.9 percentage points.
2. The REVEL Score: The most powerful single clue the robot used was a prediction score called REVEL. This is a sophisticated calculator that looks at the shape and chemistry of the protein to guess if a change is bad. When the researchers turned off the REVEL score, the robot's ability to find bad variants crashed by 36.9 percentage points. It turned out that REVEL was the robot's most important superpower.

What the Robot Missed

The paper explicitly points out what the robot couldn't do. The reason it struggled to identify "safe" variants is that it lacks access to certain types of evidence that humans have.

• Lab Tests: Humans can look at results from actual lab experiments (functional assays) that prove a gene is working fine. The robot can't see these.
• Family Trees: Humans can look at how a variant runs through a family tree to see if it causes disease. The robot can't do that.
• Trusted Sources: Humans can read a trusted medical database that says, "We know this variant is safe." The robot was programmed to ignore this to avoid circular logic (where the robot just repeats what the database says).

Because the robot couldn't use these "non-computational" clues, it was much less confident about calling things "Benign."

The Verdict

The paper concludes that while we can't yet replace the human expert panel with a robot, we can build a very helpful assistant. VarTriage, using just 10 of the 28 possible rules, can act as a highly reliable filter. It can quickly sort out the clearly dangerous and clearly safe variants from the "maybe" pile, leaving the human experts to focus only on the tricky cases.

The authors suggest that the next big leap won't come from making the robot smarter at math, but from giving it access to more types of data—specifically, those lab test results and family history clues that currently keep it in the dark. Until then, the robot is a fantastic partner for the "bad vs. good" decision, but it still needs a human to help it feel confident about the "safe" ones.