💻 bioinformatics

A learned fungal ITS embedding does not outperform correctly configured alignment in open-world evaluation

This study demonstrates that a purpose-trained fungal ITS embedding does not outperform a correctly configured alignment-based approach in open-world evaluation, revealing that methodological flaws such as heuristic defaults, missing coverage filters, and data leakage—not the representation itself—were responsible for previously reported advantages of learned embeddings.

Original authors: O'Brien, A., Gardette, A.

Published 2026-09-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: O'Brien, A., Gardette, A.

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

In the hidden world beneath our feet, fungi are everywhere, yet they remain largely invisible to the naked eye. To understand this vast kingdom of life, scientists rely on a specific stretch of genetic code known as the ITS region. Think of this code as a unique barcode printed on every fungal species. When researchers collect a sample from soil or air, they sequence this barcode and try to match it against a massive library of known fungi to identify what they have found. However, nature is full of surprises. Often, the fungus in the sample is so new or so distant from known relatives that it does not match any entry in the library. The challenge for scientists is twofold: they must decide when a fungus is truly new and should be left unnamed, and they must place it as accurately as possible within the family tree of life, even if the exact species is unknown.

For years, the standard way to solve this matching problem has been to line up the new genetic sequence against every known sequence in the library, looking for the closest fit. This method, called alignment, is like checking every page of a dictionary to find the word that looks most like the one you are trying to spell. Recently, a newer, more high-tech approach has emerged. Instead of checking every page, scientists have begun using artificial intelligence to translate the genetic code into a mathematical point in a vast, multi-dimensional space. In this new system, similar fungi are placed close together, and different ones are pushed apart. The hope was that this learned system could spot new fungi and place them in the right family faster and more accurately than the old, slow method of checking every page.

A team of researchers set out to test whether this new artificial intelligence method truly outperformed the traditional approach. They built a custom AI model trained specifically to understand fungal genetics, using a massive, up-to-date collection of fungal sequences. To ensure a fair test, they designed a strict experiment where the AI and the traditional method were judged on the exact same set of unknown samples. They also took extraordinary care to seal their test data before the experiment began, ensuring that no part of the test could accidentally influence the training of the AI. This "firewall" meant that when the results were finally revealed, they would be a true measure of how well the methods worked on completely new data.

The researchers discovered that the outcome of the test depended entirely on how the rules were set. When they ran the traditional alignment method with its default, standard settings, the new AI model appeared to perform better. However, the researchers realized that the standard settings were not actually set up to do the best possible job. The traditional method had been skipping some potential matches and stopping its search too early. When the team reconfigured the traditional method to search exhaustively and carefully, checking every possibility without shortcuts, the results changed completely. The old method, now running at its full potential, became more accurate than the AI at identifying known fungi and placing new ones into the correct family groups.

The study revealed that the AI model had a hidden weakness: its performance changed depending on how many samples were fed to it at once. When the AI processed samples in large groups, the results were slightly different than when it processed them one by one. This inconsistency meant the AI was not truly stable. In contrast, the traditional method gave the same result every time, regardless of how the data was grouped. Furthermore, the researchers found that the AI's ability to spot new fungi was not actually better than the traditional method when measured at the specific points where a scientist would make a real-world decision. The AI did not find more new fungi, nor did it make fewer mistakes. In fact, the traditional method found more new fungi while making fewer errors.

Perhaps most surprisingly, the researchers found that the AI's success in earlier tests was partly due to a flaw in how the test was set up. Some of the "new" fungi in the test had actually been seen by the AI during its training, even though they were supposed to be hidden. When the researchers fixed this by removing those specific fungi from the training data and retraining the AI, the gap between the two methods widened. The traditional method pulled further ahead, showing a clear and statistically significant advantage. The AI's ability to compare different parts of the genetic code, a feature that was supposed to be its superpower, turned out to be no better than what the traditional method could do simply by looking at the shared parts of the code directly.

The final conclusion was clear: a carefully configured traditional method matched or exceeded the performance of the new artificial intelligence model in every category that mattered. The AI did not outperform the old method; it was the way the old method was being tested that had made it look weaker. The study showed that the choice of evaluation rules, such as how strictly the search was conducted and how the data was grouped, decided the winner. The researchers emphasized that these checks are simple and inexpensive to apply to any new method. They argued that before declaring a new artificial intelligence system superior, scientists must ensure that the traditional baseline is running at its best, that the data is truly new to the system, and that the results are stable under different conditions. In the end, the most reliable tool for identifying the unseen fungal world remained the one that had been used for decades, provided it was used correctly.

Technical Summary: "A learned fungal ITS embedding does not outperform correctly configured alignment in open-world evaluation"

Problem Statement
Fungal metabarcoding frequently yields sequences where the closest named relative is distant or the genus is absent from reference collections. In such "open-world" scenarios, methods must decide whether to withhold a genus assignment (novelty detection) and place the query as precisely as possible. Learned sequence embeddings are increasingly proposed to solve this by creating a unified similarity space for unseen taxa and different barcode views (ITS1, ITS2, ITS-core). However, the paper argues that these methods are often evaluated against weaker variants of themselves or default-configured alignment baselines, using metrics like AUROC that may not reflect deployable operating points, and treating queries as independent when they share genus-level information. The authors question whether a purpose-trained encoder truly outperforms percent identity when both are evaluated under strict, open-world conditions.

Methodology
The study utilizes the UNITE dynamic release (February 19, 2025) to construct three barcode views: ITS1, ITS2, and the contiguous ITS-core (ITS1-5.8S-ITS2).

Data Partitioning: A genus-separated split was created with sealed calibration and test partitions. The training set (TRAIN_REF) contained 46,453 records; the test set included known and novel genera.
Learned Model (M4): A convolutional encoder (597k parameters) was trained using a "ladder" approach, adding objectives incrementally: genus-proxy classification, cross-view invariance, hierarchical taxonomic geometry, and leave-one-genus family episodes. The model was frozen (SHA-256 sealed) before test data were opened.
Baseline: A VSEARCH-based alignment baseline using percent identity. Crucially, the authors compared the learned model against two configurations: (1) default VSEARCH heuristics and (2) an exhaustive search with a query coverage filter (≥0.8).
Evaluation Protocol: The study employed a "firewall" design where the model was frozen before test data were accessed. Corrections were applied post-hoc (e.g., changing from padded to exact-length inference) and logged in a version-controlled deviation log.
Metrics: Performance was measured via AUROC, known-genus nearest-neighbor accuracy, novel-genus placement (family/order/class), and conformal novelty detection rates at specific false-positive thresholds ( $\alpha$ ). Uncertainty was computed using genus-clustered bootstrapping to account for query dependence.

Key Contributions
The paper's primary contribution is an evaluation protocol that exposes how specific methodological choices can alter the conclusion of a comparison between learned embeddings and alignment baselines. The authors demonstrate that:

Baseline Configuration Matters: Default VSEARCH heuristics often return lower-identity hits than exhaustive search (48.3% of queries) and, without a coverage filter, allow partial alignments over conserved regions (5.8S) to incorrectly classify novel ITS-core queries.
Batch Invariance is Critical: The learned model's embeddings were not invariant to inference batching; padded batches introduced artifacts that inflated cross-view performance. Correcting to exact-length inference significantly reduced the model's reported accuracy.
Evaluation Metrics are Deceptive: AUROC alone failed to detect that the alignment baseline was misconfigured (it remained stable while placement accuracy collapsed without coverage filters) and failed to reveal that the learned model was dominated at deployable operating points.
Leakage Audits are Necessary: A historical benchmark (Leave-One-Genus-Out) was found to be contaminated, with 85% of held-out genera present in the original training set. A leakage-safe retraining was required for a fair comparison.

Results
Under the corrected, rigorous evaluation:

Alignment Superiority: The correctly configured alignment baseline (exhaustive search + coverage filter) matched or exceeded the learned encoder in every comparison.
- Known-Genus Accuracy: Identity achieved 80.8–83.4% across views, compared to 61.6–71.2% for the corrected encoder.
- Novel-Genus Placement: Identity placed novel genera in the correct family at 80.0–84.6%, significantly outperforming the encoder (53.8–64.8%).
- Cross-View Retrieval: Alignment using ITS-core references outperformed the encoder in retrieving ITS1/ITS2 queries (e.g., 79.8% vs 32.9% known-genus accuracy for ITS1 queries). The encoder's cross-view performance was near chance for direct ITS1-to-ITS2 retrieval.
Novelty Detection: While the encoder's novelty AUROC was comparable to identity (within 0.023), identity detected more novel genera at lower false-novelty rates at every deployable operating point ( $\alpha = 0.05, 0.10$ ).
Development-to-Test Gap: The gap between development and test performance for the encoder was smaller than that of the parameter-free identity baseline, indicating that the encoder's performance drop was due to partition composition rather than selection optimism.
Historical Benchmark: On a leakage-safe version of the historical benchmark, identity outperformed the retrained encoder with a statistically significant AUROC advantage of +0.065.

Significance and Claims
The paper concludes that a purpose-trained fungal ITS embedding did not outperform a correctly configured alignment baseline on any comparison. The authors assert that the decisive factors were not the representation (embedding vs. alignment) but the evaluation choices:

Baseline configuration (exhaustive search, coverage filtering).
Batch invariance in inference.
Computing uncertainty at the genus level.
Using parameter-free controls for selection optimism.
Auditing benchmarks for data leakage.

The authors claim that these checks are general, inexpensive, and should be applied to any learned barcode method. They argue that for homologous queries, alignment compares sequences at full nucleotide resolution, whereas embeddings compress this information and struggle to learn non-homologous cross-view structures without relying on the very regions alignment can access directly. The study suggests that learned methods do not automatically dominate distance baselines and that the "open-world" advantage of embeddings remains unproven under strict evaluation.

Technical Summary: "A learned fungal ITS embedding does not outperform correctly configured alignment in open-world evaluation"

More like this