🤖 machine learning

Criticality in Dissimilar Decomposition and Undersampling of Random Datasets with Anomalies

This paper investigates how AI-generated anomalies affect random datasets by using redundancy graphs to demonstrate a phase transition in the minimum size of strongly dissimilar decompositions, where the dataset's structure shifts from being determined by main data points to being dominated by anomalies once a critical threshold is reached.

Original authors: Ghurumuruhan Ganesan

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ghurumuruhan Ganesan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of modern artificial intelligence, the quality of a model's intelligence is inextricably tied to the quality of the data used to teach it. Imagine a student trying to learn a subject; if their textbooks are filled with errors, contradictions, or repetitive nonsense, their understanding will be flawed. Today, as we train massive language models to write, reason, and create, these systems are consuming vast oceans of text and images. A growing concern for researchers is that the internet is becoming saturated with content generated by artificial intelligence itself. This creates a feedback loop where future models are trained on data produced by previous models. The core challenge is to understand how this influx of "synthetic" or AI-generated data, which acts as a distinct type of anomaly, disrupts the way we organize and process information. Specifically, scientists want to know how to break these massive datasets into smaller, manageable groups for training, ensuring that each group contains enough variety to teach the model something new, without being overwhelmed by the strange patterns introduced by the synthetic data.

Ghurumuruhan Ganesan, a researcher at the University of Bristol, tackled this problem by treating the dataset as a collection of points that can be either similar or different, and either linked or unlinked. In this context, "similarity" refers to how much two pieces of data look alike, while "linkage" describes a hidden connection between them, often based on their source. For instance, a piece of AI-generated text might look very different from a human-written one, but it is "linked" to the human data because it was created using the same underlying model. The researcher investigated how to split a mixed dataset of human and AI data into groups where every item is distinct from the others and not linked to them. The goal was to find the smallest possible number of groups needed to achieve this separation, a metric that determines the efficiency of the training process.

The study reveals a surprising and sharp shift in how these datasets behave as the amount of AI-generated content increases. When the number of synthetic anomalies is small, the difficulty of organizing the data is determined almost entirely by the original, human-generated points. The system behaves as if the anomalies are barely there, and the number of groups required remains stable. However, the research identifies a specific threshold where this dynamic flips. Once the number of anomalies crosses this line, the task of organizing the data changes fundamentally. Even if the synthetic data still makes up a tiny fraction of the total collection, it suddenly becomes the dominant factor. The minimum number of groups required to keep the data distinct is no longer set by the human data but is instead dictated by the anomalies. This suggests that a small amount of synthetic data, if it is highly connected to the rest of the dataset, can force a complete restructuring of how the data must be handled, potentially making training much less efficient than anticipated.

To reach these conclusions, the author used a method that visualizes the data as a network of points connected by lines. If two data points are too similar or too closely linked, a line connects them. The problem then becomes one of grouping these points so that no two points in the same group share a line. The researcher applied rigorous mathematical techniques to estimate the size of these groups under different conditions. The results show that for datasets where the total space of possible data is much larger than the dataset itself—a common scenario with categorical data like words or image tags—the transition from "human-dominated" to "anomaly-dominated" organization is abrupt. The study provides precise boundaries for when this switch happens, offering a way to predict when a dataset has become too contaminated with synthetic links to be processed efficiently without special handling.

The paper also explored what happens when researchers randomly select a smaller subset of the data for training, a common practice known as undersampling. The findings indicate that if the sample size is kept below a certain critical limit, the randomly chosen data will almost certainly remain distinct and unlinked, preserving the integrity of the training batch. However, if the sample size grows beyond this limit, the probability of accidentally including linked or similar points skyrockets, causing the batch to lose its diversity. This critical size depends heavily on the number of anomalies present; even a few anomalies can lower the threshold at which the data becomes "messy."

Ultimately, this work provides a theoretical framework for understanding the fragility of data organization in the age of AI-generated content. It demonstrates that the presence of anomalies is not just a matter of volume but of connectivity. A small number of highly linked synthetic points can exert a disproportionate influence on the entire dataset, forcing a phase transition in how the data must be decomposed. For those building the next generation of artificial intelligence, these findings serve as a cautionary note: the mere presence of AI-generated data, even in small quantities, can fundamentally alter the mathematical landscape of training, requiring new strategies to ensure models learn effectively from a world increasingly populated by their own kind.

Technical Summary: Criticality in Dissimilar Decomposition and Undersampling of Random Datasets with Anomalies

Problem Statement
The paper addresses the challenge of decomposing and undersampling random datasets that contain "anomalies" deeply linked to the primary data points. This scenario is motivated by the increasing prevalence of training datasets for Large Language Models (LLMs) that include significant amounts of AI-generated text or image data. These AI-generated samples are treated as anomalies because they are "linked" to the true data points via their source (e.g., generated from previous LLMs trained on similar data) rather than being independent. The core problem is to understand how these linked anomalies affect the minimum size of a "strongly dissimilar" (SD) batch decomposition and the critical size of a randomly undersampled dataset required to maintain strong dissimilarity.

Methodology
The author employs a probabilistic framework combining graph theory and random variable analysis:

Modeling: The dataset is modeled as a collection of $n$ independent data points, where $n-v_0$ are "main" points (i.i.d. with distribution $p$ ) and $v_0$ are "anomalous" points (with distinct distributions).
Definitions:
- Similarity: Defined via a function $\rho(x, y)$ (e.g., feature agreement). Points are $\lambda$ -dissimilar if $\rho \le \lambda$ .
- Linkage: Modeled using a Binomial random graph where edges represent a "link" between data points based on their source indices, independent of their actual values.
- Strong Dissimilarity (SD): Two points are SD if they are $\lambda$ -dissimilar and delinked (no edge between them in the linkage graph).
Graph Construction: A "similarity graph" $\Gamma$ is constructed where an edge exists between two vertices if the corresponding data points are "strongly similar" (i.e., similar and linked).
Analytical Tools:
- Decomposition: The minimum size of an SD decomposition ( $\tau_{SD}$ ) is analyzed using bounds related to the maximum vertex degree ( $\Delta$ ) and maximum average degree ( $h_{av}$ ) of the similarity graph. The paper utilizes iterative stable set extraction techniques.
- Undersampling: The properties of a randomly undersampled dataset (RUS) are studied using first and second moment methods (expectation and variance estimates) to determine the probability that the sampled subset contains no edges in the similarity graph.
- Concentration Inequalities: Chernoff-type bounds and Chebyshev's inequality are used to establish high-probability results for the existence of edges and the size of stable sets.

Key Contributions and Results

Phase Transition in Decomposition Size (Theorem 1):
The paper establishes a phase transition regarding the minimum size of a strongly dissimilar decomposition ( $\tau_{SD}$ ) as the number of anomalies ( $v_0$ ) varies:
- Low Anomaly Regime: When $v_0$ is small (specifically $v_0 = o(v_{low})$ ), the minimum decomposition size is primarily determined by the "main" data points. The size scales roughly with $n \zeta_{low} q_{low} / \log n$ .
- High Anomaly Regime: When $v_0$ exceeds a certain threshold (even if $v_0 \ll n$ ), the minimum size becomes crucially dependent on the anomalies. In this regime, the size scales with $v_0 p_{low} s_{low} / \log n$ .
- This demonstrates that a small number of highly linked anomalies can dominate the structural properties of the dataset decomposition.
Critical Size for Random Undersampling (Theorem 2):
The paper determines the critical sample size $t$ for a randomly undersampled dataset to remain strongly dissimilar with high probability:
- Sub-critical Regime: If the sample size $t$ is sufficiently small ( $t = o(t_{low})$ ), the undersampled dataset is strongly dissimilar with probability approaching 1.
- Super-critical Regime: If $t$ exceeds a specific threshold ( $t \gg t_{up}$ ), the probability that the dataset contains at least one pair of strongly similar points (violating SD) approaches 1.
- The critical thresholds depend on the extremal similarity and linkage probabilities of both main and anomalous data points.
Illustrative Examples:
The results are illustrated using categorical datasets where the overall space size $L$ is much larger than the dataset size $n$ . The paper derives specific bounds for these cases, showing how the interplay between the number of features, the similarity threshold, and the linkage probability dictates the criticality.

Significance and Claims
The paper claims to provide a theoretical foundation for understanding the impact of AI-generated data (anomalies) on dataset processing. Specifically:

It highlights that anomalies linked to main data points can induce a "singularity" or "skynet threshold" (Remark 2), where the dataset's decomposability is taken over by the anomalies even if their count is negligible compared to the total dataset size.
It offers rigorous bounds for the minimum batch size required for training models to ensure balanced learning (dissimilar batches) in the presence of such anomalies.
It provides critical size estimates for random undersampling, ensuring that reduced datasets retain the property of strong dissimilarity, which is vital for efficient model training.

The work relies on standard coloring techniques and graph-theoretic bounds but presents them in a self-contained manner without explicit coloring terminology. The results are asymptotic, holding with high probability ( $1 - o(1)$ ) as the dataset size $n$ tends to infinity.

More like this