📊 statistics

One of the Above, None of the Above: Why the Same Model Should Not Predict Both

This paper argues that novelty prediction requires explicitly separating category absorption from category opening within a well-specified context, demonstrating that a simple combinatorial null model often outperforms complex rich-get-richer models like the Pitman-Yor process when the task structure is properly accounted for.

Original authors: Lukasz Konowalek

Published 2026-08-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lukasz Konowalek

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, our minds are busy sorting the world into groups. When you hear a new word, see a strange animal, or notice a new type of car, your brain has to decide: is this something I have already seen before, or is it something entirely new? This split-second judgment is the difference between recognizing a familiar face and spotting a stranger. For decades, scientists studying how we learn and categorize have often treated this decision as a single coin flip. They assumed that if an item does not fit into an existing group, it must automatically start a new one, and that the math describing how we stick to old groups is the same math describing how we start new ones.

However, a new study suggests this simple view might be wrong. The research, conducted by Lukasz Konowalek at the Medical University of Warsaw, argues that the brain likely uses two different strategies for these two tasks. One strategy is for recognizing what you already know, and the other is for deciding when to create a new category. The study shows that trying to use a single model to explain both behaviors leads to errors, and that the context in which we learn—whether we are looking at a closed set of options or an open, endless stream of possibilities—changes the rules of the game entirely.

To understand the experiment, imagine a stream of information, like a conversation, a video, or a list of words. As you move through this stream, you are constantly asking yourself if the next item is a repeat of something you just saw or a fresh discovery. The researchers took five different types of real-world data to test this. They looked at how people use words in documents, how they refer to people and objects in stories, how people speak in conversations, and how video editors mark changes in a scene. In some of these streams, the list of possible items is fixed and small, like the limited set of dialogue tags used in a specific software program. In others, the list is open and potentially infinite, like the endless variety of words or names we might encounter.

The researchers built computer models to predict what would happen next in these streams. They tested two main approaches. The first approach, which they call a "combinatorial" model, is a simple counting method. It looks only at how many distinct items have appeared so far and how many total items there are, ignoring how often specific items repeat. It asks a basic question: given the number of things I have seen, how many ways can I organize them? The second approach is a "rich-get-richer" model. This model assumes that if an item has appeared many times before, it is more likely to appear again. It gives extra weight to popular items, assuming that popularity breeds more popularity.

The study found that the answer to which model works best depends entirely on what the computer is trying to predict. When the task is to predict whether an item is a repeat of something already seen (absorption), the simple counting model works best in open, infinite streams. But when the task is to predict whether an item is a brand-new discovery (opening), the "rich-get-richer" model is often better. Surprisingly, this pattern flips when the stream has a small, fixed list of options. In those closed systems, the "rich-get-richer" model is better at spotting repeats, while the simple counting model is better at spotting new items.

This "crossed" pattern is the core discovery. It means that no single mathematical rule can perfectly explain both recognizing the familiar and spotting the new. The same model that is excellent at predicting that a word will be repeated is often terrible at predicting that a new word will appear, and vice versa. The researchers also tested this idea in a controlled experiment where people were asked to categorize items, finding that a model that simply checked if the task allowed for a "new" answer outperformed complex models that tried to guess based on past frequency.

The results suggest that our minds do not treat novelty as a simple leftover from recognition. Instead, the brain likely runs two separate processes. One process is tuned to the structure of the current situation, checking if an item fits into the existing pattern. The other process is tuned to the possibility of the unknown, deciding when the current pattern is no longer enough and a new category must be opened. This distinction is crucial because it changes how we should build artificial intelligence and how we understand human learning. If we want machines to learn like humans, they cannot just be good at spotting patterns; they must also have a separate mechanism for knowing when to break the pattern and start something new. The study concludes that the context of the task—whether the world is closed and finite or open and endless—determines which of these two mental tools we should be using at any given moment.

Technical Summary: "One of the Above, None of the Above: Why the Same Model Should Not Predict Both"

Problem Statement
Novelty detection is frequently modeled as a binary partition-extension problem: determining whether a new observation belongs to an existing category ("one of the above") or necessitates a new category ("none of the above"). Standard approaches, particularly Bayesian species-sampling models like the Chinese Restaurant Process (CRP) and the Pitman–Yor (PY) process, treat "opening" a new category as the probabilistic residual of "absorbing" an item into an existing one.

This paper argues that this formulation obscures a fundamental epistemic asymmetry. Absorption is a judgment of whether an item can be resolved within an already generated partition, whereas opening is a judgment of whether the active context licenses a new response identity. Treating these as two sides of a single generative process risks collapsing distinct dependencies into a single probability. Furthermore, standard models often fail to account for the specific "context" (local stream unit, equivalence relations, and task-protocol inventory size) that makes novelty judgments meaningful.

Methodology
The study operationalizes novelty prediction using a dependency-logic-inspired framework where prediction rows are defined by a "dependency atom" $a_t = (C_u, H_{<t}, r_t, z_t)$ .

Context ( $C_u$ ): Includes the local stream unit ( $u$ ), the equivalence relation ( $R_u$ ) defining response identity, and the task-protocol inventory size ( $\kappa_{task}$ ).
History ( $H_{<t}$ ): The prefix of earlier episodes within the same local unit.
Target ( $z_t$ ): A binary label where $z_t=1$ indicates "opening" (a new identity) and $z_t=0$ indicates "absorption" (reuse of an existing identity).

The paper compares two primary predictors on held-out data across five naturalistic datasets (GUM coreference, GUM lexical recurrence, SWDA lexical recurrence, SWDA dialogue acts, and Kinetics-GEBD) and a reanalysis of Navarro and Kemp's (2017) category-learning experiments:

The Combinatorial Null ( $x_{comb}$ ): A block-size-blind score derived from log Stirling numbers ( $\log S(N_t, K_t-1) - \log S(N_t, K_t) - \log K_t$ ). It predicts novelty based solely on the change in the number of admissible partitions, ignoring the occupancy distribution of existing classes (no "rich-get-richer" effect).
The Pitman–Yor Process (PY): A standard species-sampling model that incorporates "rich-get-richer" dynamics, where the probability of joining an existing block is proportional to its current occupancy.

The analysis decomposes performance into absorption ( $z_t=0$ ) and opening ( $z_t=1$ ) regimes and tests whether the "winner" depends on the task-protocol inventory size ( $\kappa_{task}$ ).

Key Contributions and Results

Context is Necessary: Replacing local histories with global corpus histories significantly degrades the performance of both $x_{comb}$ and PY models. This confirms that novelty prediction is not defined by a partition alone but by the local context in which that partition is extended.
The Crossed Regime Pattern: The paper demonstrates that no single model universally outperforms the other across both absorption and opening. Instead, the optimal predictor reverses based on the task-protocol inventory size:
- Open Inventories ( $\kappa_{task} = \infty$ ): In datasets with open vocabularies (e.g., GUM lexical/coreference), the combinatorial null ( $x_{comb}$ ) outperforms PY on absorption, while Pitman–Yor outperforms $x_{comb}$ on opening.
- Closed Inventories ( $\kappa_{task} < \infty$ ): In datasets with fixed, finite inventories, the pattern reverses on absorption: Pitman–Yor outperforms $x_{comb}$ . On opening, the results are mixed: $x_{comb}$ outperforms PY in the Kinetics-GEBD dataset (the smallest finite-inventory case), but the result for SWDA dialogue acts (a larger finite inventory of 42 tags) remains unresolved, as the bootstrap confidence interval includes zero. This suggests SWDA dialogue acts represent a boundary case rather than a clear reversal of the pattern.
Functional Form Nuance: The exact Stirling-based combinatorial score is not merely interchangeable with simpler density surrogates (like $\log(N/K)$ ). In low-count streams (specifically Kinetics), the exact finite-count correction provides a real, measurable advantage over density approximations, suggesting the exact form captures structural properties of the stream beyond simple density.
Reanalysis of Controlled Experiments: In a reanalysis of Navarro and Kemp's (2017) Experiment 4, a structural model incorporating a "task gate" (indicating if "none of the above" is licensed) and partition density, but without rich-get-richer terms, outperformed the HG–CRP benchmark. Furthermore, a decomposition showed that HG–CRP performed better on "one of the above" (absorption) while the gate model performed better on "none of the above" (opening), mirroring the crossed pattern found in naturalistic corpora.

Significance and Claims
The paper argues against the view that category opening is merely the residual of a single absorption process. Instead, it posits that absorption and opening are distinct regimes governed by different dependency structures:

Absorption in open systems is best predicted by the combinatorics of partition extension (ignoring specific occupancies).
Opening in open systems requires "rich-get-richer" information (occupancy/persistence).
In closed systems, the dynamics reverse on absorption, with occupancy information aiding absorption. On opening, the advantage of the combinatorial model is clearest in smaller inventories like Kinetics, while larger finite inventories like SWDA dialogue acts show an unresolved pattern, suggesting a boundary case rather than a definitive reversal.

The author concludes that future models of novelty should explicitly separate these regimes. The combinatorial partitioning score should be treated as a "fixed-context null model." Its failures are not merely errors but are informative about the local stream structure, indicating when identity-specific persistence (rich-get-richer effects) becomes necessary. The results suggest that the "one of the above" vs. "none of the above" distinction should not be treated as a single probability with one generative process behind it, but rather as two distinct operations that may recruit different representational and control processes.

Technical Summary: "One of the Above, None of the Above: Why the Same Model Should Not Predict Both"

More like this