From Topic to Transition Structure: Unsupervised Concept Discovery at Corpus Scale via Predictive Associative Memory

1. Problem Statement

Current text embedding models (e.g., BGE, BERT) primarily organize text based on semantic content (what the text is about). While effective for topic modeling, they fail to capture transition structure (what the text does in a narrative context).

• The Gap: Two passages with vastly different topics (e.g., a tropical horror scene vs. a Victorian domestic scene) may perform the exact same narrative function (e.g., "the moment of confrontation"). Standard similarity-based models group them by topic, missing their structural kinship.
• The Challenge: How to discover unsupervised, cross-author, cross-genre concepts that represent narrative functions, discourse registers, and scene templates without relying on topic annotations or manual labeling.

2. Methodology

The authors propose a method that shifts the training signal from semantic similarity to temporal co-occurrence within texts, utilizing a capacity-constrained contrastive learning framework.

A. Corpus and Preprocessing

• Dataset: 9,766 English texts from Project Gutenberg (spanning the 16th–20th centuries), comprising ~24.96 million passages.
• Chunking: Texts are split into 50-token passages with 15-token overlap to capture single narrative beats while ensuring continuity.
• Embedding: Passages are initially embedded using BGE-large-en-v1.5 (1024 dimensions).

B. Training Signal: Temporal Co-occurrence

• Pairs: For each text, pairs of passages are extracted if they appear within a 15-chunk window (approx. 525 tokens).
• Rationale: Passages that reliably co-occur locally share a "transition structure" (similar predecessors and successors), regardless of their vocabulary or genre.
• Scale: The model is trained on 373 million unique co-occurrence pairs.

C. Model Architecture: Association Model

• Architecture: A 4-layer MLP with GELU activations, LayerNorm, and a learned residual connection.
  • $f(x) = \text{normalise}(\alpha \cdot x + (1-\alpha) \cdot g(x))$
  • The residual connection preserves original semantic information while the MLP learns an "associative perturbation."
• Parameters: 29.4 million parameters.
• Loss Function: Symmetric contrastive loss (InfoNCE) with in-batch negatives (batch size 512).
• Capacity Constraint: The model is intentionally under-parameterized relative to the data volume. It achieves only 42.75% training accuracy (top-1 retrieval within a batch). This bottleneck forces the model to compress individual co-occurrences into generalizable structural patterns rather than memorizing specific pairs.

D. Clustering and Analysis

• Multi-Resolution Clustering: K-means clustering is applied to the transformed "association space" embeddings at six granularities ($k=50$ to $k=2,000$).
• Validation:
  • Baselines: Compared against raw BGE clustering (semantic similarity) and a context-enriched baseline (averaging local embeddings).
  • Controls: Verified against position-in-book bias, token count, and book concentration. A temporal shuffle control collapsed recall by 95.2%, confirming the signal is genuine co-occurrence structure.
  • Inductive Transfer: Five canonical novels absent from training (e.g., Dracula, Pride and Prejudice) were assigned to existing clusters without retraining.

3. Key Contributions

1. Unsupervised Concept Discovery: A method to extract "transition-structure concepts" (narrative functions, registers, traditions) purely from temporal co-occurrence statistics, requiring no labels or metadata.
2. Association $\neq$ Similarity: Empirical proof that organizing text by what follows what yields qualitatively different clusters than organizing by semantic similarity.
3. Multi-Resolution Concept Map: A hierarchical map of literary structure ranging from broad modes (e.g., "direct confrontation") to specific registers (e.g., "Franco-Prussian War dispatches," "sailor dialect").
4. Extension of PAM Framework: Demonstrates that the Predictive Associative Memory (PAM) principle, previously limited to episodic recall, can achieve inductive transfer and concept formation when applied under severe compression regimes.

4. Key Results

A. Structural vs. Semantic Grouping

• BGE (Semantic): Groups passages by topic (e.g., all "fear" passages together, all "money" passages together).
• PAM (Association): Groups passages by function.
  • Example: A "moment of departure" in a Jane Austen novel, a Western, and a Gothic horror are clustered together because they share the same narrative beat (building tension $\to$ leaving), despite having zero vocabulary overlap.
  • Example: "Cynical worldly wisdom" clusters passages from 18th-century satire, Victorian social commentary, and philosophical essays based on their discourse register.

B. Multi-Resolution Hierarchy

• Coarse ($k=50$): Captures broad narrative modes (e.g., "historical combat," "lyrical meditation").
• Fine ($k=2,000$): Resolves into specific templates and traditions (e.g., "courtroom cross-examination," "Darwin-Huxley correspondence," "witchcraft trials").
• Cross-Genre: Clusters consistently draw from hundreds of books across different centuries and genres, proving the patterns are not author-specific artifacts.

C. Inductive Transfer (Unseen Novels)

• Selectivity: The association model concentrates unseen novels into a selective subset of clusters (e.g., Alice in Wonderland maps to only 51/100 clusters, with 77.6% of text in the top 5).
• Contrast: Raw BGE assignment saturates nearly all clusters, failing to distinguish structural consistency from topic diversity.
• Structural Tracking: PAM assignments track structural shifts (e.g., a change from travel observation to supernatural dread in Dracula) even when vocabulary remains similar, whereas BGE shifts with every new object mentioned.

D. Compression Hypothesis

• The model's low training accuracy (42.75%) is a feature, not a bug. The capacity bottleneck forces the model to discard specific episodic details and retain only the recurrent structural patterns that generalize across the corpus. This mimics biological "neocortical consolidation."

5. Significance and Implications

• New Lens for Literary Analysis: Provides a data-driven way to analyze "narrative pacing signatures" and structural universals across literature, revealing how different authors deploy specific structural modes (e.g., how Ulysses shifts modes vs. War and Peace).
• Beyond Text: The mechanism suggests a general framework for unsupervised concept formation in any sequential data (legal cases, gene expression, user behavior) where temporal co-occurrence under compression reveals underlying structural regularities.
• Theoretical Shift: Challenges the dominance of semantic similarity in NLP, proposing that transition structure is a distinct and vital dimension of text organization that enables robust inductive transfer where semantic models fail.

6. Limitations

• Single Training Run: Results are from a single run; multi-seed evaluation is pending.
• Corpus Bias: Relies on Project Gutenberg (19th/early 20th-century English literature); generalizability to modern or non-English texts is untested.
• No Downstream Task: Unlike previous work (AAR), this study does not evaluate the concepts on specific downstream tasks (e.g., QA), focusing instead on the intrinsic quality of the discovered structure.
• Post-hoc Labeling: Cluster labels are generated by LLMs based on sample passages, not learned by the model itself.

Conclusion

The paper successfully demonstrates that training a contrastive model on temporal co-occurrence under capacity constraints extracts a "concept map" of text based on what text does (transition structure) rather than what it says (semantic content). This approach yields transferable, cross-genre structural concepts that outperform traditional semantic embeddings in capturing narrative function and discourse register.