Technical Summary: Discovering Multi-Scale Semantic Structure in Text Corpora Using Density-Based Trees and LLM Embeddings

1. Problem Statement

The semantic organization of large text corpora remains a central challenge in information retrieval and knowledge representation. While Large Language Models (LLMs) have revolutionized text representation through high-fidelity dense embeddings, current systems largely rely on flat clustering (e.g., k-means, DBSCAN) or predefined taxonomies. These approaches fail to capture the hierarchical nature of semantic relationships, obscuring how fine-grained concepts merge into broader thematic domains. Furthermore, traditional hierarchical methods often depend on strong modeling assumptions, global distance matrices sensitive to noise, or fixed ontologies that do not adapt well to heterogeneous, evolving text collections. There is a need for a data-driven approach that can reconstruct semantic hierarchies directly from embedding spaces without imposing a fixed resolution or predefined structure.

2. Methodology

2.1 Data and Embeddings

The study utilizes a diverse set of English-language text corpora, including standard benchmarks (20 Newsgroups, IMDB 50K Reviews, AG News) and large institutional scientific corpora (American University of Beirut [AUB] and TU Wien abstracts, plus French doctoral theses).

• Embedding Generation: Texts are transformed into 4096-dimensional vectors using high-performance models (Qwen3-Embedding-8B or SFR-Embedding-Mistral) selected based on MTEB-Leaderboard performance.
• Dimensionality Reduction: Principal Component Analysis (PCA) is applied to reduce dimensionality. The authors found that using L2-norm distances on a reduced set of principal components (e.g., 2–10 components) yields significantly more interpretable and structurally distinct trees compared to using cosine distance on full 4096-dimensional vectors.

2.2 Hierarchical Density-Based Tree Construction

The core innovation is an adaptive density clustering procedure that operationalizes hierarchical structure through the progressive relaxation of local density constraints.

• Algorithm: The method adapts DBSCAN but iterates through successive density thresholds rather than outputting a single flat clustering.
• Process:
  1. Clusters are defined at iteration $L$ where local density $\rho_L > \frac{minpts}{V(\epsilon_L)}$.
  2. The neighborhood radius $\epsilon$ is increased in fixed increments ($\Delta\epsilon$), effectively lowering the density threshold.
  3. As $\epsilon$ increases, previously dense clusters merge into larger, more diffuse clusters.
  4. This process generates a tree where terminal branches represent tightly related (dense) semantic groups, and higher-level nodes represent increasingly general thematic regions.
• Outcome: The resulting tree encodes multi-scale semantic organization directly from data, revealing how compact groups merge into broader regions without assuming a fixed taxonomy.

2.3 Automated Semantic Annotation

To ensure interpretability, the method integrates LLMs as semantic annotators:

• Context: The Llama-4-Scout-17B-16E-Instruct model (quantized) is used to process large contexts (up to ~600k tokens).
• Workflow: For each tree node, the LLM receives the textual content of all abstracts in that cluster and is prompted to infer a unifying research field label.
• Handling Large Clusters: If a cluster exceeds the token limit, abstracts are sampled (with oversampling for stability), processed in chunks, and the resulting candidate labels are summarized by a secondary model (gpt-oss-20b) to produce a single overarching label.
• Visualization: Trees are visualized using color-coded maps (via MNHN-Tree-Tools and newick-utilities) and an interactive tile-based viewer to allow detailed exploration of node-level labels.

2.4 Evaluation

For benchmark datasets with ground-truth labels, the authors evaluate alignment using the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) computed per tree layer. This allows for the assessment of how semantic agreement evolves across different resolutions (depths) of the hierarchy.

3. Key Results

3.1 Impact of Dimensionality Reduction

Experiments demonstrated that trees constructed using cosine distance on full embeddings exhibited poor structural resolution. In contrast, applying PCA followed by L2-norm distance clustering produced trees with clearer branching and detectable fine-scale substructures. The authors attribute this to PCA's ability to prioritize variance and suppress noise.

3.2 Benchmark Performance

• 20 Newsgroups: The inferred tree successfully recovered intuitive relationships, such as the proximity of sports-related groups and the merging of religion and politics. ARI and NMI peaked at intermediate tree depths, indicating that the strongest alignment with ground-truth labels occurs at a resolution where fine-grained lexical noise is absorbed but broad thematic distinctions remain.
• AG News: World and Sports categories formed distinct, separable subtrees. Business and Science/Technology categories showed substantial overlap, reflecting their topical similarity in headlines.
• IMDB 50K Reviews: The tree was shallow and homogeneous, with low ARI/NMI scores across all depths. This indicates that the embedding models used encode sentiment only weakly relative to other semantic dimensions, and sentiment-based clusters do not persist as stable large-scale structures.
• French Theses (Theses-fr-en): Despite highly heterogeneous author-supplied labels, the tree clearly separated Humanities from Natural Sciences. Further analysis using OECD Fields of Science (FOS) classifications revealed that Engineering and Medical sciences merged earlier in the hierarchy than Social Sciences and Humanities, a structural relationship not explicitly defined in the standard FOS taxonomy.

3.3 Institutional Insights

Applied to AUB and TU Wien corpora, the method revealed institution-specific research landscapes without relying on predefined categories:

• AUB: Dominated by a cohesive medical/health subtree, with a distinct humanities branch (Middle Eastern Studies) that merged structurally closer to engineering than to medicine, suggesting unique interdisciplinary connections.
• TU Wien: Revealed a homogeneous but internally structured tree highlighting strengths in quantum physics, materials science, and information technology, with foundational mathematics and theoretical physics forming early distinct branches.

4. Key Contributions

1. Operationalization of Density-Based Hierarchies: The paper introduces a method to construct semantic trees by progressively relaxing density constraints on LLM embeddings, a novel application of density-based clustering to text corpora.
2. Multi-Scale Representation: The approach provides a single, unified representation that captures both local semantic coherence and global thematic organization, accommodating documents that do not belong to dense groups at fine resolutions.
3. Data-Driven Taxonomy Discovery: The method infers hierarchical relationships unsupervised, making it suitable for exploratory analysis of heterogeneous corpora where existing classifications are incomplete or non-existent.
4. Integrated Annotation Pipeline: The work combines unsupervised structure discovery with automated, large-context LLM annotation to generate human-readable labels for tree nodes.
5. Empirical Validation: The study demonstrates that semantic alignment peaks at intermediate density levels and that abrupt transitions in the tree correspond to meaningful changes in semantic resolution.

5. Significance and Claims

The authors claim that framing hierarchical structure as an emergent property of density in embedding spaces provides an interpretable, multi-scale representation suitable for large, evolving text collections. The method offers a flexible alternative to fixed taxonomies, capable of revealing:

• Dominant fields and cross-disciplinary proximities within institutional research.
• Emerging thematic clusters that may not be captured by existing classification systems.
• Structural ambiguities in standard taxonomies (e.g., the relationship between Engineering, Medicine, and Natural Sciences).

The paper concludes that while the method is constrained by the representational capacity of the underlying embedding models (e.g., inability to capture weak signals like sentiment in IMDB), it provides a powerful framework for mapping semantic structure and exploring complex textual knowledge spaces. The authors modestly note limitations in annotation stability for very large clusters and suggest future work could involve domain-specific embeddings or application to other modalities (images, audio).