An Automatic Text Classification Method Based on Hierarchical Taxonomies, Neural Networks and Document Embedding: The NETHIC Tool

This paper presents NETHIC, an automatic text classification tool that combines scalable neural networks with hierarchical taxonomies and document embeddings to achieve significant improvements in both effectiveness and efficiency across generic and domain-specific corpora.

The Gist

Imagine you have a massive, chaotic library where millions of books are thrown onto the floor in a giant pile. Your job is to sort them all into the correct shelves. But this isn't just any library; it's a library where the shelves are arranged in a giant, branching tree. At the very top, you have broad categories like "Science" or "Sports." As you go down the branches, the shelves get more specific, like "Physics" $\rightarrow$ "Quantum Mechanics" or "Sports" $\rightarrow$ "Team Sports" $\rightarrow$ "Soccer."

This is the problem the NETHIC tool solves. It's a smart computer program designed to automatically read a piece of text and figure out exactly which "shelf" (or category) it belongs to in this giant tree.

Here is how the paper explains the evolution of this tool, using simple analogies:

1. The Old Way: The "Keyword Detective" (Bag-of-Words)

In the original version of NETHIC, the computer acted like a detective who only looked for specific keywords.

• How it worked: If the text said "goal," "kick," and "field," the computer thought, "Ah, this is about Soccer!"
• The Problem: This is like trying to guess a movie by only counting how many times the word "love" appears. It misses the context. If a text talks about "banking" in a financial sense vs. "banking" a river, a simple keyword counter gets confused. Also, if the library is huge, looking for every single word makes the computer slow and overwhelmed.

2. The New Upgrade: The "Gist Reader" (Document Embedding)

The authors added a new superpower to NETHIC called Doc2Vec (Document Vector).

• The Analogy: Imagine instead of just counting words, the computer reads the whole paragraph and writes a "secret code" (a vector) that captures the meaning or the "vibe" of the text.
• The Result: Now, the computer understands that "I ate a delicious apple" and "The fruit was sweet" are very similar, even if they don't share many of the exact same words. It understands the concept, not just the spelling.

3. The Smart Strategy: The "Specialized Team" (Hierarchical Taxonomy)

This is the most clever part of NETHIC. Instead of hiring one giant, overworked librarian to sort everything at once, NETHIC hires a team of specialized librarians arranged in a hierarchy.

• Level 1 (The Generalist): The first librarian looks at the text and asks, "Is this about Science or Sports?" They don't need to know the difference between a "Tennis racket" and a "Golf club" yet; they just need to know it's sports.
• Level 2 (The Specialist): Once the text is passed down to the "Sports" branch, a new, more specialized librarian takes over. They ask, "Is this Team Sports or Individual Sports?"
• Level 3 (The Expert): Finally, the text reaches the bottom branch, where an expert says, "This is definitely Soccer."

Why is this better?
If you tried to teach one librarian to know the difference between "Quantum Physics," "Soccer," and "Cooking" all at once, they would get confused. By breaking the problem down into small steps, NETHIC reduces "noise" and makes fewer mistakes.

4. The Experiment: Mixing the Best of Both Worlds

The researchers tested three versions of their tool:

1. The Keyword Detective (Old NETHIC): Good, but sometimes confused by context.
2. The Gist Reader (New Doc2Vec only): Good at understanding meaning, but sometimes missed specific details because it ignored the exact words.
3. The Hybrid (NETHIC-2): They combined the two! They gave the computer the "secret code" of the meaning plus the list of specific keywords.

The Result:
The Hybrid version was the champion. It was like giving the librarian both a map of the city (the meaning) and the street names (the keywords).

• Performance: It correctly sorted about 60 more documents out of a test batch than the old version.
• Real-world Example:
  • Text about a mineral called "Bukovskyite": The old tool might have struggled. The new tool correctly identified it as both "Geology" (because of the mineral description) and "Iron and Steel Industry" (because of the mining context).
  • Text about "Overeaters Anonymous": The new tool realized this wasn't just about "Food" (eating), but also about "Addiction" (health/fitness), showing it understood the deeper emotional context of the text.

The Bottom Line

The paper shows that by combining old-school keyword counting with modern AI that understands meaning, and organizing the sorting process into a step-by-step hierarchy, we can build a much smarter, faster, and more accurate system for organizing the world's information. It's not just about finding the right word; it's about understanding the story behind the words.

Technical Summary

1. Problem Statement

The paper addresses the challenge of managing the deluge of unstructured textual data generated by social networks and online platforms. Specifically, it focuses on automatic text classification, a task critical for organizing information but complicated by:

• Semantic Ambiguity: Words often have multiple meanings depending on context (e.g., "tree" as a plant vs. a data structure).
• Hierarchical Complexity: Classifying documents into a large, multi-level taxonomy (e.g., 117 leaf categories) using a single flat classifier often leads to error propagation and confusion between semantically similar but distinct categories.
• Data Sparsity and Noise: Traditional methods like Bag-of-Words (BOW) ignore word order and semantic relationships, while pure deep learning approaches can struggle with high dimensionality and lack of interpretability in hierarchical structures.

2. Methodology

The authors propose NETHIC, a software tool that integrates Hierarchical Taxonomies, Artificial Neural Networks (ANNs), and Document Embedding (Doc2Vec).

A. Core Architecture

• Hierarchical Taxonomy: The system uses a tree-structured taxonomy (based on RDF and SKOS standards) derived from Wikipedia categories. It features a root node, intermediate nodes, and 117 leaf nodes.
• Neural Network Hierarchy: Instead of a single monolithic classifier, NETHIC employs a hierarchy of neural networks.
  • One ANN is trained for each non-leaf concept in the taxonomy.
  • Top-level networks handle broad concepts (e.g., Science, Economy) using generic vocabularies.
  • Deep-level networks handle specific sub-topics (e.g., Geology, Iron and Steel) using specialized, granular vocabularies.
  • This approach reduces noise by restricting the vocabulary of each network to its specific conceptual domain, preventing semantic errors caused by polysemous words.

B. Data Processing & Embedding

• Dataset: The system was trained on a corpus of 57,304 Wikipedia documents, mapped to the taxonomy via the Wikipedia Category Graph.
• Feature Extraction Strategies: The paper compares three feature extraction approaches:
  1. BOW (Bag-of-Words): Traditional frequency-based vectors.
  2. Doc2Vec: A document embedding model (based on Google's Doc2Vec) that converts entire documents into 300-dimensional semantic vectors.
  3. Hybrid (BOW + Doc2Vec): A concatenation of BOW vectors and Doc2Vec vectors to leverage both keyword frequency and semantic context.

C. Classification Algorithm

1. Input: An unstructured text document.
2. Vectorization: The text is converted into vectors using the selected strategy (BOW, Doc2Vec, or Hybrid).
3. Top-Down Traversal:
   • The root network predicts the most probable top-level category.
   • If the probability exceeds a threshold (initially set at 0.7), the process descends to the child node's specific network.
   • This continues until a leaf node is reached or the probability drops below the threshold.
4. Path Scoring: The final score for a classification path is the average of the scores of all nodes in that path.

3. Key Contributions

• Integration of Doc2Vec into Hierarchical Classification: The primary contribution is the extension of the original NETHIC tool (which relied solely on BOW) by integrating Doc2Vec. This allows the system to capture semantic relationships between words and documents that BOW misses.
• Hybrid Feature Model: The authors demonstrate that combining BOW (lexical frequency) with Doc2Vec (semantic meaning) yields superior results compared to using either method in isolation.
• Refined Hierarchical Training Strategy: The paper details a rigorous training process using Cross-Validation (k-fold) and Leave-One-Out (LOO) techniques to prevent overfitting and underfitting across the 117 leaf categories.
• Scalable Architecture: The design allows for modular training of individual networks, making the system scalable and adaptable to specific domains without retraining the entire model.

4. Experimental Results

The authors conducted experiments comparing the original NETHIC (BOW only) against the extended version (NETHIC-2, BOW + Doc2Vec) using a test corpus of 2,843 documents.

• Performance Metrics:
  • Accuracy Improvement: The hybrid model (BOW + Doc2Vec) achieved an overall accuracy improvement of approximately 2% compared to the BOW-only model.
  • Root Category: A 1% improvement was observed at the root level, which is critical for the stability of the entire classification path.
  • Leaf Categories: Most leaf categories saw a 2% improvement.
• Model Comparison:
  • Doc2Vec Only: Performed the worst, indicating that semantic embedding alone is insufficient for this specific complex classification task without lexical frequency data.
  • BOW Only: Solid performance but prone to confusion between semantically close categories.
  • BOW + Doc2Vec: Achieved the highest Precision, Recall, and F1-scores.
• Confusion Matrix Analysis: The hybrid model significantly reduced misclassifications in ambiguous categories such as "Art and Entertainment" (less confused with "Society"), "Health and Fitness" (less confused with "Food and Drink"), and "Science" (specifically Geology).
• Case Studies:
  • Bukovskyite (Mineral): Correctly classified into both "Business/Industrial" and "Science/Geology," demonstrating the system's ability to handle multi-faceted documents.
  • Overeaters Anonymous: Correctly identified as "Health and Fitness/Addiction" (Score 0.64) over "Food and Drink/Healthy Eating" (Score 0.38), showing the model's ability to distinguish between the topic (food) and the context (addiction/health).

5. Significance and Future Work

• Significance: The paper validates that combining traditional statistical methods (BOW) with modern deep learning embeddings (Doc2Vec) within a hierarchical framework creates a robust, scalable, and highly accurate text classification system. It solves the "semantic gap" in hierarchical classification without sacrificing the interpretability of the taxonomy.
• Future Directions: The authors suggest exploring the integration of more advanced state-of-the-art models like BERT (Bidirectional Encoder Representations from Transformers) to further enhance semantic understanding and classification accuracy.

In conclusion, NETHIC represents a significant step forward in automated text classification by effectively bridging the gap between structured hierarchical knowledge and unstructured semantic data representation.