Natural Language Processing Models for Robust Document Categorization

1. Problem Formulation

The study addresses the challenge of automating the categorization and distribution of IT support tickets within a technical support department. The core business problem involves processing unstructured textual data (ticket subjects and bodies) and routing them into one of three specific categories:

• Problem: Issues, failures, or crashes.
• Request: Information requests, assistance, or advice.
• Change: Issues related to recent updates or proposed changes.

Key Challenges:

• Class Imbalance: The dataset is heavily skewed, with "Problem" tickets (7,120) vastly outnumbering "Change" tickets (1,280), while "Request" tickets number 3,479. This imbalance poses a risk of models biasing toward the majority class.
• Semantic Ambiguity: Similar wording can describe different classes (e.g., "tools crashed" vs. "tools stopped working due to network issues"), requiring models capable of deep contextual understanding rather than simple keyword matching.
• Operational Constraints: The solution must balance high classification accuracy with computational efficiency to ensure scalability in a real-world automation pipeline.

2. Methodology

The authors evaluated three distinct machine learning approaches representing varying levels of complexity and computational cost:

A. Data Preprocessing

• Input: Combined "subject" and "body" text fields.
• Cleaning: Converted to lowercase, removed special characters and excess whitespace, and eliminated English stop words.
• Validation: Excluded empty or malformed entries.

B. Models Evaluated

1. Naïve Bayes Classifier:
   • A probabilistic statistical model based on Bayes' theorem.
   • Role: Served as a fast, lightweight baseline.
   • Characteristics: Extremely low training time; assumes feature independence.
2. Bidirectional Long Short-Term Memory (BiLSTM):
   • A deep learning architecture using recurrent neural networks that process sequences in both forward and backward directions.
   • Role: To capture sequential dependencies and bidirectional context without the massive overhead of transformers.
   • Characteristics: Tested with varying hidden layers (16 to 64) and batch sizes.
3. BERT (Bidirectional Encoder Representations from Transformers):
   • A fine-tuned transformer-based Large Language Model (bert-base-uncased).
   • Role: To leverage state-of-the-art contextual understanding.
   • Characteristics: Utilized transfer learning (fine-tuning) with class weighting to mitigate dataset imbalance.

C. Experimental Setup

• Dataset: "Customer IT Support - Ticket Dataset" (Total: ~11,879 tickets).
• Validation: 10-fold cross-validation executed 10 times to ensure robustness.
• Hardware: NVIDIA GeForce RTX 3060 GPU; Python stack (Keras, TensorFlow, scikit-learn).
• Metrics: Accuracy, Precision, Recall, and F1-score.

3. Key Contributions

• Comparative Analysis: The paper provides a rigorous comparison of traditional statistical models, deep learning (RNN/LSTM), and transformer architectures specifically for an imbalanced, real-world IT ticketing scenario.
• System Architecture Design: The authors designed and implemented a fully functional demonstrative system featuring:
  • Modularity: Separation of classification logic from the application layer.
  • Scalability: A stateless, containerized architecture allowing horizontal scaling via orchestration tools to handle varying data volumes.
  • Automation: End-to-end automated routing of technical requests.
• Trade-off Evaluation: The study explicitly quantifies the trade-off between computational cost (training/inference time) and predictive performance, offering practical guidance for system architects.

4. Results

The experiments revealed significant performance differences across the three models:

Model	Avg. Accuracy	Training Time (Approx.)	Key Performance Characteristics
Naïve Bayes	94.23%	0.01 seconds	Fastest inference. High precision (1.0) for the minority "Change" class but very low recall (0.55), indicating it rarely predicts this class unless extremely confident.
BiLSTM	98.56%	~67–320 seconds	Best balance. Achieved high recall (0.95) for the minority "Change" class. Robust contextual understanding with moderate computational cost.
BERT	99.23%	~1,200 seconds	Highest accuracy and F1 scores across all classes. However, training time was ~20 minutes per cycle, making it computationally expensive.

Key Findings on Imbalance:

• All models struggled slightly with the minority "Change" class compared to the majority "Problem" class.
• Naïve Bayes suffered the most from this, exhibiting a "conservative" strategy where it only predicted "Change" when highly certain, leading to missed detections (low recall).
• BiLSTM and BERT, with their ability to learn complex semantic patterns, significantly improved the detection of minority classes.

5. Significance and Conclusion

The paper concludes that while BERT offers the highest theoretical accuracy (>99%), it may be impractical for high-throughput, real-time automation pipelines due to its substantial computational overhead and long training times.

• Optimal Solution: The BiLSTM model is identified as the most balanced solution for the specific scenario. It offers a near-optimal accuracy (~98.56%) with significantly lower resource requirements and faster inference times than BERT, while vastly outperforming Naïve Bayes in handling semantic nuance and class imbalance.
• Practical Implication: For organizations implementing automated document routing, the study suggests that "heavier" models like BERT should be reserved for scenarios where accuracy is the sole priority and latency is not a constraint. For scalable, efficient production systems, moderately complex deep learning models (like BiLSTM) provide the best return on investment.

The work successfully demonstrates that a well-architected system using BiLSTM can achieve expert-level automation in IT support workflows, bridging the gap between theoretical AI performance and industrial operational efficiency.