💻 computer science

Explainable Job Category Detection Using DistilBERT: A Three-Class Transformer Model for Classifying Real, Fake, and Invalid Online Job Offers

This study presents an explainable, lightweight DistilBERT-based system that improves online job offer screening by classifying postings into Real, Fake, and Invalid categories with 98.74% accuracy, addressing the limitations of traditional binary models through a balanced three-class approach and SHAP-based interpretability.

Original authors: Nnaemeka Ugwumba, Ugochi A. Okengwu, Clement Barisua Ndeekor, Peter Sunday Jaja

Published 2026-11-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nnaemeka Ugwumba, Ugochi A. Okengwu, Clement Barisua Ndeekor, Peter Sunday Jaja

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The digital landscape of job hunting has transformed from a local newspaper classified section into a global, instantaneous marketplace. Platforms like LinkedIn and Indeed allow employers to reach millions of candidates instantly, but this same openness has created a haven for a specific type of cybercrime: online recruitment fraud. Scammers craft deceptive advertisements designed to steal money, harvest personal data for identity theft, or lure victims into illegal schemes. For years, researchers have tried to build computer systems that can spot these fake postings. Early attempts relied on simple rules and basic statistical patterns, which often failed when scammers learned to mimic the language of legitimate companies. More recently, scientists have turned to advanced artificial intelligence models capable of understanding the context and nuance of human language, similar to how a human reader senses when a story feels wrong. However, these powerful models are often too slow for real-time use, and most existing systems are limited to a simple yes-or-no decision: is this job real or fake? This binary approach struggles with the messy reality of the internet, where many postings are not malicious scams but are simply broken, incomplete, or nonsensical.

A team of researchers at the University of Port Harcourt in Nigeria has developed a new approach that addresses these limitations. They created a system that does not just ask if a job offer is real or fake, but adds a third category: invalid. This distinction is crucial because it allows the system to separate dangerous scams from harmless errors, such as a posting that is missing key details or contains gibberish. To build this tool, the researchers combined three different collections of job advertisements. One set contained thousands of verified real and fake postings from previous studies, while another provided examples of text that was clearly broken or malformed. The resulting dataset included over 34,000 records, but the numbers were heavily skewed, with real job offers vastly outnumbering the fake and invalid ones. To teach the computer effectively, the team did not simply copy and paste examples to balance the numbers, a common technique that can create artificial data. Instead, they carefully reduced the number of real examples and created new, varied examples of fake and invalid text by rearranging words and adding noise, ensuring the computer learned from genuine linguistic patterns rather than synthetic duplicates.

The core of their system is a streamlined version of a large language model called DistilBERT. Think of this model as a highly efficient student who has read millions of books and learned the rules of language, but is smaller and faster than its larger counterparts. The researchers trained this model on their balanced dataset of 9,000 job postings, teaching it to recognize the subtle differences between a legitimate offer, a scam, and a broken post. The training process took less than ten minutes on a standard computer, after which the model was tested on a completely new set of 1,350 job offers it had never seen before. The results were remarkably precise. The system correctly identified the category of the job offer in nearly 99 percent of cases. It was particularly good at spotting invalid posts, getting them right every single time in the test group, and it successfully identified fake scams with a success rate of 98 percent. This level of accuracy is significant because it outperforms previous systems that were limited to binary choices, even though this new system had to solve a more difficult three-way classification problem.

Beyond just making a prediction, the researchers wanted to ensure the system was transparent. They integrated a feature that explains why the computer made its decision. When the system flags a job as a scam, it highlights the specific words in the text that triggered the alarm, such as urgent requests for bank details or promises of unrealistic salaries. If a post is marked as invalid, the system points out the broken sentences or random characters that make it nonsensical. This capability builds trust, allowing a human user to see the evidence behind the classification rather than just receiving a black-box verdict. The team also built a working web application that demonstrates this technology in real time. When a user pastes a job description into the interface, the system analyzes the text and returns a classification along with a confidence score in just one to three seconds. This speed proves that sophisticated artificial intelligence can be deployed in practical, everyday tools without requiring massive computing power.

The study also compared their streamlined model against two other types of artificial intelligence architectures, one that processes text sequentially and another that is even simpler. While these alternative models performed well, the streamlined language model remained the most accurate, correctly classifying the job offers with the highest success rate. The researchers acknowledge that their system is not perfect and that real-world fraud tactics are constantly evolving. They note that their training data was limited to English-language postings and that the test set was balanced artificially, which might not reflect the natural rarity of scams in the wild. However, the work demonstrates that a three-class framework is not only possible but highly effective. By distinguishing between real, fake, and invalid content, and by providing clear reasons for its decisions, this system offers a more reliable and user-friendly way to protect job seekers in an increasingly complex digital environment.

Technical Summary: Explainable Job Category Detection Using DistilBERT

Problem Statement
Online recruitment platforms have become primary vectors for Online Recruitment Fraud (ORF), where malicious actors use deceptive job postings to extract fees, harvest data, or facilitate illicit schemes. Existing computational approaches to this problem predominantly rely on binary classification (Real vs. Fake). The authors argue that this binary framing is a significant limitation because it forces incomplete, corrupted, or nonsensical postings—content that is not necessarily fraudulent but is structurally invalid—into either the "Real" or "Fake" categories. This misclassification reduces system reliability and hinders platform moderators from distinguishing between malicious intent and low-quality submissions. Furthermore, prior transformer-based studies often lack real-time deployment capabilities and fail to provide explainable AI (XAI) mechanisms, limiting user trust and accountability.

Methodology
The study employs a Design Science Research (DSR) orientation, following the CRISP-DM process to develop a functional three-class classification system.

Data Integration and Preprocessing: The model was trained on a combined corpus of 34,325 records derived from three sources: the Employment Scam Aegean Dataset (EMSCAD), the job-scam domain of the DIFrauD benchmark, and a researcher-curated "Invalid Token" dataset. Text fields (title, company, description, etc.) were concatenated, cleaned of HTML and extraneous characters, and standardized.
Addressing Class Imbalance: The initial dataset exhibited a severe 21.8:1 imbalance (Real:Fake:Invalid). To resolve this without relying on synthetic interpolation (SMOTE), which operates on feature vectors rather than raw text, the authors employed a hybrid strategy:
- Undersampling: The dominant "Real" class was reduced to 3,000 records.
- Textual Augmentation: The "Fake" and "Invalid" classes were expanded to 3,000 records each using specific construction patterns (e.g., alphanumeric noise, symbol spam, scrambled sequences) rather than SMOTE.
- Class Weighting: Balanced class weights were applied during training.
- Partitioning: The resulting 9,000-record balanced dataset was split into a stratified 70:15:15 train/validation/test ratio.
Model Architecture: The core classifier is a fine-tuned DistilBERT (distilbert-base-uncased), chosen for its balance of performance and efficiency (40% smaller and 60% faster than BERT). The model processes inputs up to 256 tokens and outputs a three-class probability distribution (Real, Fake, Invalid).
Training Configuration: The model was trained for four epochs using AdamW (learning rate $2 \times 10^{-5}$ ) with a linear scheduler and warm-up. The best checkpoint was selected based on validation loss.
Baselines: To contextualize performance, the study trained Bidirectional LSTM and RNN models under identical data conditions, balancing, and hyperparameters.
Deployment and Explainability: The final model was deployed via a FastAPI backend with an HTML/JS frontend. It integrates SHAP (SHapley Additive exPlanations) to provide token-level attribution, highlighting specific words that influenced the prediction (e.g., urgency cues for "Fake" or structural incoherence for "Invalid").

Key Results
Evaluated on a held-out test set of 1,350 samples (450 per class), the DistilBERT model demonstrated superior performance:

Accuracy: Achieved 98.74% overall accuracy and 98.74% balanced accuracy.
Class Performance:
- Fake: F1-score of 0.98, Recall of 0.98, ROC-AUC of 0.998.
- Invalid: Perfect scores (1.00) across Precision, Recall, F1, and ROC-AUC (1.000).
- Real: F1-score of 0.98, Recall of 0.98, ROC-AUC of 0.998.
Error Analysis: Of the 17 misclassified samples, all occurred between the "Real" and "Fake" classes, reflecting the inherent difficulty of distinguishing fraudulent posts that mimic legitimate language. No "Invalid" samples were misclassified.
Baseline Comparison:
- LSTM: Achieved 98.44% accuracy.
- RNN: Achieved 97.04% accuracy.
- The results confirm that while all models benefited from the balanced dataset, the transformer-based approach outperformed recurrent architectures, likely due to pretrained contextual representations.
Efficiency: The system achieved inference times between 1.0 and 2.9 seconds, with a total training time of approximately 622 seconds for four epochs.

Significance and Claims
The paper claims four primary contributions:

Three-Class Formulation: It moves beyond binary detection to explicitly categorize "Invalid" content, addressing a gap where nonsensical or malformed posts were previously forced into binary classes.
Imbalance Strategy: It proposes and validates a textual augmentation and weighting strategy specifically suited for transformer pipelines, avoiding the limitations of SMOTE on raw text.
Efficiency-Accuracy Trade-off: It provides empirical evidence that a distilled transformer (DistilBERT) can achieve high accuracy on a complex three-class task with minimal computational cost (training in ~10 minutes, inference <3 seconds), making it viable for real-time screening.
Explainability in Deployment: It integrates SHAP-based explanations into a deployed web application, offering transparency that allows users to understand why a posting was flagged, thereby supporting trust and accountability.

Limitations and Future Work
The authors modestly qualify their findings, noting that:

The datasets may not fully represent the diversity of global recruitment fraud across all platforms and regions.
The study is limited to English-language postings.
Augmentation was applied before partitioning, potentially leading to optimistic performance estimates for the augmented classes.
The balanced test set does not reflect the natural prevalence of fraud (approx. 4%), meaning real-world precision for the "Fake" class may be lower.
The three-class scheme does not differentiate between specific fraud sub-types (e.g., advance-fee vs. multi-level marketing).

Future work is proposed to include external validation on cross-platform data, restricting augmentation to the training partition only, evaluating under natural class prevalence, extending to multilingual content, and incorporating non-textual signals like recruiter verification.

Technical Summary: Explainable Job Category Detection Using DistilBERT

More like this