When LLM meets Fuzzy-TOPSIS for Personnel Selection through Automated Profile Analysis

Technical Summary: When LLM meets Fuzzy-TOPSIS for Personnel Selection through Automated Profile Analysis

Problem Statement
The paper addresses the challenges inherent in traditional personnel selection, particularly within the software engineering sector. Conventional recruitment methods rely heavily on subjective human judgment, leading to inconsistencies, biases, and inefficiencies. While Artificial Intelligence (AI) and Natural Language Processing (NLP) have advanced, existing solutions often fail to adequately account for the inherent uncertainty and subjectivity in human evaluations. Furthermore, there is a gap in integrating sophisticated NLP models with robust decision-making frameworks capable of handling vague criteria. The authors identify a need for an intelligent system that combines comprehensive candidate analysis with consistent, interpretable, and unbiased decision-making to automate and improve recruitment outcomes.

Methodology
The proposed solution is a hybrid framework termed LLM-TOPSIS, which integrates Large Language Models (LLMs) with Fuzzy Multi-Criteria Decision Making (MCDM). The methodology proceeds through three primary phases:

1. Dataset Construction and Expert Annotation:

   • A dataset was constructed from 100 LinkedIn profiles of software developers, focusing on four key attributes: Experience, Skills, Education, and About (self-introduction).
   • Three senior industry experts evaluated each profile using a standardized five-point Likert scale, which was subsequently mapped to three proficiency classes: Poor, Fair, and Excellent.
   • To address data scarcity and class imbalance, the dataset was augmented using paraphrasing, synonym substitution, and contextual expansion, resulting in 10,000 samples per sub-dataset. The data was balanced using random oversampling.

2. Transformer-Based Multi-Class Classification:

   • The DistilRoBERTa model was fine-tuned to perform multi-class classification for each candidate attribute. DistilRoBERTa was chosen for its efficiency (40% smaller and 60% faster than RoBERTa-base) while maintaining high performance.
   • Comparative experiments were conducted with RoBERTa-base and LastBERT to evaluate lightweight distilled architectures.
   • The models predict proficiency levels based on textual inputs, generating stable, aspect-wise scores.

3. Fuzzy-TOPSIS Based Ranking:

   • To aggregate the attribute-level predictions into a final candidate ranking, the Fuzzy Technique for Order of Preference by Similarity to Ideal Solution (Fuzzy TOPSIS) was applied.
   • Triangular Fuzzy Numbers (TFNs) were utilized to model the uncertainty and subjectivity of expert judgments. Linguistic terms (e.g., "Very High") were mapped to TFNs.
   • Criteria weights were determined via an Analytic Hierarchy Process (AHP)-inspired pairwise comparison by experts, resulting in the following weights: Skills (0.60), Experience (0.20), Education (0.15), and About (0.05).
   • The algorithm calculates the distance of each candidate from the Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) to derive a closeness coefficient for ranking.

Key Contributions
The paper outlines five primary contributions:

• Dataset Creation: Development of a unique dataset comprising LinkedIn profiles augmented with expert evaluations across multiple attributes.
• Hybrid Framework: Integration of LLM-generated proficiency scores (via fine-tuned DistilRoBERTa) with Fuzzy TOPSIS to explicitly address subjectivity and uncertainty, producing reliable rankings.
• Model Validation: Systematic fine-tuning and empirical validation of DistilRoBERTa as a lightweight, scalable backbone suitable for real-world deployment.
• High Accuracy: Demonstration of high concordance between model-generated rankings and expert judgments, with DistilRoBERTa achieving up to 91% accuracy on the Experience and Overall attributes.
• Scalable Framework: Establishment of an interpretable AI-driven recruitment framework that merges NLP with fuzzy decision-making for consistent personnel selection.

Results

• Classification Performance: The DistilRoBERTa model achieved an overall accuracy of 91% for the Experience and Overall attributes, and 87% for Skills. The "About" attribute achieved 99% accuracy. The model demonstrated high precision and recall, particularly in the "Poor" and "Excellent" categories, though some overlap was observed in the "Fair" category.
• Ablation Study: The study confirmed that data augmentation and fine-tuning were critical, as the baseline model on non-augmented data performed poorly (F1-score of 0.16). The integration of Fuzzy-TOPSIS improved ranking metrics (MAP increased from 0.93 to 0.99) without altering classification accuracy.
• Ranking Alignment: The LLM-Fuzzy TOPSIS rankings showed strong alignment with Human-TOPSIS and direct Human Expert rankings. Key metrics included:
  • Cosine Similarity: 0.983 (LLM-Fuzzy TOPSIS) vs. 0.981 (Human-TOPSIS).
  • Mean Average Precision (MAP) & Mean Reciprocal Rank (MRR): Both exceeded 0.99.
  • RMSE: 0.043, indicating minimal error in ranking deviation.
• Model Comparison: DistilRoBERTa and LastBERT outperformed the full RoBERTa-base model in terms of F1-score and error rate under limited-data conditions, validating the efficacy of distilled architectures.

Significance and Claims
The paper claims that merging NLP with fuzzy decision-making methods offers a scalable, transparent, and unbiased solution to recruitment difficulties. The study highlights that:

• The proposed framework can effectively replicate human expert judgment patterns with statistical consistency.
• It addresses the "black box" nature of some AI recruitment tools by providing an interpretable decision process through the Fuzzy TOPSIS mechanism.
• The system reduces reliance on subjective human methodologies, potentially minimizing prejudice and improving efficiency.
• While the current study focuses on software engineering, the authors suggest the framework has potential applicability in other sectors involving complex multi-criteria decisions under uncertainty, such as defense, healthcare, education, and finance.

The authors maintain a modest tone regarding future implications, noting that the current dataset is limited to software engineering and requires expansion for broader generalizability. They emphasize that future work will focus on augmenting the dataset, enhancing model interpretability (e.g., using SHAP or LIME), and verifying the system in actual recruitment scenarios.