💻 computer science

What Do Resume–Job Matching Benchmarks Actually Measure? Keywords, Embeddings and LLMs on Public Data

This paper critically evaluates public resume–job matching benchmarks by demonstrating that current methods often rely on data leakage and noisy labels, are vulnerable to superficial formatting artifacts like markdown, and offer only modest gains over simple keyword baselines (with LLM judges not significantly beating embeddings), thereby recommending stricter evaluation protocols including unseen-resume splits and data cleaning.

Original author: Nusrat Jahan Rita

Published 2026-11-11✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original author: Nusrat Jahan Rita

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the author. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, millions of people apply for jobs by sending digital résumés to automated systems. These systems act as the first gatekeepers, scanning thousands of applications to decide which human candidates are worth a recruiter's time. For years, these tools relied on simple keyword searches, looking for exact matches between the words in a job description and the words in a résumé. Recently, however, the technology has shifted toward more sophisticated artificial intelligence. These newer systems use "embeddings," a method that converts text into mathematical representations to understand the meaning behind the words, rather than just counting them. Even more advanced are large language models, which can read and reason about complex text much like a human does. The promise of these tools is a fairer, more efficient hiring process where the best candidates rise to the top based on their actual qualifications.

But how do we know these systems are actually working as intended? Most of the research behind these tools relies on private data owned by large companies, making it impossible for outside scientists to verify the results or check if the training data was flawed. To understand what is really happening, a researcher named Nusrat Jahan Rita decided to audit the only public dataset available that grades how well a résumé fits a job. This dataset contains thousands of pairs of job descriptions and résumés, each labeled by humans as a "Good Fit," a "Potential Fit," or "No Fit." The goal was to see if the latest artificial intelligence methods truly improve hiring decisions, or if they are simply exploiting hidden patterns in the data that have nothing to do with a candidate's actual suitability.

The investigation began by testing three different types of matching methods on this public data. First, the researcher compared the old-school keyword approach against the newer embedding systems. The results showed that the embedding models did perform slightly better, ranking candidates more accurately than the keyword search. However, the improvement was modest. The more surprising discovery came when the researcher examined how the data was split for testing. In the standard setup, a résumé might appear in the training data for one job and then be tested against a different job. Because the system had already seen that specific résumé before, it could simply memorize the label from the first job and apply it to the second. When the researcher removed these repeated résumés from the training set to ensure the system was truly learning to match new candidates, the performance of the advanced models dropped significantly. This revealed that much of the reported success in previous studies was actually due to the system recognizing the résumé itself, rather than understanding the connection between the résumé and the new job.

The study also put a small, open-source language model to the test, asking it to act as a judge to rank the candidates. While this model could read the text and make a decision, it did not perform significantly better than the simpler embedding systems. More importantly, it was vastly slower, taking about 850 times longer to process a single job application than the embedding models. For a hiring platform that needs to sort through thousands of applications quickly, this speed difference makes the expensive language model a poor practical choice, despite its ability to reason through text.

Perhaps the most critical finding concerned the quality of the human labels themselves. The dataset assumed that the distinction between a "Good Fit" and a "Potential Fit" was a clear measure of how well a candidate matched a specific job. However, the analysis showed that these labels were often determined more by the job posting or the résumé itself than by the actual match between the two. In many cases, every candidate for a specific job received the same label, regardless of their individual differences. When the researcher asked a second human to blindly re-evaluate a sample of these pairs, the agreement between the two humans was very low. In fact, the second human agreed with the original dataset labels more often than they agreed with the first human. This suggests that the labels in the public dataset are noisy and inconsistent, meaning that training a computer to predict these labels might just be teaching it to mimic human confusion rather than to find the best candidate.

Finally, the study uncovered a subtle but powerful bias related to formatting. The researcher took résumés and added standard formatting marks, such as bold headings and bullet points, without changing a single word of the content. When these formatted résumés were fed into the embedding models, their scores jumped dramatically. A candidate with a plain text résumé could suddenly appear much more qualified than an equally qualified candidate with a formatted résumé, simply because the model rewarded the visual structure. This effect was so strong that it happened regardless of whether the candidate was actually a good fit for the job. The models were not judging the content; they were reacting to the style.

The study concludes that while modern embedding tools offer a small improvement over simple keyword searches, the current benchmarks used to test them are deeply flawed. The data leaks allow systems to obtain inflated scores by memorizing résumés, the human labels are unreliable, and the models are easily tricked by formatting tricks. The researcher recommends that future tests must strictly prevent résumés from appearing in both training and testing sets, focus on the clearer distinction between "fit" and "no fit" rather than the vague "potential" category, and remove formatting before analysis. Until these issues are addressed, the automated systems guiding hiring decisions may be ranking candidates based on how well they fit a flawed dataset or how nicely they are formatted, rather than on their true ability to do the job.

Technical Summary: What Do Resume–Job Matching Benchmarks Actually Measure?

Problem Statement
Automated resume–job matching systems are critical for filtering candidates in online hiring, yet the field lacks reliable public benchmarks. Most high-performing models are trained on private data, preventing reproducibility and independent verification of label quality or data splits. The only known public dataset with graded fit labels ("Good Fit," "Potential Fit," "No Fit") is the resume-job-description-fit dataset. However, this dataset lacks documentation regarding label assignment, data splitting protocols, or potential data leakage. This paper investigates whether current benchmarks accurately measure candidate suitability or if they inadvertently measure artifacts such as resume memorization, job-specific biases, or formatting preferences.

Methodology
The study conducts a comprehensive audit of the resume-job-description-fit dataset (7,987 pairs, 351 jobs, 643 resumes) by comparing three families of methods and performing rigorous data integrity checks:

Method Comparison:
- Keyword Methods: TF-IDF and BM25.
- Sentence Embeddings: Zero-shot models (E5-base-v2, BGE-base-en-v1.5, MPNet) and fine-tuned variants (BGE-base with CoSENT loss).
- LLM Judges: Small open instruction-tuned models (Qwen3-4B and Qwen3-1.7B) used as zero-shot judges and re-rankers.
- Baselines: A "Resume-prior" baseline that scores candidates based solely on their historical labels across other jobs (never reading the job description) and a resume-text-only classifier.
Experimental Design & Splits:
- Standard Split: Jobs are split 70/15/15, but 430 of 434 test resumes appear in the training set (leakage).
- Strict Split: Removes all training pairs containing test resumes to ensure unseen resume evaluation.
- 5×5 Cross-Validation: A grid approach to separate the effects of resume memorization from training set size.
- Formatting Bias Test: Resumes were rewritten by an AI (Phi-4-mini) to improve style without adding facts, and markdown was added/removed to isolate formatting effects on embedding scores.
Label Audit:
- Human Relabelling: The author and a second annotator blind-relabelled 150 and 40 pairs respectively to assess agreement with the original dataset labels.
- Statistical Analysis: Used NDCG@10, AUC (Fit vs. No Fit and Good vs. Potential), and Cohen's $\kappa$ for agreement. Statistical significance was tested via paired permutation tests and bootstrap resampling.

Key Contributions

Reproducible Benchmark Audit: The first reproducible ranking comparison of keywords, embeddings, and LLMs on the only public graded resume–job benchmark, including per-job significance tests and cost analysis on free hardware.
Evidence of Data Leakage: Demonstrated that the standard dataset split allows 430 of 434 test resumes to appear in training. The paper quantifies that the performance gain of fine-tuned models is largely driven by seeing these specific resumes, not just by learning general fit signals.
Label Reliability Audit: Revealed that the "Good vs. Potential" labels are largely determined by the specific job and resume identity rather than the match quality between them. Human agreement with the dataset labels is weak ( $\kappa \approx 0.32$ ), and inter-annotator agreement is even lower.
Formatting Bias Discovery: Identified that E5-family embedding models significantly reward markdown formatting (e.g., bold headings, bullets) even when content is unchanged. Adding markdown increased scores by 0.91 standard deviations, regardless of actual candidate fit.
Practical Recommendations: Proposed a "strict split" protocol, the use of "fit vs. no fit" as the primary target over fine-grained labels, and the necessity of resume-only baselines to detect memorization.

Results

Embeddings vs. Keywords: Zero-shot embeddings (E5, BGE) outperformed TF-IDF significantly but modestly (NDCG@10: 0.739 vs. 0.706).
Fine-Tuning & Memorization: Fine-tuning improved performance (NDCG@10 0.833 on standard split), but this gain collapsed when test resumes were excluded from training (dropping to 0.791). The difference between standard and strict splits was almost entirely attributable to the model having seen the test resumes previously.
LLM Judges: The Qwen3-4B judge did not significantly outperform zero-shot embeddings (NDCG@10 0.807 vs. 0.784) but was approximately 850 times slower (27s vs. 0.03s per job).
Label Quality: No job-aware method separated "Good" from "Potential" fit significantly better than chance against the dataset labels. Conversely, a baseline that ignored the job description entirely achieved an AUC of 0.727 for "Good vs. Potential," suggesting the label reflects resume characteristics more than the match.
Human Agreement: The author's blind relabelling agreed weakly with the dataset ( $\kappa = 0.324$ for fit vs. no fit). A second annotator agreed with the dataset more than with the author, indicating no single source serves as a gold standard.
Formatting Bias: E5-base scores increased by 0.74 SD for AI-polished resumes and 0.91 SD for resumes with added markdown, with no corresponding improvement in actual fit. This bias affected both fit and non-fit candidates equally.

Significance and Claims
The paper argues that current ranking scores on this benchmark reflect how closely a method mimics an unknown, noisy labelling process rather than how well it judges candidate suitability. The author claims that:

Leakage inflates performance: Standard job-grouped splits make fine-tuning appear more effective than it is by allowing models to memorize specific resumes.
Labels are unreliable: The "Good vs. Potential" distinction is not a robust signal for ranking; "Fit vs. No Fit" is a more reliable target.
Formatting is a confounder: Embedding models can be manipulated by presentation (markdown) rather than content, posing a risk for fairness in automated screening.
LLMs are not a silver bullet: Small open LLMs do not offer significant accuracy gains over efficient embeddings for this task and incur prohibitive latency costs.

The author concludes that future work requires strict data splits (unseen resumes), reliable multi-annotator labels, and preprocessing steps to remove formatting bias before embedding. The author releases her code, splits, and human labels to facilitate these improvements.

More like this