🤖 AI

From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

This paper presents an end-to-end pipeline that leverages large language models constrained by competency graphs and evidence spans to automatically and transparently tag learning resources with structured competencies, achieving superior performance over existing baselines while enabling human-auditable curriculum analytics.

Original authors: Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the librarian of a massive, chaotic digital library where every book, video, and quiz is a "learning resource." Your goal is to help students find the right materials to learn specific skills (called "competencies"), like "Linear Algebra" or "Public Speaking."

Currently, a human librarian has to read every single page of every document and manually write down which skills it teaches. This is slow, boring, and prone to errors. If the librarian misses a page, the student might never find the help they need.

This paper proposes a new, smarter way to do this tagging job using a team of three specialists working together: a Searcher, a Smart Reader, and a Logic Checker.

The Problem: The "Black Box" Librarian

Previous attempts to use Artificial Intelligence (AI) to do this tagging had two big problems:

It was too much work for humans: They had to check everything manually.
The AI was a "Black Box": If the AI said, "This PDF teaches 'Probability'," it couldn't explain why. It just gave an answer. If the answer was wrong, no one knew where the AI went wrong, making it hard to trust.

The Solution: The "LBG" Pipeline

The authors built a system called LBG (LLM + BM25 + Graph). Think of it as a three-step assembly line that turns messy course materials into a perfectly organized skill map.

Step 1: The Searcher (Retrieval)

Before the AI tries to guess the skills, it first looks at a "menu" of possible skills.

The Metaphor: Imagine you are trying to describe a fruit. Instead of guessing from the entire dictionary of the world (which includes "car," "cloud," and "toaster"), you first ask a search engine to find the top 20 fruits that sound like what you see.
How it works: The system scans the document and uses a fast search tool (BM25) to pull up a short list of the most likely skills (candidates) from the university's official skill list. This stops the AI from making wild guesses.

Step 2: The Smart Reader (The LLM)

Now, a Large Language Model (the "Smart Reader") looks at the document and that short list of candidates.

The Metaphor: This is like a strict editor who is only allowed to choose from the 20 fruits the Searcher found. The editor reads a specific paragraph, picks the best fruit from the list, and must point to the exact sentence that proves it.
The Innovation: The AI doesn't just say "Probability." It says, "This paragraph teaches Probability, and here is the specific sentence: 'The chance of rolling a six is 1 in 6.'" This creates a "paper trail" of evidence.

Step 3: The Logic Checker (Graph Constraints)

Finally, the system checks the "Family Tree" of skills.

The Metaphor: Imagine a family tree where "Advanced Calculus" is the child of "Basic Math." If the Logic Checker sees that a document is tagged with "Advanced Calculus" but has no mention of "Basic Math," it raises a red flag. It doesn't necessarily delete the tag, but it checks if the logic makes sense. It also cleans up duplicates (e.g., if the AI tagged "Math" and "Calculus" for the same tiny sentence, it might decide to keep just the more specific one).

The Results: How Well Did It Work?

The team tested this system on a real university's Computer Science department, covering 430 different course materials (PDFs, quizzes, slides) and 22 different skills.

Accuracy: The LBG system was the best at correctly matching documents to skills. It beat:
- AI that just guessed without help (Zero-shot).
- AI that got a few examples but no search help (Few-shot).
- Traditional search engines that just looked for similar words.
- Old-school computer programs trained on past data.
Trust: Because the system had to point to the exact sentence (the evidence), it was much easier for humans to verify the work. The "evidence" it produced was mechanically valid (the sentences actually existed in the text).
Efficiency: It worked well even when the system had to guess between many options, proving that narrowing down the choices first (Step 1) and checking the logic later (Step 3) makes the AI smarter.

The Bottom Line

This paper shows that you don't need a super-intelligent, magical AI to tag learning materials. Instead, you get better results by giving the AI a limited menu to choose from, forcing it to show its work (the evidence), and having a logic checker ensure the skills make sense together.

This creates a system that is not only accurate but also transparent, allowing teachers and students to trust the tags because they can see exactly where the AI found the proof.

Technical Summary: From Learning Resources to Competencies: LLM-Based Tagging with Evidence and Graph Constraints

Problem Formulation
The paper addresses the challenge of automatically aligning heterogeneous Learning Management System (LMS) resources (e.g., PDFs, slides, quizzes, assignments) with structured institutional competency frameworks. While manual tagging is accurate, it is labor-intensive and difficult to scale. Fully automatic methods often lack transparency, making them unsuitable for educational auditing. The core problem is defined as computing a resource-level competency mapping $M: R \to 2^C$ (where $R$ is a set of resources and $C$ is a set of competency nodes in a directed graph $G$ ) that is both accurate and traceable through verifiable evidence spans within the text.

Methodology: The LBG Pipeline
The authors propose an end-to-end pipeline, termed LBG (LLM+BM25+Graph), which operates at the fragment level before aggregating to the resource level. The process consists of four distinct stages:

Resource Representation and Fragmentation:
Heterogeneous LMS resources are decomposed into pedagogical fragments ( $\phi(r)$ ) based on instructor-authored structures (e.g., sections, slides, quiz items). If explicit structure is missing, fallback segmentation uses paragraph and sentence boundaries. Each fragment retains provenance metadata to enable later auditing.
Candidate Generation (Retrieval):
To constrain the prediction space and avoid unconstrained generation, the system retrieves a bounded set of top- $K$ candidate competencies ($Cand(x)$) for each fragment $x$ . This is achieved by comparing the fragment text against competency profiles (containing labels, descriptions, synonyms, and graph context) using lexical ranking (BM25).
LLM-Based Tagging with Evidence:
A Large Language Model (LLM), specifically gpt-4o-mini, acts as a constrained semantic decision module. Given a fragment and its retrieved candidate set, the LLM:
- Selects relevant competencies exclusively from the candidate set.
- Assigns a confidence score.
- Extracts a verifiable evidence span (character offsets) from the fragment text supporting the selection.
  Outputs are formatted in a strict JSON schema to ensure machine parsability.
Graph-Aware Reconciliation and Aggregation:
Fragment-level predictions are refined using the structural relations of the competency graph $G$ (e.g., prerequisites, hierarchical relations). This stage performs:
- Granularity Control: Reducing redundant co-assignments between parent and child nodes.
- Prerequisite Coherence Checks: Flagging potentially incoherent tags (e.g., an advanced competency predicted without evidence of its prerequisites) without forcing automatic correction.
- Deduplication: Merging overlapping predictions.
  Finally, fragment scores are aggregated (via weighted sum or max) to produce a resource-level competency score, which is thresholded to generate the final mapping.

Key Contributions

Evidence-Grounded Tagging: Unlike standard classification, the pipeline requires the model to output specific text spans justifying each tag, facilitating human auditing and transparency.
Constrained Decision Space: By integrating retrieval (BM25) with LLM selection, the method mitigates the issue of LLMs hallucinating competencies or performing poorly on local, less-represented frameworks.
Graph-Aware Reconciliation: The approach explicitly utilizes the competency graph's topology (prerequisites and hierarchies) to post-process predictions, ensuring structural consistency in the final output.
End-to-End Pipeline: The system bridges the gap between raw LMS content and structured competency analytics, operating effectively across heterogeneous resource types.

Experimental Results
The approach was evaluated on a dataset from the Computer Science department at the Université de Technologie de Compiègne (UTC), comprising 430 resources, 26 course units, and 22 competencies. The evaluation used 5-fold cross-validation at the course unit level.

Performance: The LBG pipeline achieved a Micro-F1 of 0.571 and Macro-F1 of 0.504 at the fragment level, and a Macro-F1 of 0.507 at the resource level.
Comparative Advantage: LBG outperformed:
- Zero-shot and Few-shot LLMs: Which lacked retrieval constraints and graph reconciliation.
- Retrieval/Similarity Baselines: Including BM25, Dense bi-encoders, and SBERT, which struggled when converting ranked lists to multi-label predictions.
- Supervised Classifiers: Including LogReg, LinearSVC, and a Supervised Transformer, which generalized less effectively across unseen course units.
Evidence Validity: LBG achieved the highest Span Validity (0.511) among span-producing methods, indicating that the constrained approach stabilizes the extraction of mechanically traceable evidence.
Ranking Quality: The pipeline achieved a Mean Reciprocal Rank (MRR) of 0.82, significantly higher than baselines, suggesting correct competencies are ranked highly in the candidate list.

Significance and Claims
The paper claims that combining retrieval-constrained LLM tagging with graph-based reconciliation offers a robust solution for competency-based curriculum analytics. The authors emphasize that their method produces mechanically traceable evidence spans, which is critical for human auditing and educational analysis, addressing the "black box" nature of standard LLM outputs.

The results suggest that while LLMs show promise for skill tagging, their performance on local frameworks is significantly enhanced by constraining the label space via retrieval and enforcing structural consistency via graph constraints. The authors note that their approach is model-agnostic and that the pipeline design supports downstream LMS applications such as competency-based retrieval, assessment alignment, and curriculum coverage analysis.

Limitations
The authors modestly acknowledge several limitations:

The evaluation is restricted to a single institutional framework (UTC Computer Science) and text-only resources.
The evidence validity metric tests mechanical offset traceability rather than semantic faithfulness (i.e., whether the text actually justifies the tag).
The system relies on competency profile text for retrieval, which may limit performance on highly ambiguous or implicit competencies without richer graph constraints.
Future work is needed to validate the approach on larger, multilingual taxonomies (e.g., ESCO, ROME) and to improve candidate generation.