💬 NLP

Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation

This paper introduces a modular, Structure Mapping Theory-based pipeline for educational analogy generation that decomposes the task into four stages, demonstrating through extensive evaluation of 12 LLMs that sub-concept grounding significantly improves explanation quality and retrieval precision while revealing cross-stage interactions critical for generating high-quality analogies.

Original authors: Mariam Barakat, Ekaterina Kochmar

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mariam Barakat, Ekaterina Kochmar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to explain a complex, unfamiliar machine (like a quantum computer) to a child. The best way to do this isn't to list technical specs; it's to compare it to something they already know, like a bicycle. This is the power of an analogy.

However, teaching computers to come up with these "aha!" moments is surprisingly hard. Large Language Models (LLMs) are great at writing, but when asked to create analogies, they often stumble, creating confusing or misleading comparisons.

This paper presents a new, modular pipeline—think of it as an assembly line—to help computers build better educational analogies. Instead of asking the computer to "just write an analogy," the researchers break the job down into four distinct stations, testing different tools at each stop to see what works best.

Here is how their "Analogy Factory" works, explained simply:

The Four Stations of the Analogy Factory

1. The Source Finder (Finding the Right Comparison)

The Job: The computer needs to find a familiar "Source System" (like a bicycle) to explain the "Target System" (the quantum computer).
The Experiment: They tested two ways to find this source:
- Closed Setting: The computer picks from a pre-approved list of 100 familiar things (like a multiple-choice quiz).
- Open Setting: The computer has to invent a source from scratch (like a free-response essay).
The Finding: When the computer is given a list to choose from, it does much better if it knows the "parts" of the target first. It's like finding a matching puzzle piece; if you know the shape of the piece you need, you find the match faster. However, when asked to invent a source from thin air, knowing the parts didn't help much. The computer just guessed.

2. The Sub-Concept Matcher (Connecting the Dots)

The Job: Once the computer has the Target (Quantum Computer) and the Source (Bicycle), it must map the specific parts. The "wheels" of the bike must map to the "qubits" of the computer.
The Experiment: They tested if the computer could align these parts correctly, with or without extra background descriptions.
The Finding: The computer is surprisingly good at this if it just focuses on the parts. Interestingly, adding too much background text sometimes confused it, like trying to solve a puzzle while someone is shouting extra instructions. The model Grok-4 was the champion here, beating previous records.

3. The Explanation Writer (Telling the Story)

The Job: Now that the parts are matched, the computer must write a sentence explaining why the comparison works.
The Experiment: They tested different "inputs" to see what helped the writer. Did it need just the names? The background story? Or a list of the matched parts?
The Finding: The biggest boost came from giving the computer the matched parts (e.g., "Wheels = Qubits"). It's like giving a writer an outline; the story becomes much clearer. The model GPT-4.1-Mini was the best storyteller, especially when it had that outline.

4. The Quality Control Inspector (Grading the Work)

The Job: How do we know if the analogy is good?
The Experiment: They used a "Judge" (another AI, Claude Sonnet 4.6) to grade the analogies on three things:
1. Coherence: Does it make sense?
2. Mapping: Are the parts connected correctly?
3. Explanatory Power: Does it actually help a learner understand?
The Finding: The AI Judge was very good at ranking analogies (saying "This one is better than that one"), but it wasn't perfect at giving exact scores (like saying "This is a 2.5 out of 3"). It's like a teacher who can easily tell you which essay is the best in the class, but might struggle to agree on the exact grade number.

The Big Takeaways

No "One-Size-Fits-All" Robot: The best computer model for finding a source isn't the same as the best one for writing the explanation. You need a team of specialists, not a single generalist.
Structure is King: The most important ingredient for a good analogy is structure (matching the parts), not just surface-level words. If you tell the computer "Here are the parts that match," the result is much better.
The "Open" Problem: Computers are still bad at coming up with new sources from scratch. They are much better at picking from a list.
Human vs. Machine: When humans graded the analogies, they agreed more on the ranking (which is better) than on the score (how good it is). The AI Judge mimicked this human behavior well.

What This Means (and Doesn't Mean)

The paper shows us how to build a better "factory" for educational analogies by breaking the task into steps and using the right tools for each step.

Important Note: The paper does not claim this system is ready to replace teachers, nor does it claim it works for medical advice or clinical uses. It strictly focuses on the technical process of generating and evaluating analogies for educational concepts using current AI models. The authors also note limitations, such as the system only working in English and the fact that errors in the first step (finding the source) can ruin the whole analogy later on.

In short, this paper is a blueprint for building a smarter, more structured way to teach computers how to make the "Aha!" moments that help us learn.

Technical Summary: Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation

Problem Statement

Analogies are critical for educational reasoning, allowing learners to map unfamiliar target systems to familiar source systems through relational structures. However, despite advances in Large Language Models (LLMs), automatically generating high-quality educational analogies remains a significant challenge. Poorly constructed analogies can reinforce misconceptions through incorrect mappings or oversimplification.

Current research is fragmented, addressing retrieval, generation, or evaluation in isolation rather than as a unified pipeline. Existing approaches often rely on surface-level similarity in closed candidate pools, fail to utilize sub-concept annotations for guiding generation, and lack robust evaluation methods that align with human judgment. Furthermore, most studies focus exclusively on specific model variants (e.g., GPT-4), leaving cross-model generalizability unexplored.

Methodology

The authors propose a modular pipeline for educational analogy generation, grounded in Structure Mapping Theory (Gentner, 1981). This pipeline decomposes the task into four distinct stages, each corresponding to a specific research question (RQ):

Source Finding (RQ1): Identifying suitable source systems for a given target. This is evaluated in two settings:
- Closed Setting: Retrieval from predefined candidate pools (SCAR and ParallelPARC datasets) using embedding models and LLM-based re-ranking.
- Open Setting: Free generation of candidate sources by LLMs without predefined pools.
Sub-concept Generation (RQ2): Aligning components of the target and source systems. The authors distinguish between matching (aligning given sub-concepts) and generation (producing source sub-concepts from scratch).
Explanation Generation (RQ3): Producing natural-language explanations that articulate why the analogy holds, given the target, source, and sub-concept mappings.
Evaluation (RQ4): Assessing analogy quality using an LLM-as-a-judge approach (Claude Sonnet 4.6) validated against human annotations.

Experimental Setup

Datasets: The study utilizes SCAR (400 system-level analogy instances with sub-concept annotations) and ParallelPARC (310 gold analogies with relational alignments).
Models:
- Embedding Models: Seven models (including OpenAI, E5, BGE, SPECTER) for closed-setting retrieval.
- LLMs: Twelve models across six families (OpenAI GPT-4.1, GPT-OSS, Meta LLaMA 3.1, Google Gemini, xAI Grok, DeepSeek, Alibaba Qwen) for open-ended generation and evaluation.
Evaluation Metrics:
- Retrieval: Hit@K (exact and semantic).
- Sub-concepts: System Accuracy (matching) and Semantic Match Accuracy (generation).
- Explanation: Cosine similarity (SBERT) against gold references.
- Quality: LLM-as-a-judge scores on Coherence, Mapping Soundness, and Explanatory Power, compared against human rankings and scores.

Key Contributions

Modular Pipeline: A systematic framework decomposing analogy generation into source finding, sub-concept generation, explanation generation, and evaluation, enabling stage-specific analysis.
Systematic Cross-Model Evaluation: An extensive evaluation of 12 LLMs and 7 embedding models, revealing that optimal model performance varies significantly across pipeline stages.
Closed-to-Open Evaluation Methodology: A transition from controlled evaluation against gold annotations to evaluating LLM-generated candidates using baseline, embedding-based, and re-ranking selection strategies.
LLM-as-a-Judge Validation: A rigorous validation of Claude Sonnet 4.6 as an evaluator, demonstrating its reliability in ranking analogies compared to human judgments.

Key Results

1. Source Finding

Closed Setting: Embedding-based retrieval benefits significantly from sub-concept grounding. Adding sub-concepts to the input improved Hit@20 scores (e.g., OpenAI-Large increased from 0.710 to 0.792 on SCAR). Background descriptions provided mixed or negligible benefits.
Selection Strategy: LLM re-ranking using gold sub-concepts achieved the best Hit@1 (24.5%), while embedding-only retrieval performed best at Hit@2 and Hit@3.
Open Setting: Adding sub-concepts to generation prompts yielded only marginal gains (0.4–1.6 percentage points) compared to the substantial benefits seen in retrieval. Target-only prompting was deemed more practical. Top models (Gemini-2.5, GPT-OSS-120B) clustered closely in performance, suggesting scale alone does not dictate quality.

2. Sub-concept Generation

Matching: Performance was slightly higher without background descriptions. Grok-4 led with 0.78 system accuracy, surpassing the prior state-of-the-art (0.62).
Generation: Background descriptions provided a clear benefit, raising average semantic match accuracy from 0.298 to 0.412. The LLaMA-3.1-70B and LLaMA-3.1-405B models led this stage (0.47), while GPT-based variants generally underperformed. Notably, larger models (e.g., GPT-OSS-120B) did not consistently outperform smaller ones.

3. Explanation Generation

Input Configuration: The introduction of sub-concepts caused a substantial jump in explanation quality (average SBERT similarity rising from ~0.72 to ~0.85). Background descriptions had negligible effects.
Best Configuration: S5 (System names + paired sub-concept mappings) yielded the highest quality.
Model Performance: GPT-4.1-Mini was the strongest overall performer, particularly in sub-concept-enriched settings.

4. Evaluation

LLM-as-a-Judge: DeepSeek-R1 and Grok-4 produced the highest-quality analogies according to the judge.
Human Alignment: Claude Sonnet 4.6 aligned more reliably with human rankings (relative ordering) than with fine-grained absolute scores. Spearman correlation for rankings was generally moderate-to-high (e.g., $\rho=0.705$ for one annotator), whereas dimension-level scores showed lower agreement.
Selection Strategy: Re-ranking improved mapping quality, while baseline selection yielded slightly higher coherence.

Significance and Claims

The paper claims that no single model dominates across all stages of the analogy generation pipeline. Instead, the optimal configuration is stage-dependent:

Grok-4 excels in sub-concept matching.
LLaMA-3.1-70B is superior for sub-concept generation.
GPT-4.1-Mini leads in explanation generation.
OpenAI embeddings with sub-concepts are best for closed retrieval, while target-only prompting is more practical for open generation.

The authors emphasize that sub-concept grounding is a key driver of quality, substantially improving retrieval precision and explanation quality, though offering limited benefit in open-ended source generation. Furthermore, the study highlights that LLM-as-a-judge is better suited for comparative evaluation (ranking) than for assigning fine-grained absolute scores.

The work concludes that a modular, stage-aware approach is necessary for effective educational analogy generation, as isolated studies fail to capture the cross-stage interactions that determine final analogy quality.

Limitations

The authors acknowledge several limitations:

Experiments are restricted to English-language data.
The Semantic Match Accuracy threshold (0.7) is a fixed heuristic.
The pipeline is not evaluated end-to-end; errors may accumulate across stages.
Retrieval relies solely on dense embeddings without hybrid search strategies.
Evaluation does not explicitly model user-dependent preferences or learner profiles.
Human evaluation involves a limited, homogeneous group of annotators from a single institution.
Open-setting source finding was evaluated only on the SCAR dataset due to computational constraints.