Can We Improve Educational Diagram Generation with In-Context Examples? Not if a Hallucination Spoils the Bunch

Technical Summary: Improving Educational Diagram Generation with In-Context Examples

Problem Statement
Generative artificial intelligence (AI) is increasingly utilized in computing education to produce instructional materials, including diagrams. However, the quality of these outputs remains a significant concern for educators and students. Existing methods for diagram code generation, such as entity/relation extraction or standard in-context learning (ICL), often suffer from verbosity, poor layout quality, and, most critically, hallucinations. Hallucinations in Large Language Models (LLMs) are defined as misalignments with provided input context or factual knowledge, categorized into factuality (contradicting world knowledge or fabricating unverifiable facts) and faithfulness (deviating from user instructions or input context). Empirical research indicates that hallucination rates can range from 3% to 86% even in top-performing models. The core challenge addressed is the generation of reliable, high-quality educational diagrams that strictly adhere to source text contexts without introducing factual errors or logical inconsistencies.

Methodology
The study proposes a novel ICL method for generating diagram code (specifically in Graphviz's dot syntax) that leverages Rhetorical Structure Theory (RST) to select relevant in-context examples. The approach is grounded in the premise that RST, which analyzes text coherence through hierarchical relations between "elementary discourse units" (EDUs), can guide LLMs to visualize coherence relations consistent with user expectations.

The proposed pipeline involves two variations, RST1 and RST2, compared against a zero-shot (0-shot) baseline:

1. RST Analysis (Step 1): The LLM analyzes the source text to identify EDUs and their rhetorical relations (nuclei and satellites) based on Stede et al.'s annotation guidelines. This step is skipped in the 0-shot baseline.
2. Similarity Search (Step 2): The system compares the generated RST analysis against a pre-made dictionary of four example text-analysis-diagram triplets. The model selects the example whose RST structure is most similar to the input.
3. Example Construction (Step 3):
   • RST1: Uses the source text and the corresponding diagram code from the selected example.
   • RST2: Uses the RST analysis and the corresponding diagram code from the selected example.
4. ICL Generation (Step 4): The model generates initial diagram code using the selected example as a demonstration.
5. Repair and Refinement (Steps 5–7): The generated code undergoes an iterative repair process to fix syntax errors until it renders. A subsequent refinement step (Step 6) prompts the model to improve layout aesthetics (e.g., reducing line crossings, ensuring symmetry) based on a specific rubric, followed by a final repair cycle.

The study utilized GPT-4o and o3 models, testing them on 25 source texts categorized by difficulty (basic, medium, advanced). A total of 150 diagrams were generated and evaluated.

Key Contributions

1. RST-Based ICL Method: The authors propose and evaluate a new method for selecting in-context demonstrations for diagram code generation based on RST analysis, aiming to reduce context inconsistencies and hallucinations compared to zero-shot generation.
2. Evaluation Rubric: A five-scale rubric was developed to assess diagram quality across three dimensions:
   • Logical Organization (C1): Flow, structure, language clarity, and adherence to conventions.
   • Connectivity (C2): Uniformity and cohesion of connections (e.g., absence of orphan nodes).
   • Layout Aesthetic (C3): Presence of issues such as line crossings, element obscuration, asymmetry, and misalignment.
3. Hallucination Analysis: An in-depth investigation into the relationship between AI hallucination (both factual and faithfulness) and diagram quality, including the propagation of errors through multi-step generation pipelines.

Results

• Model Performance: The o3 model consistently outperformed GPT-4o across all methods and metrics, suggesting that models with stronger reasoning capabilities benefit more from this approach.
• Method Efficacy:
  • RST1 with o3 produced the highest rate of high-quality diagrams (scores > 3) for logical organization (C1).
  • RST2 generally performed worse than RST1 and 0-shot for connectivity (C2), likely due to the complexity of processing LLM-generated RST analyses as input.
  • The RST-based methods generally decreased the rate of factual hallucinations and improved faithfulness to the input context compared to zero-shot generation.
• Hallucination Findings:
  • Despite using verified source texts, up to 20% of AI-generated diagrams contained factual hallucinations.
  • Context Complexity: Higher complexity in source texts (advanced level) correlated with higher rates of hallucination and lower diagram quality scores.
  • Error Propagation: In the RST2 pipeline, hallucinations from the initial RST analysis (Step 1) or similarity search (Step 2) were frequently inherited by the final diagram output.
  • Self-Correction Limitations: LLMs struggled to detect and repair their own layout errors; the refinement step (Step 6) often failed to eliminate hallucinations or introduced new layout issues.
• Automated Evaluation: Automated assessment using LLMs (E1, E2, E3 methods) showed moderate alignment with human evaluations, with instruction-based reflection (E3) and combined instruction/ICL (E2) yielding the best results, particularly with the o3 model.

Significance and Claims
The paper claims that its RST-based ICL method offers a promising avenue for improving the reliability of automatically generated educational diagrams by aligning model outputs with user expectations and reducing context inconsistencies. However, the authors maintain a modest stance, acknowledging that the stochastic nature of LLMs leads to variable quality and that multi-step pipelines introduce risks of error propagation.

The study concludes that while the proposed method decreases factual hallucination and improves faithfulness, it is not a complete solution. The authors emphasize that due to the persistent risk of hallucination, the application of generative AI in educational contexts requires necessary safeguards to protect key stakeholders (teachers and students). They explicitly state that further empirical investigation is required to fully account for LLM stochasticity and that their current implementation, using a limited set of examples, does not encompass all existing coherence relations.