Evaluating LLM-Generated Lessons from the Language Learning Students' Perspective: A Short Case Study on Duolingo

1. Problem Statement

The paper addresses a critical gap in current Large Language Model (LLM)-driven language learning applications, specifically Duolingo. While these tools effectively generate lessons for general real-world scenarios (e.g., greetings, ordering food), they lack support for profession-specific contexts.

• The Gap: Learners in multinational and multilingual environments require "professional fluency," defined as the ability to comfortably communicate work-related and domain-specific information. Current LLM-generated lessons rarely cover technical project specifications, stakeholder negotiations, or industry jargon.
• The Consequence: This limitation hinders learners from transitioning from foundational fluency to professional-level fluency, creating a disconnect between what the app teaches and the actual communicative needs of professionals (e.g., software engineers).

2. Methodology

The authors conducted a formative, exploratory study using a survey-based approach to evaluate user perceptions.

• Participants: Five employees from a multinational corporation in the Philippines, all recruited from a Korean language class. All participants were software engineers.
• Instrument: A survey deployed via Qualtrics containing both single-choice and open-ended questions.
• Data Collection Focus:
  • Frequency of usage and encounter rates with general vs. work-related scenarios.
  • Perceived impact of these scenarios on achieving professional fluency.
  • Specific suggestions for lesson scenarios that would bridge the fluency gap.
• Theoretical Framework: The study is grounded in Intelligent Tutoring Systems (ITS) architecture (Expert Knowledge, Student Model, Tutoring modules) and Second Language Acquisition (SLA) theories, specifically Krashen's Input Hypothesis and Hymes' Communicative Competence Theory (emphasizing context and social/professional appropriateness over mere grammatical accuracy).

3. Key Contributions

The paper contributes to the field of Human-Computer Interaction (HCI) and Educational Technology in three main ways:

1. User-Centered Evaluation of LLM Lessons: It provides a preliminary evaluation of Duolingo's AI-generated content from the specific perspective of professional learners, highlighting the limitations of current "one-size-fits-all" generation.
2. Definition of Professional Fluency: It operationalizes "professional fluency" not just as vocabulary expansion, but as the ability to navigate domain-specific discourse and technical contexts.
3. Design Considerations for Adaptive ITS: It proposes a shift from static personalization (adjusting difficulty via "Birdbrain") to content personalization, where lesson scenarios adapt to the user's specific professional background and goals.

4. Key Results & Findings

The study yielded four primary findings based on the five respondents:

• Foundational Value of General Scenarios: Participants found general, non-work scenarios (e.g., daily conversations) highly effective for building foundational grammar, vocabulary, and cultural knowledge. These were relatable and frequent, serving as a necessary entry point for novices.
• The "Fluency Gap": Work-related scenarios were encountered much less frequently than general ones. One respondent reported never encountering them. Those who did encounter them felt these scenarios were crucial for bridging the gap between novice and professional fluency.
• Contextual Relevance: A software engineer participant noted the absence of technical jargon (e.g., CI/CD pipelines, user interfaces, defect tracking) in the lessons, which widened their perceived learning gap for professional settings.
• Divergent Needs & Desire for Personalization: When asked for suggestions, participants proposed highly divergent topics (e.g., negotiating deadlines vs. traveling vs. more fundamentals). This indicates that a single generic curriculum cannot serve all professionals; the system must leverage individual backgrounds to generate personalized, domain-specific scenarios.
• Learning Stage Awareness: Participants acknowledged that work-specific scenarios are less applicable at the beginner level, suggesting a need for a staged approach where foundational general scenarios precede specialized professional content.

5. Significance and Future Work

Significance:
The paper argues that for LLMs to be truly effective in professional language training, they must move beyond generating generic "real-world" scenarios. The significance lies in the proposal to integrate domain-specific context into the "Expert Knowledge Module" of Intelligent Tutoring Systems. This aligns with Communicative Competence Theory, which posits that language proficiency is defined by the ability to communicate appropriately within specific social and professional contexts, not just grammatical correctness.

Future Work:
The authors outline a roadmap for a more rigorous experimental study:

1. Fine-tuning an LLM: Developing a model specifically trained to generate lessons applicable to the technology industry.
2. Long-term Between-Subjects Study: Conducting a controlled experiment with software engineers comparing three groups:
   • Control Group (No ITS).
   • Standard Duolingo Group.
   • Experimental Group (Using the fine-tuned, domain-specific LLM).
3. Metrics: Comparing language test scores and reported user experiences to validate the efficacy of personalized, domain-specific content.

Conclusion:
The authors envision an agentic Intelligent Language Tutoring System that dynamically adapts to a user's changing background, goals, and professional environment, creating a unique learning path for every individual learner rather than a static curriculum.