Learning Through Dialogue: Engagement and Efficacy Matter More Than Explanations

1. Problem Statement

While Large Language Models (LLMs) are increasingly used as conversational partners for political information seeking, the specific interactional dynamics that drive learning remain understudied. Existing research often treats LLMs as static information retrieval tools or focuses solely on the quality of model outputs (explanations). However, learning is a bidirectional process where user engagement, reflection, and individual differences (such as political efficacy) play critical roles.

The study addresses three key gaps:

1. Lack of Paired Data: Most existing LLM datasets (e.g., InfinityChat, LMSYS-Chat-1M) lack paired pre-post learning outcomes and fine-grained engagement signals.
2. Mechanism Ambiguity: It is unclear how LLM explanations influence learning (mediation) or when they are most effective (moderation).
3. Interactional Dynamics: Current literature often overlooks the interplay between explanatory richness and user engagement behaviors (e.g., asking follow-up questions, expressing uncertainty).

2. Methodology

Experimental Design & System (Arthos)

The authors developed Arthos, a purpose-built experimental chatbot system designed to simulate real-world political information seeking while enabling rigorous measurement.

• Model: Meta's LLaMA-3.1-70B (Versatile) deployed via Groq's inference-only API to ensure data privacy (no training on user data).
• Configuration: 128k token context window; temperature set to 1.0.
• Role: The bot acted as a "tutoring assistant" using Retrieval-Augmented Generation (RAG). It was instructed to guide users through evaluating evidence and reasoning with provided news articles, without directly supplying final answers.
• Participants: 152 participants from the US (English/Spanish) and India (English).
• Procedure:
  1. Pre-survey: Measured baseline political knowledge and confidence.
  2. Dialogue: Multi-turn conversations on specific socio-political issues (e.g., crime, air quality).
  3. Post-survey: Re-measured knowledge and confidence.
• Dataset: 397 conversations (approx. 4,286 human turns, 5,154 bot turns).

Feature Extraction

Features were extracted from both human and bot turns using established lexicons (e.g., LIWC) and automated text analysis:

• LLM Explanatory Features: Syntactic complexity (sentence length, conjunctions), readability (Flesch–Kincaid), information density, and the Explanatory Ratio (proportion of response devoted to explanation).
• User Engagement Features: Linguistic markers for cognitive processing, including:
  • Insight/Reasoning: (INSIGHT, CAUSE)
  • Uncertainty/Tentativeness: (TENTAT, DISCREP)
  • Certainty/Differentiation: (CERTAIN, DIFFER)
  • These were aggregated into a composite Cognitive Engagement score via factor analysis.
• Controls: Demographics, political ideology, and Political Efficacy (perceived ability to understand/engage with politics).

Statistical Analysis

The study employed a combined mediation-moderation strategy:

• Mediation Analysis: Structural Equation Modeling (SEM) with bootstrapping to test if engagement processes (Insight, Cognitive Engagement) explain the link between LLM explanations and learning outcomes.
• Moderation Analysis: Multilevel Linear Mixed-Effects Models (random intercepts for participants) to test how individual differences (Political Efficacy) and interactional cues (Conversation Length, Help-seeking) condition the effects of LLM features.

3. Key Contributions

1. Interactional Perspective: Demonstrates that political learning from LLMs is an interactional achievement, not merely a result of high-quality explanations.
2. Process vs. Condition: Clarifies the process (mediation: how learning occurs via engagement) and the conditions (moderation: when learning is likely based on user efficacy).
3. Differentiation of Outcomes: Distinguishes between Knowledge Gain (factual understanding) and Confidence Change (self-assessment), showing they are driven by different mechanisms.

4. Key Results

A. Mediation Analysis (The "How")

• Confidence Gain: Partially mediated by User Insight.
  • Finding: LLM explanatory richness increases confidence both directly and indirectly by fostering reflective insight. Users who reflect on the content gain more confidence.
• Knowledge Gain: Fully mediated by Cognitive Engagement.
  • Finding: LLM explanations do not directly increase knowledge. Their effect is entirely dependent on the user's cognitive engagement (elaboration and effortful processing). Without active engagement, explanations do not lead to knowledge acquisition.

B. Moderation Analysis (The "When" and "For Whom")

• Confidence Dynamics:
  • High-Efficacy Users: Confidence gains depend on how they experience and resolve uncertainty. For these users, confusion/help-seeking acts positively predict confidence only if accompanied by high insight (active reflection).
  • Low-Efficacy Users: Confusion/help-seeking acts increased confidence primarily for those with low political efficacy, suggesting they rely more on external validation to feel confident.
• Knowledge Dynamics:
  • Conversation Length: Longer conversations benefit High-Efficacy users significantly, but offer limited gains for low-efficacy users.
  • Insight Interaction: Conversation length positively predicts knowledge gain only at high levels of user insight. Users with low insight do not benefit from extended dialogue, regardless of length.

5. Significance and Implications

• Redefining AI Education: The study challenges the assumption that "better explanations" automatically equal "better learning." Instead, learning is contingent on the alignment between the AI's explanatory behavior and the user's engagement state.
• Design Guidelines for Human-AI Systems:
  • AI systems should not just provide more information; they must scaffold engagement.
  • Systems should detect user states (e.g., confusion vs. reflection) and adapt. For example, extending a conversation is only beneficial if the user is actively reflecting; otherwise, it may be counterproductive.
  • Scaffolding: AI should encourage users to articulate uncertainty and reflect on evidence, particularly for users with lower political efficacy.
• Theoretical Contribution: Extends the "Computers as Social Actors" (CASA) paradigm by showing that LLMs function as conditional social partners. Their influence is not uniform but varies based on the user's capacity to interpret and sustain interaction.

Conclusion

The paper concludes that learning from LLMs is a collaborative, effortful process. Explanations support confidence and knowledge only when they elicit forms of engagement that users can sustain and interpret productively. Future conversational agents should be designed to align with user engagement states rather than simply maximizing output complexity.