The Agency Gap in AI-Supported Writing: How Reactive and Proactive Agent Designs Shape Multimodal Reasoning

Technical Summary: The Agency Gap in AI-Supported Writing

Problem Statement
The integration of Generative AI (GenAI) into academic writing introduces a critical design challenge: the "agency gap." This concept describes a potential mismatch between the initiative demanded by an AI agent (e.g., reactive vs. proactive) and a learner's capacity to initiate, monitor, and evaluate AI-supported reasoning. While GenAI tools offer scaffolding opportunities, their educational value depends on how control is shared. Reactive agents require learners to possess high prompting skills and metacognitive awareness, potentially disadvantaging those with lower GenAI literacy. Conversely, proactive agents that provide sequenced questions and feedback may reduce cognitive load but risk over-structuring learning and diminishing learner agency. Existing research lacks empirical evidence on how these interaction designs influence epistemic reasoning, feedback uptake, and immediate independent performance in multimodal analytical writing tasks.

Methodology
The study employed a two-phase, mixed-method experimental design with 79 medical and nursing students. Participants completed two multimodal analytical writing tasks based on healthcare simulation data visualizations (a bar chart, a communication network, and a ward map).

• Experimental Conditions: Participants were randomly assigned to one of two AI agent designs:
  • Reactive Agent: Responded only when prompted by the learner, requiring the student to identify uncertainties, formulate prompts, and sequence the inquiry.
  • Proactive Agent: Initiated a fixed sequence of guiding questions and provided task- or process-level feedback based on expert-authored scaffolds, steering the learner through visual interpretation without generating the final text.
• Measures:
  • GenAI Literacy: Assessed using the validated 20-item Generative AI Literacy Assessment Test (GLAT).
  • Writing Performance: Evaluated via an ordinal rubric covering five dimensions: Insightfulness, Visual Data Integration, Organisation and Coherence, Linguistic Quality, and Critical Thinking.
  • Epistemic Reasoning: Analyzed using Epistemic Network Analysis (ENA) to map the relational structure of reasoning and engagement (based on ICAP and argumentation frameworks) within human-AI dialogues.
• Analytical Approach: The study utilized ordinal logistic regression to predict independent writing performance (after AI removal) based on literacy and design. Mediation analysis tested whether AI-supported performance mediated the relationship between literacy and independent performance. Thematic analysis of learner reflections provided qualitative insights into the experience of agency.

Key Results

• Epistemic Reasoning (RQ1): ENA revealed distinct relational structures between conditions. Proactive interactions fostered stronger links between conceptual reasoning, adequate evidence application, and constructive engagement. In contrast, reactive interactions were characterized by stronger associations between factual descriptions and off-task or procedural exchanges. Proactive scaffolding appeared to externalize planning and monitoring functions, directing attention from observations to synthesis.
• GenAI Literacy and Performance (RQ2): GenAI literacy significantly predicted immediate independent performance in Visual Data Integration, Critical Thinking, and overall Composite Score after AI support was removed. However, literacy did not significantly predict performance in Insightfulness, Organisation, or Linguistic Quality. Crucially, the interaction between GenAI literacy and agent design was not statistically significant; the study did not find evidence that one design produced higher scores or different literacy slopes than the other.
• Mediation Effects (RQ3): Mediation analysis showed a significant total association between GenAI literacy and independent performance in the reactive condition, but not in the proactive condition. However, indirect effects were not statistically significant in either condition, and the confidence intervals were wide. The results suggest that while proactive support may reduce observable literacy-based performance differences, the study does not establish a compensatory causal effect or confirm mediation.
• Learner Experience (RQ4): Qualitative analysis identified three design heuristics: (1) sustaining autonomy through contextual and confirmatory feedback (reactive); (2) promoting integrative reasoning through dialogic scaffolding (proactive); and (3) ensuring equity by aligning initiative with perceived learner expertise and task complexity. Learners valued proactive guidance for connecting evidence but found it redundant for simple tasks or when they felt competent.

Key Contributions
The paper's primary theoretical contribution is the conceptualization of the "agency gap" as a relational mismatch between AI agent initiative and learner capability, rather than an individual deficit. It positions the distribution of initiative as a design-level boundary condition that shapes the allocation of epistemic work.

• Process vs. Outcome: The study demonstrates that interaction design significantly alters the process of reasoning (the network structure of dialogue) even when direct effects on immediate outcomes (writing scores) are not statistically significant.
• Design Implications: The findings suggest that effective AI writing support requires contextual feedback, dialogic scaffolding, and the calibration of initiative to learner needs. Proactive designs can deepen conceptual integration, while reactive designs preserve learner control but demand higher literacy.

Significance and Claims
The authors claim modest significance, emphasizing that the "agency gap" is an emerging explanatory framework rather than a validated general theory. The study does not claim that proactive agents universally outperform reactive ones, nor does it assert that proactive scaffolding definitively compensates for low literacy. Instead, the findings support the hypothesis that interaction design reorganizes the epistemic work of learning. The practical implication is that educational AI agents should offer adjustable levels of initiative, allowing learners to request guidance or reduce scaffolding as they demonstrate competence. The authors caution that these results are specific to short-term, multimodal evidence-synthesis tasks and call for future research with larger samples, direct measures of agency, and longitudinal assessments to test the durability of these effects.