Tracing GenAI Literacy: Uncovering Student-AI Interaction Patterns in Academic Writing through Epistemic Network Analysis
This study utilizes Learning Analytics and Epistemic Network Analysis on interaction logs from 162 students to demonstrate that GenAI literacy can be effectively characterized by distinct questioning patterns, such as iterative refinement versus direct commands, offering a data-driven alternative to traditional self-reported assessments.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a class of students how to write a research summary, but you've given them a super-smart, magical robot assistant (Generative AI) to help them. The big question isn't just if they can use the robot, but how they talk to it. Do they treat the robot like a vending machine that spits out answers, or do they treat it like a thinking partner they can debate and refine ideas with?
This paper is like a detective story where the researchers (Angxuan Chen and Jiyou Jia from Peking University) tried to solve the mystery of "AI Literacy" by watching how 162 university students actually interacted with the robot, rather than just asking them, "How good are you at using AI?"
Here is the breakdown of their investigation:
The Problem: The "Fake It Till You Make It" Trap
Usually, schools ask students to fill out a survey: "On a scale of 1 to 10, how good are you at using AI?" The researchers realized this is like asking a driver, "Do you know how to drive?" and taking their word for it. It doesn't tell you if they actually know how to merge onto the highway or if they just think they do.
The researchers wanted to see the actual driving, not just the driver's confidence. They wanted to see the "process" of how students used the AI while writing an abstract for a paper.
The Setup: A Strict Playground
To get honest data, the researchers set up a tricky game:
- The Task: Students had to write an academic abstract.
- The Tool: They used a custom platform connected to a powerful AI (DeepSeek).
- The Rules: The students couldn't just copy-paste the AI's work. They had to type short prompts (under 30 words), and the system forced them to keep talking to the AI to get the final result. This meant they had to keep the conversation going.
The Detective Tool: "Epistemic Network Analysis" (ENA)
To understand the students' behavior, the researchers used a special tool called Epistemic Network Analysis (ENA).
Think of ENA like a dance floor map.
- Instead of just counting how many times a student asked a question (like counting steps), ENA looks at the connections between moves.
- It draws lines between the things students did right after each other. For example, did they ask for a fact and then immediately ask for a correction? Or did they ask for a draft, then ask for a refinement, then ask for a clarification?
- This creates a visual "signature" or "fingerprint" of how their brain was working while talking to the AI.
The Findings: Two Very Different Dancers
When the researchers looked at the "dance floor maps," they saw two completely different styles of interaction based on the students' AI literacy levels:
1. The Low-Literacy Group (The "Vending Machine" Users)
- The Pattern: These students mostly treated the AI like a search engine or a vending machine.
- The Moves: They would say, "Write this for me" (Generation Command) and then, if the result was weird, they would ask, "Is this true?" (Fact Check).
- The Metaphor: Imagine someone ordering a burger, taking a bite, and if it tastes bad, immediately asking the cook, "Did you put the right ingredients in?" They are reacting to the output rather than guiding the process. They rely on the AI to do the heavy lifting and only check if the result is "real."
2. The High-Literacy Group (The "Co-Author" Users)
- The Pattern: These students treated the AI like a thinking partner or a co-author.
- The Moves: They would say, "Make this sentence sound more academic" (Improvement), then ask, "What does this term actually mean?" (Clarification), and then, "How should I ask you to get a better result next time?" (Meta-Command).
- The Metaphor: Imagine a chef and a sous-chef. The chef says, "Let's try seasoning this differently," then asks, "What does this spice do to the flavor?" and then, "How can we ask the spice rack for the perfect amount?" They are in a loop of Sense-Making → Refinement. They are constantly tweaking, questioning, and improving the work together.
The Conclusion: It's About the Conversation, Not the Score
The researchers found that these two groups didn't just have different scores on a test; they had fundamentally different ways of thinking while using the tool.
- Low Literacy looks like a transaction: "I give you a command, you give me a result."
- High Literacy looks like a collaboration: "I give you a thought, we refine it together, and I learn how to ask better questions."
The paper concludes that we can't just rely on surveys to see if someone is "AI literate." Instead, we can watch their "dance moves" (their interaction logs) to see if they are just pushing buttons or actually thinking critically. This helps educators understand that true AI literacy isn't about knowing the tool's name; it's about how you structure your conversation with it to learn and create.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.