Benefits and Challenges of Integrating a Generative AI Assisted Reading Guide in an Undergraduate Journal Club Assignment

This study evaluates the implementation of a ChatGPT-assisted reading guide in an undergraduate chemistry journal club, finding that while it effectively enhanced students' comprehension and class discussions, it also raised concerns regarding AI accuracy and its potential impact on independent learning.

The Gist

Imagine you are handed a dense, complex instruction manual for building a futuristic spaceship. The language is full of alien jargon, the diagrams are confusing, and if you try to read it alone, you might get frustrated and give up. This is what reading scientific research papers feels like for many undergraduate students.

This paper is about a study where two professors tried a new "co-pilot" system to help students navigate these difficult manuals. They tested whether using Generative AI (like ChatGPT) as a guided reading partner could help students understand science better and discuss it more effectively in class.

Here is the breakdown of their experiment, the results, and the lessons learned, using some everyday analogies.

The Problem: The "Alien Manual"

Scientific papers are notoriously hard to read. They are like that spaceship manual: full of specialized words (jargon) and complex data that don't make sense without a guide.

• The Old Way: When students got stuck, they would usually just Google a word, skip the confusing part, or stare at the page until they felt anxious. It was like trying to assemble furniture without the instructions, just guessing where the screws go.
• The Result: Students often felt lost, didn't understand the main points, and their class discussions were quiet or went off-track because no one really "got" the paper.

The Solution: The "AI Reading Guide"

The professors created a specific set of rules (a "reading guide") for how to use an AI chatbot. They didn't just say, "Ask the AI anything." They gave students a step-by-step recipe:

1. Preview: Look at the title and pictures first (like scanning the furniture box to see what the final product looks like).
2. Read & Highlight: Read the text and mark confusing spots.
3. The AI Co-Pilot: Instead of just asking "What does this word mean?", students were taught to introduce themselves to the AI (e.g., "I'm a 3rd-year student who knows basic chemistry...") and then ask specific questions about the confusing parts.
4. Summarize: After reading, write a short summary of what they learned.

The Experiment: Two Weeks, Two Approaches

The class did the same "Journal Club" assignment (reading a paper and discussing it) twice:

• Week 1 (The Solo Run): Students read the paper on their own, using whatever tools they wanted (mostly Google).
• Week 2 (The AI Co-Pilot): Students used the new AI reading guide.

What Happened? The Good, The Bad, and The Skeptical

1. The Good: The "Lightbulb" Moment

When students used the AI guide, the class discussions changed dramatically.

• Analogy: In Week 1, the discussion was like a group of people trying to drive a car with the windows fogged up; they were arguing about which way to turn because they couldn't see. In Week 2, the AI wiped the windows clean. Everyone could see the road.
• The Result: Students understood the "big picture" better. They asked smarter questions, and the conversation lasted longer and was more lively. One student noted that without the AI, they spent the whole time just trying to figure out what the words meant, but with the AI, they could discuss the ideas.

2. The Bad: The "Cheat Code" Trap

Not everyone used the tool correctly. Some students treated the AI like a "magic summary button."

• Analogy: Instead of learning to cook the meal, they just asked the AI to order takeout and tell them what was in the box.
• The Issue: Some students asked the AI to summarize the entire paper for them. This is dangerous because if you let the AI do the reading for you, you aren't actually learning how to read. The professors warned that this is like outsourcing your brain—it's efficient for the moment, but you don't get stronger.

3. The Skeptical: The "Trust Issues"

Interestingly, as students got more comfortable with AI, they also got more suspicious of it.

• The Shift: In a previous study (about a year and a half earlier), students loved the AI and trusted it blindly. In this study, many students raised red flags.
• The Concerns:
  • "Hallucinations": Students worried the AI might make up facts (like a friend who tells a great story but gets the details wrong).
  • "The Crutch": One student called the AI a "crutch," saying it helped them get through this assignment but didn't teach them how to read future papers on their own.
  • Over-reliance: Students worried that if they always have a co-pilot, they might forget how to fly the plane themselves.

The Takeaway: A Balanced Diet

The study concludes that AI is a powerful tool, but it needs to be used with a "safety harness."

• It works: It helps students break down the "alien language" of science and makes class discussions much richer.
• But it needs rules: Teachers need to tell students, "Use the AI to clarify, not to replace." You still have to do the heavy lifting of reading the paper.
• Critical Thinking is Key: The most important result wasn't just that students understood the paper better, but that they started thinking critically about the AI itself. They realized that while AI is helpful, it isn't perfect, and they need to be the ones in charge of their own learning.

In short: The AI reading guide is like a pair of glasses that helps students see the text clearly. But if they wear the glasses so much that they never try to read without them, they might forget how to see on their own. The goal is to use the glasses to learn how to read, so eventually, they can read the manual even when the glasses are off.

Technical Summary

1. Problem Statement

Undergraduate STEM students frequently struggle with reading primary scientific literature due to unique formatting, specialized jargon, and the complexity of interpreting data and figures. This difficulty creates a disconnect between students' perceived ability to read literature and their actual comprehension, often leading to academic anxiety and reduced persistence in STEM fields. While active reading strategies (e.g., "figure facts," scaffolded worksheets) have shown success, they do not fully address the barrier of unfamiliar language. Furthermore, students often skip or skim complex sections, leading to poor learning outcomes. The authors sought to determine if integrating Generative AI (GenAI) with established active reading strategies could bridge this gap, while also investigating the potential risks of over-reliance on AI.

2. Methodology

The study employed a comparative, mixed-methods approach within a second-year chemistry seminar course (N=29) at Cal Poly San Luis Obispo.

• Experimental Design: The course included two "journal club" iterations:
  1. Control Condition: Students read an assigned paper independently using traditional methods (e.g., internet search engines) and engaged in small-group discussions.
  2. Intervention Condition: Students used a ChatGPT-assisted reading guide to read a different assigned paper.
• The Intervention (Reading Guide): The guide combined active reading strategies (adapted from McGuire's Teach Students How to Learn) with specific protocols for interacting with Large Language Models (LLMs). The four-step process included:
  1. Previewing: Analyzing titles, abstracts, and figures.
  2. Guided Reading: Highlighting and annotating text.
  3. AI Clarification: Using ChatGPT to define terms and explain concepts. Crucially, students were instructed to provide context (their academic background and the paper's title) to the AI to tailor responses.
  4. Post-reading Summary: Synthesizing notes and AI transcripts.
• Data Collection:
  • ChatGPT Transcripts: 21 anonymized transcripts were collected and coded using a hybrid scheme (initial themes from a prior study + emergent open coding).
  • Student Surveys: 25 students completed surveys regarding their reading habits, AI usage, and perceptions of the intervention.
  • Instructor Observations: Qualitative notes on the quality and duration of in-class group discussions.

3. Key Contributions

• Development of a Structured AI Protocol: The paper operationalizes a specific pedagogical framework that moves beyond "using AI" to "using AI correctly." It emphasizes providing context to the AI and using it for clarification rather than content generation.
• Taxonomy of Student-AI Interactions: The study categorizes how students interact with GenAI in a scientific reading context, identifying both productive and non-productive behaviors.
• Longitudinal Comparison: By comparing results with an initial study conducted 1.5 years prior, the authors highlight how student attitudes and behaviors toward GenAI have evolved as the technology has become more ubiquitous.

4. Results

A. Analysis of ChatGPT Transcripts (Interaction Themes)

• Productive Interactions:
  • Context Setting: 95% of students provided background information, a key factor for productive AI interaction.
  • Definitions & Explanations: 71% of students asked for definitions of jargon and follow-up explanations to deepen understanding.
  • Emergent Themes: Students began asking "big picture" questions about the application of the technology (19%) and requesting explanations of figures (14%), indicating higher-order engagement.
• Non-Productive Interactions:
  • Summarization: 33% of students asked for summaries of the entire paper. The authors note this is a risk, as students may use AI to bypass reading the actual text ("outsourcing thinking").

B. Student Survey Findings

• Perceived Benefits: 88% of students intended to use ChatGPT for future reading. The primary benefits cited were obtaining definitions (8 responses), clarity of explanations, and contextualizing information to their specific background.
• Perceived Challenges:
  • Accuracy: 5 students expressed skepticism regarding the accuracy of AI (hallucinations).
  • Learning Impact: 3 students felt AI acted as a "crutch" that hindered long-term learning skills.
  • Prompting: 8 students struggled to get the desired response from the AI.
• Discussion Quality:
  • Instructor Observation: Discussions in the AI-assisted week were more lively, sustained, and balanced compared to the control week, where discussions often lagged after 20 minutes.
  • Student Feedback: 13 students felt discussions were better with ChatGPT because everyone arrived with a baseline understanding. Conversely, 5 students felt the non-AI week forced deeper individual reading, leading to better preparation.

C. Comparison to Initial Study

• Shift in Behavior: Unlike the initial study, current students frequently used AI to summarize entire papers, suggesting a shift toward using AI as a substitute for reading rather than a supplement.
• Shift in Attitude: While the initial study showed unbridled optimism, the current study revealed a more nuanced, critical view. Students now express concerns about AI accuracy, environmental impact, and the ethical implications of over-reliance.

5. Significance and Recommendations

The study demonstrates that a structured, guided approach to integrating GenAI can significantly improve undergraduate engagement with scientific literature and enhance peer-to-peer discussion quality. However, the technology is a double-edged sword; without strict pedagogical guardrails, students may use it to avoid the cognitive struggle necessary for deep learning.

Key Recommendations for Educators:

1. Explicit Instruction: Faculty must explicitly teach students how to use AI for clarification (defining terms, explaining concepts) rather than substitution (summarizing entire texts).
2. Critical Evaluation: Curricula should include discussions on the limitations of GenAI, including the risk of hallucinations and the importance of verifying information.
3. Ethical Frameworks: Instructors should address the ethical dimensions of AI use, fostering a nuanced understanding of how these tools affect critical thinking and creativity.
4. Contextual Prompting: Students should be trained to provide sufficient context (background, paper details) to the AI to ensure high-quality, relevant responses.

In conclusion, the ChatGPT-assisted reading guide is a viable pedagogical tool for STEM education, provided it is implemented with a focus on active learning and critical evaluation of the AI's output.