Auditing Student-AI Collaboration: A Case Study of Online Graduate CS Students

This mixed-methods study audits online graduate CS students' preferences and concerns regarding AI collaboration by comparing their actual usage and desired automation levels across academic tasks to identify gaps between current AI capabilities and student expectations, ultimately aiming to guide the development of more trustworthy educational AI systems.

The Gist

Imagine you've just hired a super-smart, incredibly fast, but occasionally daydreaming personal assistant named "AI." You want to use this assistant to help you with your schoolwork, but you have a nagging question: "How much of the work should I let them do, and how much should I keep doing myself?"

This paper is like a report card on that relationship. The author, Nifu Dan, went to a group of graduate computer science students (the "students") and asked them to audit their own relationship with AI. They wanted to see where the students' hopes for AI matched up with their reality of using it.

Here is the breakdown of what they found, using some everyday analogies:

1. The "Traffic Light" System

The researchers looked at 12 different school tasks (like writing an essay, fixing code, or making flashcards) and sorted them into four "traffic light" zones based on what students wanted vs. what they actually did:

• 🟢 Green Light (Go!): These are tasks where students want AI to do the heavy lifting, and they actually let it do it.
  • The Analogy: Think of formatting a bibliography or fixing grammar. It's like asking a robot to organize your bookshelf or proofread a grocery list. It's tedious, rules-based, and easy to check. Students happily let AI handle this.
• 🔴 Red Light (Stop!): These are tasks where students know AI could do it, but they refuse to let it.
  • The Analogy: This is like asking a robot to write your heart-felt apology letter to a friend. Even if the robot writes perfect words, you feel it lacks the "soul" or accountability. Students are wary of AI handling professional emails or deep emotional communication.
• 🟡 R&D Opportunity (Under Construction): This is the most interesting zone. Students really want AI to help here, but they don't trust it enough to use it fully yet.
  • The Analogy: Imagine you want a robot to plan your entire vacation or solve a complex math puzzle. You want the help, but you're afraid the robot will pick a hotel that doesn't exist or solve the math wrong. The technology isn't quite there to earn your full trust yet.
• ⚪ Low Priority (Not Worth It): Tasks where neither the students nor the AI seem interested.

2. The "Why" and the "Worry"

The study found that students have two main drivers, which act like a tug-of-war:

• The "I'm Busy" Driver: The biggest reason students use AI is to save time and reduce mental fatigue. It's like using a power drill instead of a screwdriver. If the task is boring or repetitive, they want the drill.
• The "Don't Get Me Fired" Driver: The biggest fear is hallucinations (AI making things up) and losing their own brain power.
  • The Analogy: If you use AI to write a research paper, you worry it might invent a fake scientist (a hallucination). If you use it to brainstorm ideas, you worry you'll forget how to think creatively yourself.

The Twist: The study found that students are smart about this. They don't just use AI everywhere.

• For writing and editing, they use AI like a spell-checker (low risk).
• For brainstorming and complex math, they are very cautious. They know that if the AI gets the idea wrong, it ruins the whole project.

3. What Students Want from AI (The "Fix-It" List)

When asked, "How can we make AI trustworthy enough for school?", students didn't say, "Make it smarter." They said, "Make it honest."

They want AI to act less like a "know-it-all" professor and more like a humble research assistant. Here is what they asked for:

• Show Your Work (Transparency): "Don't just give me the answer; show me the source."
  • Analogy: If a chef gives you a soup, they should show you the recipe and the fresh ingredients, not just say "Trust me, it's good." Students want clickable links to prove the AI isn't making things up.
• Admit When It's Guessing (Uncertainty): "If you aren't sure, tell me!"
  • Analogy: Imagine a GPS that says, "I'm 90% sure this is the way," instead of confidently driving you into a lake. Students want AI to say, "I'm not 100% sure about this fact," so they can double-check it.
• Let Me See the Thinking (Explainability): "Show me how you got there."
  • Analogy: Don't just hand me the finished puzzle; let me see the pieces you tried first. This helps students learn how to solve the problem, not just get the answer.

The Big Takeaway

The paper concludes that students aren't lazy; they are cautious. They want to use AI to be more efficient, but they are terrified of losing their own ability to think critically or getting caught with fake information.

The Golden Rule for AI in Education:
Don't try to replace the student. Instead, build an AI that acts like a transparent, honest co-pilot that shows its work, admits its mistakes, and lets the human student stay in the driver's seat.

Technical Summary

1. Problem Statement

As generative AI (GenAI) tools (e.g., ChatGPT, Claude, Gemini) become embedded in higher education, there is a critical lack of understanding regarding how students navigate the trade-offs between automation (efficiency) and agency (learning autonomy). While AI offers productivity gains, concerns persist regarding over-automation, diminished critical thinking, hallucinated outputs, and academic integrity.

The core problem addressed is the misalignment between:

1. Student Normative Expectations: What level of AI involvement students desire for specific academic tasks.
2. Current AI Capabilities/Usage: How students actually use AI and perceive its current reliability.
3. Design Gaps: The lack of system features that address student concerns (e.g., trust, verification) to facilitate safe and effective collaboration.

2. Methodology

The study employs a mixed-methods audit involving online graduate Computer Science students (OMSCS program at Georgia Tech).

• Participants: $N=44$ students (filtered from an initial pool of 57) who reported at least occasional use of AI for coursework.
• Framework: The study adapts the Human Agency Scale (HAS) and a Four-Zone Classification model to map the relationship between desired automation and actual usage.
• Task Scope: 12 distinct academic tasks were selected, ranging from routine (e.g., citation formatting) to complex cognitive tasks (e.g., debugging code, brainstorming).
• Data Collection (Two-Phase Survey):
  • Phase 1 (Quantitative): Participants rated 12 tasks on two 5-point Likert scales:
    1. Desired Automation: How much AI involvement they want (1: No involvement to 5: Fully autonomous).
    2. Actual Usage/Perceived Capability: How much they currently rely on AI for the task (1: No use to 5: Total reliance).
    • Additional Metrics: Primary reasons for use (e.g., saving time) and primary concerns (e.g., hallucinations, reduced critical thinking).
  • Phase 2 (Qualitative): A follow-up open-ended survey focused on the most prevalent concern identified in Phase 1 (inaccurate information/hallucinations). Participants described desired system features to mitigate these risks.
• Analysis:
  • Four-Zone Mapping: Tasks were categorized into:
    • Green Light: High desire, High usage (Safe to automate).
    • Red Light: Low desire, High usage (Ethical/Integrity risks).
    • R&D Opportunity: High desire, Low usage (Gap between want and capability).
    • Low Priority: Low desire, Low usage (Not suitable for AI).
  • Thematic Analysis: Open-ended responses were coded for recurring design principles.

3. Key Contributions

1. Empirical Audit of Student-AI Alignment: Provides a granular, task-specific map of where student desires for AI automation align or diverge from actual usage patterns in a technical graduate program.
2. Task-Dependent Risk Profiling: Demonstrates that student concerns and motivations are not uniform but are highly dependent on the nature of the task (e.g., procedural vs. generative vs. high-stakes reasoning).
3. Design Requirements for Trustworthy AI: Synthesizes student-driven design principles, moving beyond generic "trust" to specific mechanisms like verifiability, uncertainty signaling, and explainability.
4. Refinement of the Human Agency Scale: Successfully adapts the HAS framework from general human-AI interaction to the specific context of educational pedagogy.

4. Key Results

RQ1: Desired vs. Actual Automation

• General Trend: Desired automation levels consistently exceeded actual usage levels across most tasks, indicating a "conditional adoption" where students want AI help but hesitate to fully rely on it.
• Four-Zone Distribution:
  • Green Light (High Desire/High Use): Procedural and verifiable tasks such as citation formatting (T2), grammar revision (T4), summarizing notes (T11), and debugging code (T5). Students trust AI here because outputs are easily verified.
  • R&D Opportunity (High Desire/Low Use): Tasks requiring personalization or complex reasoning, such as study planning (T6), flashcard generation (T7), resource recommendation (T8), and step-by-step quantitative problem solving (T10). Students want help but lack confidence in AI's reliability for these specific contexts.
  • Red Light/Low Priority: Tasks involving professional email drafting (T9) showed moderate desire but low usage due to interpersonal/accountability concerns. Brainstorming (T3) showed high concern regarding reduced critical thinking, limiting adoption.

RQ2: Motivations and Concerns

• Motivations: "Saving time" and "Reducing cognitive load" were the dominant drivers across all tasks.
• Concerns by Task Type:
  • Writing/Revision: Concerns were minimal; "None" was the most common response. Students view AI as a pragmatic editor.
  • Ideation (Brainstorming): The dominant concern was "Reduced critical thinking," indicating students view idea generation as a core learning activity they do not want to outsource.
  • Technical/Quantitative: The dominant concern was "Inaccurate information" (hallucinations). Students fear that errors in code or math are harder to detect and have higher consequences than in writing.

RQ3: Design Expectations

To address the primary concern of inaccurate information, students proposed specific system features:

• Transparency & Verifiability: Mandatory clickable source citations and traceable data origins to allow independent verification.
• Explicit Uncertainty Communication: Systems should display confidence scores or percentages indicating the factual reliability of a response, rather than presenting answers with false confidence.
• Explainability: Visibility into the "inner thought process" or reasoning steps to allow students to intervene and critique the logic.
• Hallucination Signaling: Explicit flags or warnings for potentially hallucinated content.
• Human-in-the-Loop: Features that allow users to flag errors and provide feedback for real-time model adjustment.

5. Significance and Implications

• Pedagogical Design: The study suggests that AI integration should be task-specific. Educators can encourage AI use for "Green Light" tasks (editing, summarizing) while scaffolding "R&D Opportunity" tasks (problem solving) with strict verification protocols.
• System Development: AI developers for education must prioritize transparency over fluency. Features that mask uncertainty (e.g., confident tone) are counter-productive. Systems must be designed as cognitive partners that enable verification, not as autonomous agents that replace reasoning.
• Human Agency: The findings support a Human-Centered AI (HCAI) approach where the goal is not maximum automation, but augmented agency. Students are willing to use AI, but only if the system preserves their ability to verify, critique, and ultimately own the learning outcome.
• Future Directions: The paper calls for broader studies across diverse disciplines and the use of open-ended survey designs to capture more nuanced, context-dependent perspectives on AI collaboration.