💻 computer science

Engineering Students' Usage and Perceptions of GitHub Copilot in Open-Source Projects

This paper presents an exploratory survey-based study investigating how engineering students utilize and perceive GitHub Copilot's various features while contributing to open-source projects, revealing that usage patterns and perceived usefulness are significantly influenced by gender, programming proficiency, and prior familiarity with AI.

Original authors: Neha Rani, Jeevan Ram Munnangi, Austin Matthew Spangler, Donald Honeycutt

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Neha Rani, Jeevan Ram Munnangi, Austin Matthew Spangler, Donald Honeycutt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a class of young chefs how to cook a complex meal. In the past, they had to chop every vegetable and measure every spice from scratch. But now, you've given them a super-smart sous-chef (let's call him "Copilot") who can whisper recipes, chop vegetables instantly, or even write the whole recipe if you just tell him what you want.

This paper is like a report card on how a group of engineering students (our young chefs) used this digital sous-chef while trying to fix a bug in a real, open-source software project (their "cooking challenge").

Here is the breakdown of what the researchers found, using simple analogies:

1. The Setup: The "Take-Home" Cooking Challenge

The researchers gave 179 engineering students a task: add a new feature to a piece of software called "JSON Crack." It was like asking them to add a new room to a house they had never seen before. They were allowed to use GitHub Copilot however they wanted.

Before they started, the teachers gave them a quick tour of Copilot's tools:

Autofill: Like a smart autocomplete on your phone that guesses the next word.
Chat: A conversation where you ask, "How do I fix this?" and it answers.
Code Generation: You describe a function in plain English, and it writes the code.
Explain/Fix/Doc: Tools that explain what code does, fix errors, or write instructions.

2. What Did the Students Actually Do? (The Usage)

The researchers looked at which tools the students reached for most.

The Favorites: The students loved the Chat and Code Generation features the most. It's like how we prefer talking to a friend for advice rather than reading a manual. They felt comfortable asking, "Hey, how do I do this?" and getting an instant answer.
The Least Used: The Documentation tool (writing instructions for others) was the least popular. Why? Because the students were focused on just getting the job done. They were like chefs rushing to finish a dish for a party; they didn't have time to write a fancy recipe card for the next person.

3. The "Trust" Factor: Who Believed the Robot?

The study asked: Do students trust this AI?

Experience Matters: Students who already knew a bit about coding or had used AI before trusted the tool more. It's like a veteran driver trusting a GPS more than a new driver, who might be scared to follow the robot's directions.
The "Faith" Gap: Students with medium-level skills had the most faith in the AI. Beginners were too scared to rely on it, and experts were too critical to trust it blindly. The "middle ground" students were the ones saying, "Yes, robot, you're the boss!"

4. The Gender Divide: A Tale of Two Groups

This was one of the most interesting findings. The researchers noticed that male and female students used the tool differently:

The "Autofill" vs. "Generation" Split: Female students used the Autofill feature (the gentle nudge) more often. Male students used the Code Generation feature (the heavy lifter that writes whole blocks of code) more often.
Confidence Levels: Male students generally reported that the AI was more useful to them than female students did.
The Analogy: Imagine a group of people learning to drive. The study suggests that one group (males) felt more comfortable taking the wheel and letting the car drive itself (generating code), while the other group (females) preferred to keep their hands on the wheel and just use the car's sensors to help them park (autofill). The researchers suggest this might be because female students felt they had less prior experience with AI, making them more cautious.

5. Why Does This Matter?

The researchers are worried about two things:

Learning vs. Leaning: If students lean too heavily on the AI, they might stop learning how to cook (code) from scratch. They might become great at ordering takeout but terrible at making a meal from scratch.
The Gap: If the tool feels more useful to one group (men) than another (women), it could widen the gap in who becomes a confident software engineer.

The Bottom Line

The paper concludes that AI coding assistants are here to stay, and students are using them heavily, especially the "Chat" and "Generation" features. However, we need to be careful. Teachers need to design assignments that make sure students are still learning the concepts, not just copying the robot's answers.

Also, we need to make sure the "robot sous-chef" feels equally helpful and trustworthy to everyone, regardless of their gender or how much experience they have, so that no one gets left behind in the kitchen.

1. Problem Statement

The rapid evolution of Large Language Models (LLMs) has introduced AI coding assistants (e.g., GitHub Copilot) into software development workflows. While these tools are known to improve developer efficiency and code quality in professional settings, their impact on computer science students remains underexplored.

The Gap: Existing research focuses heavily on task-level outcomes (efficiency, output) or introductory classroom settings with specific, constrained tasks. There is a lack of understanding regarding how students use these tools in real-world, unconstrained open-source contribution scenarios.
The Concern: Educators need to understand how students interact with AI to design effective pedagogical strategies. Key questions include: Which features do students use? How do they perceive the utility of these features? Do demographic factors (gender) and prior experience (proficiency, familiarity) influence usage patterns and trust?

2. Methodology

The authors conducted an exploratory, quasi-experimental, survey-based study integrated into a Software Engineering course at a large public US university.

Participants: 179 engineering students (130 male, 47 female, 2 others) enrolled in a software engineering course. The cohort primarily consisted of Computer Science and Computer Engineering majors, with varying levels of programming proficiency (self-rated mean 2.33/5) and familiarity with C++ (mean 3.84/5).
Task: Students were assigned a take-home open-source contribution task: adding node editing functionality to an existing, unfamiliar codebase (an interactive JSON visualization tool called "JSON Crack") using Visual Studio Code (VS Code).
Intervention:
- Students received an in-person demo of GitHub Copilot features.
- They were free to use any Copilot feature (Code autofill, Code generation, Copilot Chat, Code explanation, Error fixing, Documentation generation) as they saw fit.
Data Collection:
- Survey: Post-assignment survey collecting demographic data, prior experience (C++, LLMs, AI tools), usage frequency of specific Copilot features (1–5 Likert scale), perceived usefulness (1–5 Likert scale), and trust metrics based on the Madsen-Gregor Human-Machine Trust Scale (reliability, understandability, technical competence, faith, personal attachment).
Analysis:
- Descriptive statistics for usage and perception.
- Independent T-tests (or Mann-Whitney U tests for non-normal data) to analyze gender differences.
- Chi-square tests to analyze the impact of programming proficiency and prior familiarity on usage and trust.

3. Key Contributions

Contextual Insight: Moves beyond introductory "toy" problems to analyze AI usage in a complex, open-source contribution scenario, bridging the gap between academic exercises and industry practices.
Granular Feature Analysis: Differentiates usage and perception across specific GitHub Copilot features (e.g., Chat vs. Code Generation vs. Documentation), rather than treating "AI" as a monolith.
Demographic & Experience Correlation: Provides empirical evidence on how gender, programming proficiency, and prior AI familiarity significantly alter how students interact with and trust AI tools.

4. Key Results

A. Feature Usage and Perceived Usefulness

Most Used: Copilot Chat (Mean usage: 4.02) and Code Generation (Mean usage: 3.92) were the dominant features.
Least Used: Documentation Generation (/doc) was the least used (Mean: 1.77), indicating students prioritize functional code completion over documentation in this context.
Perceived Usefulness: Chat and Code Generation were rated highest for usefulness (Mean > 3.8). Documentation was rated lowest (Mean: 2.05).

B. Impact of Gender

Familiarity: Male students reported significantly higher familiarity with LLMs than female students ( $p=0.059$ , near-significant).
Usage Patterns:
- Code Autofill: Females used this significantly more than males ( $p=0.008$ ).
- Code Generation: Males used this significantly more than females ( $p=0.02$ ).
- Code Explanation: Females used this more than males (near-significant, $p=0.056$ ).
Perceived Usefulness:
- Males rated the overall usefulness of AI coding assistants significantly higher than females ( $p=0.002$ ).
- Females found Code Autofill and Code Explanation more useful than males did.
- Males found Code Generation more useful than females did.

C. Impact of Programming Proficiency & Familiarity

Proficiency: Students with medium to high programming proficiency used the Copilot Chat feature significantly more than those with low proficiency. Low-proficiency students tended to prompt less, likely due to an inability to identify necessary fixes in generated code.
Trust: Higher proficiency correlated with "median" faith in the tool, while medium proficiency correlated with "somewhat higher" faith.
Familiarity: Prior familiarity with AI coding assistants significantly impacted trust constructs (understandability, personal attachment) and the usage of the /fix feature. Students familiar with AI showed higher trust and attachment.

5. Significance and Implications

Educational Design: Instructors should recognize that students do not use AI uniformly. The preference for Chat and Code Generation suggests students view AI as a conversational partner and a code synthesizer.
Addressing Genders Gaps: The study highlights a potential gender disparity in AI familiarity and perceived usefulness. The authors suggest that targeted training and exposure to AI systems could help bridge this gap, ensuring female students do not feel alienated or less confident in using these tools.
Pedagogical Shift: The low usage of documentation features suggests students focus on "getting code to work" rather than holistic software engineering practices. Educators may need to explicitly incentivize documentation generation in assignments.
Risk of Over/Under-reliance: The findings suggest that low-proficiency students may under-utilize AI (missing out on efficiency gains) or over-rely on it without understanding the output, while high-proficiency students use it more strategically (iterative prompting).

Conclusion

The paper concludes that while GitHub Copilot is widely adopted by engineering students, its usage is highly heterogeneous. Usage patterns are strongly influenced by gender and prior experience. Understanding these nuances is critical for developing responsible AI-integrated curricula that foster appropriate reliance, trust, and skill development among future software engineers.