What makes an Expert? Comparing Problem-solving Practices in Data Science Notebooks
By analyzing 440 Jupyter notebooks through multi-level sequence analysis, this study reveals that data science expertise is distinguished not by different phase transitions between novices and experts, but by experts' adoption of shorter, more iterative, and context-specific action sequences compared to novices' linear processes.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Question: What Makes a Data Scientist an "Expert"?
Imagine you are teaching two people how to bake a complex cake.
- The Novice follows the recipe exactly: Mix ingredients, bake, frost, serve. If the cake is dry, they might just add more frosting to hide it. They treat the recipe like a strict line of marching orders.
- The Expert also knows the recipe, but they taste the batter, adjust the sugar, realize the oven is too hot, lower the temperature, and maybe even decide to switch from a sponge cake to a cheesecake halfway through because they see the ingredients don't quite work together. They treat the recipe like a map, not a train track.
This paper asks: How can we actually see the difference between the Novice and the Expert when they are working on a Data Science problem?
In Data Science, people use tools called Jupyter Notebooks. Think of these as digital notebooks where you write code, run it, see the results, and write more code. It's like a chef's journal where they write down every ingredient they tried and every taste test they did.
The researchers looked at 440 of these notebooks (some from beginners, some from masters) to figure out what the "Expert" way of thinking actually looks like.
The Three Levels of Investigation
The researchers didn't just look at the final cake; they looked at the cooking process on three different levels, like zooming in with a camera.
1. The "Big Picture" View (The Whole Journey)
The Analogy: Imagine looking at a map of a road trip.
- The Novice's Map: They drive in a straight line from Point A to Point B. They stop at every single gas station and rest stop in order, even if they don't need them. It's a long, linear, "one-way street" journey.
- The Expert's Map: They take a shorter, winding route. They might drive from A to B, realize they missed a turn, loop back to A, take a different road, and then zoom to B. It's a series of short, quick loops.
The Finding: Experts don't necessarily visit different "cities" (stages of data science) than novices. They visit the same places (Importing Data, Exploring, Training Models, Visualizing). But, experts do it in a shorter, more looping way. They are willing to go back and forth quickly. Novices tend to march forward in a long, straight line, hoping they get it right the first time.
2. The "Traffic Flow" View (Moving Between Stages)
The Analogy: Imagine looking at traffic lights at an intersection.
- The Finding: When the researchers looked at how people moved from one stage to the next (e.g., from "Cleaning Data" to "Building a Model"), the traffic patterns were almost identical for both groups.
- The Takeaway: Experts don't have a secret "secret sauce" order of steps. They don't skip steps or do them in a magical new order. If you just looked at the order of their steps, you couldn't tell them apart from the beginners.
3. The "Micro-Moves" View (The Specific Actions)
The Analogy: This is like watching a chef's hands in slow motion.
- The Novice: They might chop an onion, then chop it again, then chop it again, just to be sure. They might load a file, look at it, load it again, and look at it again. They are "exploring" by repeating the same basic actions over and over.
- The Expert: They load the file, look at it, immediately tweak the model, check the result, and if it's wrong, they tweak the model again instantly. Their moves are efficient, specific, and purposeful. They don't waste time doing the same thing twice unless they have a very specific reason.
The Finding: This is where the real difference lies. Experts use efficient, tight loops of specific actions. Novices use long, repetitive, trial-and-error loops.
Why Does This Matter? (The "So What?")
The paper argues that we are teaching Data Science all wrong, especially now that Generative AI (like the tool you are talking to right now) is everywhere.
The Problem:
AI is great at writing code. If a student asks AI, "How do I clean this data?" the AI gives the answer.
- If the student is a Novice, they will just copy the answer, paste it, and move to the next step. They are following that long, straight line. They aren't learning how to think; they are just learning how to ask for answers.
- If the student is an Expert, they use the AI as a partner. They ask, "What if I try this?" and "Why did this fail?" They use the AI to speed up their loops, not to replace their thinking.
The Solution:
We need to stop grading students only on their final cake (the final code or model). We need to start grading their cooking process.
- Are they stuck in a long, straight line?
- Are they willing to loop back and try a different approach?
- Are they using AI to think strategically or just to finish the task?
The Bottom Line
Being an expert in Data Science isn't about knowing a secret list of steps. It's about how you move.
- Novices are like hikers following a trail marker, one step at a time, afraid to step off the path.
- Experts are like mountain climbers who know the terrain. They might climb up, realize the path is blocked, slide down, find a new route, and climb up again. They are flexible, they loop back, and they move efficiently.
The paper suggests that to create better experts (especially in an AI world), we need to teach students to be comfortable with getting lost, looping back, and iterating, rather than just trying to get the "right answer" on the first try.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.