To Vibe Research or Not to Vibe Research? Generative AI in Qualitative Research
This paper synthesizes the ongoing debate regarding the suitability of generative AI in qualitative research, highlighting how factors such as research philosophy (small-q vs. Big Q), skills, ethics, and personal preferences influence software engineering researchers' decisions to adopt these tools.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Question: Can a Robot Do the "Human" Work of Research?
Imagine a group of researchers trying to figure out how to use a powerful new tool: Generative AI (like the chatbots you might use today). They are arguing about whether this tool is a helpful assistant or a dangerous replacement for human researchers, especially when studying people, feelings, and social issues (which is called Qualitative Research).
The authors of this paper, who are experts in software engineering, want to help their fellow researchers decide when it's okay to use AI and when it's not. They say the answer depends entirely on what kind of research you are doing.
To explain this, they split research into two distinct "flavors": Small-q and Big-Q.
Flavor 1: Small-q Research (The "Recipe" Approach)
The Philosophy: This approach is like following a strict recipe or a scientific formula. The goal is to find universal truths, test hypotheses, and get results that can be repeated by anyone else. It values objectivity (removing human feelings) and consistency.
- The Analogy: Imagine you are baking 1,000 identical cakes. You want to make sure every single cake tastes exactly the same. You use a precise scale, a timer, and a standardized oven. If a cake comes out slightly different, you discard it as a "mistake."
- Can AI help here? Yes.
- AI is great at spotting patterns and following rules.
- If you use AI to analyze data in this type of research, you can treat the AI like a second baker. You can check if the AI and a human baker agree on the results (this is called "intercoder reliability").
- If they agree, the research is solid. The AI acts as a fast, efficient assistant that helps you bake more cakes quickly without losing the "recipe."
Flavor 2: Big-Q Research (The "Vibe" Approach)
The Philosophy: This approach is less about recipes and more about exploring the unknown. The goal is to uncover surprising, unique, or weird phenomena in human life. It values subjectivity (the researcher's own feelings and perspective) as a tool, not a bug.
- The Analogy: Imagine you are a jazz musician improvising a solo. You aren't following a sheet music script. You are listening to the other musicians, feeling the mood of the room, and reacting in the moment. The "truth" isn't a fixed note; it's the unique connection between the players. If you try to use a robot to play the solo, it might hit the right notes, but it won't feel the music.
- Can AI help here? Probably Not (or at least, not easily).
- The authors argue that AI cannot "vibe." It doesn't have feelings, it doesn't have a personal history, and it can't truly understand the human experience.
- In Big-Q research, the connection between the researcher and the person being studied is the most important part. If you put an AI in the middle of that conversation (like a translator or a middleman), you break the connection.
- AI is built on statistics and patterns, not on human intuition. Using it here is like trying to use a calculator to write a poem about heartbreak; it might look like a poem, but it lacks the soul.
The "Positivism Creep" Problem
The paper warns of a danger called "Positivism Creep."
- The Metaphor: Imagine a group of jazz musicians (Big-Q researchers) who start trying to measure their improvisation using a ruler and a stopwatch (Small-Q tools) because they think that's what "real" science looks like.
- The Result: They stop playing jazz and start playing a rigid, boring march. They lose the very thing that made their research special.
- The Warning: Many software engineers and AI developers are used to the "Small-q" world (measuring, counting, testing). When they build AI tools for researchers, they often try to force Big-Q research into Small-Q boxes (e.g., asking an AI to prove it's "reliable" by counting how often it agrees with itself). The authors say this is a mistake because it destroys the integrity of the research.
The Human Cost (The "Dark Side" of the Tool)
The paper also points out that using these AI tools isn't just about research quality; it's about ethics.
- The Analogy: Imagine a super-fast delivery service that gets your package in an hour. But, to make that speed possible, the workers in the warehouse are overworked, underpaid, and exposed to dangerous conditions, and the trucks are polluting the air.
- The Reality: The authors note that the companies making these AI tools often rely on exploiting workers (sometimes in poorer countries) and cause environmental harm. They argue that individual researchers shouldn't just feel guilty; the whole system needs to change.
The Final Verdict: What Should You Do?
The authors give a clear recommendation based on their "Vibe" analogy:
If you are doing Big-Q Research (The Jazz/Improvisation):
- Don't use AI for the core analysis. Unless you are a triple-expert (a philosopher, a qualitative research master, and an AI engineer), you should stick to doing the work yourself.
- Why? The joy and the learning come from the human struggle of finding patterns and understanding people. If you let the AI do it, you might get a report faster, but you lose the deep understanding of the world.
If you are doing Small-q Research (The Recipe/Baking):
- AI is okay to use. It can help you process data faster. Just make sure you check its work carefully to ensure it's following the rules.
The Big Picture:
- The paper asks a deep question: Are we doing research just to write papers quickly, or are we doing it to truly understand the human experience?
- If the goal is just to produce text, AI is a perfect tool.
- If the goal is to understand, explain, and interpret the world, humans must remain in the driver's seat. We shouldn't let AI take away our ability to "vibe" with the data.
In short: AI is a great calculator, but it's a terrible jazz musician. If you need a calculation, use it. If you need to understand the human soul, you have to do the work yourself.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.