Navigating the Prompt Space: Improving LLM Classification of Social Science Texts Through Prompt Engineering
This paper demonstrates that while systematically varying prompt engineering elements like label descriptions, instructional nudges, and few-shot examples can significantly improve LLM classification accuracy for social science texts, performance gains are often marginal beyond minimal context increases, can sometimes decline with excessive context, and vary substantially across models and tasks, necessitating individual validation rather than reliance on general rules.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you have a super-smart robot librarian who can read millions of books and sort them into shelves instantly. This is what Large Language Models (LLMs) are like for social scientists. They are being used to sort text—like sorting congressional bills into policy topics or deciding if a survey answer is "emotional" or "neutral."
For a while, researchers were just asking, "Can this robot do the job?" The answer was a shaky "maybe." But now, the question has changed to: "How do we give the robot the best instructions so it does the job perfectly?" This is called prompt engineering.
Think of the "prompt" as the instruction manual you hand the robot. The authors of this paper decided to play a massive game of "What if?" with these instruction manuals. They didn't just guess; they ran a huge experiment with two different robots (GPT 4o and Gemini 2.0 Flash) and two different sorting tasks:
- The Policy Sorter: Taking titles of 21 different types of government bills and sorting them into the right policy category.
- The Mood Sorter: Reading open-ended survey answers and deciding if they are "emotional" or "neutral."
They tweaked three specific parts of the instruction manual to see what worked best:
- Label Descriptions: Did they just give the robot a list of shelf names (e.g., "Health," "Education"), or did they also write a little paragraph explaining what goes on each shelf?
- Instructional Nudges: Did they add little notes to the robot saying, "Hey, be careful! Abortion bills go in 'Civil Rights,' not 'Health'!"?
- Few-Shot Examples: Did they show the robot a few examples of bills that were already sorted correctly before asking it to do the rest?
The Big Surprise: More isn't always better
Here is the most important thing the paper found, and it's a bit of a plot twist. You might think that giving the robot more information in its instruction manual would always make it smarter. The paper suggests this is not true.
In fact, the authors found that a minimal increase in context (adding just a little bit of explanation or examples) gave the biggest boost in performance. But if you kept adding more and more details, the robot didn't get much better. It actually hit a wall where adding more text only gave tiny, almost useless improvements.
Even more alarmingly, the paper suggests that sometimes, adding more context actually made the robot worse at its job. It's like trying to help a friend solve a puzzle by shouting every single clue at them at once; eventually, they just get confused and drop the pieces.
The "Batch" Trick
The researchers also tested how many items the robot should sort at once. They tried sorting just 1 bill at a time, then 10, 100, 500, and even 1,000 at a time.
The paper suggests that sorting just one item at a time is usually the worst strategy. It's not only the most expensive way to use the robot, but it also doesn't perform as well. Instead, the paper indicates that batch classification (giving the robot a whole stack of 100 or 500 items to sort in one go) is usually the superior choice.
The "One Size Fits All" Myth
Perhaps the most critical lesson is that you cannot just copy-paste a "perfect prompt" from one project to another. The paper suggests there is substantial heterogeneity (a fancy way of saying "big differences") across different models, different tasks, and even different batch sizes.
What worked perfectly for sorting political bills might fail miserably for sorting survey emotions. Even the same robot might act differently depending on how you ask it. Because of this, the authors argue that researchers cannot rely on general rules. If you want to use these robots for your own research, you have to run your own specific tests to see what works for your specific task.
The "Zero-Temperature" Mystery
Finally, the paper points out something spooky about these robots. Even when researchers set the robot's "temperature" (a setting that controls randomness) to zero—which should make the robot act like a strict, deterministic calculator—the results weren't always identical. The paper suggests that even with the exact same prompt, the robot's output can vary slightly depending on the model and the prompt details. This means you can't always assume the robot will give you the exact same answer twice, even if you try to force it to be consistent.
The Bottom Line
The paper doesn't claim to have found the "magic spell" that makes LLMs perfect. Instead, it suggests that the path to better results is careful, specific testing.
- Don't assume more text in the prompt is better.
- Don't sort items one by one if you can batch them.
- Do expect that what works for one task won't work for another.
- Do test your own specific setup before trusting the results.
It's a reminder that while these AI tools are powerful, they are still a bit like wild horses: you can't just throw a saddle on them and ride; you have to learn exactly how to handle this specific horse for this specific ride.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.