Introducing HALC: A general pipeline for the systematic and reliable construction of prompts for automated coding with LLMs in the computational social sciences

In the social sciences, researchers have long relied on a slow, careful process to understand human communication. They read thousands of documents—newspaper articles, political speeches, or social media comments—and sort them into categories based on what the text says. This is called content analysis. For decades, the most trusted method has been to hire human readers to do this sorting. Humans are flexible and can understand nuance, but they are also slow, expensive, and limited by how many hours they can work. In recent years, powerful computer programs known as large language models have emerged. These programs can read and write human language with remarkable skill, offering the promise of automating this sorting process. However, simply asking a computer to read a text and categorize it often leads to inconsistent results. The computer might give a different answer to the same text if asked twice, or it might misunderstand the specific rules a researcher has in mind. The central challenge has been how to translate the careful, human-made rules of a study into instructions that a computer can follow reliably, without losing the accuracy that human experts provide.

A team of researchers at the University of Hohenheim has developed a new method to solve this problem, which they call HALC. Their goal was to create a step-by-step pipeline that allows scientists to use these powerful computer models to code data systematically and transparently, rather than relying on guesswork. The researchers began by acknowledging that while these computer models are impressive, they are not perfect. They can be unpredictable, and their performance changes depending on how they are asked to do a task. To build a reliable system, the team first conducted a large preliminary study. They took a set of comments about climate movements and asked the computer models to read them repeatedly. They tested hundreds of different ways of phrasing the instructions, or "prompts," to see which ones produced the most consistent and accurate results. They discovered that asking the computer to read the same text five times and then taking the most common answer significantly stabilized the results. They also found that the quality of the initial human data used to teach the computer mattered greatly; if the human ground truth was shaky, the computer's results were shaky too. Most importantly, they learned that the best instructions were not short or vague. The most successful prompts were detailed, asking the computer to explain its reasoning step-by-step and to justify its final decision, much like a human coder would think through a problem before writing down an answer.

Building on these findings, the researchers formalized the HALC pipeline. This process starts with a standard human-made codebook, which contains the definitions and rules for a specific research project. Instead of trying to reinvent the wheel for the computer, the researchers translate these human rules directly into a structured prompt. They then test this prompt on a small sample of data, comparing the computer's output against the original human coding. If the computer does not match the human reliability standards, the researchers do not just guess a new instruction. Instead, they follow a strict, rule-based process to refine the prompt, perhaps by adding more specific examples or clarifying a definition, and then test it again. This cycle continues until the computer achieves a level of agreement with the human coders that is considered scientifically acceptable. Only then do they use the computer to process the entire dataset. This approach treats the computer not as a magic box that works on its own, but as a tool that must be carefully calibrated against human judgment.

To prove that this system works in the real world, the team applied the HALC pipeline to two completely different types of data. The first was a massive collection of political party manifestos from five different countries, written in English, German, and Spanish. The task was to identify which sentences discussed immigration and whether they focused on border control or the integration of people already living in the country. The second dataset consisted of annual reports from thirty major German companies, where the goal was to determine the tone of the writing—whether it was positive, negative, or neutral. In both cases, the researchers used the same English-language instructions for the computer, relying on the model's ability to understand different languages without needing a separate prompt for each one. The results were encouraging. For the political texts, the computer achieved reliability scores that were quite close to the original human coders, successfully navigating different languages and political contexts. For the business reports, the computer performed even better when using a specific type of model designed for complex reasoning, showing that the pipeline could adapt to different types of tasks.

The study concludes that while large language models are powerful, they cannot simply be handed a task and expected to perform perfectly. They require a structured, systematic approach to ensure their output is trustworthy. The HALC pipeline offers a way to bridge the gap between human expertise and machine speed. By grounding the computer's instructions in established human rules and rigorously testing them before full deployment, researchers can automate their work without sacrificing the validity of their findings. The authors suggest that this method makes the process of automated coding transparent and reproducible, allowing other scientists to see exactly how the instructions were built and verified. While the computer models continue to evolve, the core principle remains: reliable automation comes from a careful partnership between human design and machine execution, ensuring that the final data is as solid as if it had been sorted by a team of human experts.