Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code
This paper introduces a structured ontology to analyze a large collection of 57,500 unique transactional prompts from GitHub, transforming them into richly structured linguistic objects to reveal diverse usage patterns across languages and domains while validating annotation quality through comprehensive error analysis.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a super-smart robot how to do a specific job, like writing a poem or fixing a bug in a computer program. You don't give the robot a formal programming code; instead, you write it a set of instructions in plain English. This set of instructions is called a prompt. Think of a prompt like a recipe card for a chef. If the recipe is vague ("make something tasty"), the chef might guess wrong. But if the recipe is detailed ("chop two onions, sauté for five minutes, add salt"), the chef knows exactly what to do. For a long time, scientists have studied the "chef" (the AI model) in great detail, trying to make it smarter. But they haven't paid much attention to the "recipe cards" (the prompts) themselves, treating them as messy, informal notes rather than something worth studying seriously. This paper argues that these prompts are actually a new kind of "language" that humans use to program AI, and they deserve to be analyzed just like any other language or code.
The researchers behind this study decided to treat prompts like a scientific object. They went digging through the public code repositories on GitHub, which is like a giant library where software developers share their work. They were looking for a specific type of prompt called a transactional prompt. To understand the difference, imagine a casual conversation with a friend where you ask, "What's the weather?" and they answer. That's an "interactive" prompt. A transactional prompt is different; it's like a pre-written instruction taped to a machine that runs automatically every time a button is pressed. It's designed to be repeated, robust, and part of a larger software system. The team collected 57,500 of these unique, real-world prompts to see what they look like, how they are structured, and what people are actually asking AI to do.
To make sense of this huge pile of text, the authors built a special "map" or ontology. Imagine trying to organize a messy attic full of random items. Without a system, it's just chaos. The ontology is like a set of labeled boxes and tags that let them sort the prompts into categories: What language is it written in? What is the task? Is there a specific format required? Is the prompt telling the AI what not to do? By turning unstructured text into these rich, structured data points, they could start asking serious questions.
What did they find? First, the data shows a huge variety, but it follows a pattern known as a Zipf-like distribution. Think of this like a city's population: a few big cities (like English) have millions of people, while thousands of tiny villages (other languages) have very few. In the world of prompts, English is the giant city, making up about 84.66% of the prompts. However, there are still 62 different languages represented, and even in those English prompts, people mention other languages constantly (like asking the AI to "translate to Spanish"), showing a hidden layer of linguistic diversity.
The study also revealed that these prompts are surprisingly complex. They aren't just simple sentences. On average, each prompt contains about 6.85 distinct "instruction blocks." These blocks are like the steps in a recipe. The most common steps are things like "here is the context," "don't make the answer too long," or "output the result as a JSON file." Interestingly, about 31% of the prompts contain "negative instructions"—telling the AI what not to do, like "don't hallucinate" or "don't use extra text." This suggests that developers are constantly trying to rein in the AI's tendency to be too chatty or make things up.
The researchers also looked at how people structure these prompts. They found that the most common setup is a two-part conversation: a "System" message (which sets the rules, like "You are a helpful assistant") and a "User" message (which gives the specific task). This "System-User" duo has become the standard unit for transactional prompts, appearing in 64% of the cases. It seems developers view the System role as a static configuration layer and the User role as the dynamic input, rather than using the AI for complex, multi-turn role-playing games within a single script.
In terms of what people are actually doing, NLP tasks (Natural Language Processing, like summarizing text or answering questions) are the most popular, with the top four tasks covering nearly half of all the data. However, the "long tail" of the data shows people using AI for all sorts of weird and wonderful things, from game strategy to policy generation. The data also suggests that most of these prompts are "grounded," meaning they rely on specific context provided in the prompt (like a document or a code snippet) rather than just the AI's internal memory. About 89.25% of the cases use this kind of grounding.
The authors are careful to note that their findings come from a specific slice of the internet (public GitHub projects using specific Python libraries), so it might not represent every single prompt in existence. They also used AI to help label and analyze the data, which means there are some errors, though they checked the quality and found the accuracy to be quite high for most categories (often over 90%).
Ultimately, this paper doesn't just give us a list of prompts; it gives us a toolkit. The authors released the dataset and a special website where anyone can browse, search, and explore these prompts by their features. They hope this will help scientists, engineers, and linguists understand how humans are really interacting with AI. By treating prompts as first-class scientific objects, we can start to see the patterns, the "unspoken norms," and the creative ways people are shaping the behavior of these powerful machines. It's a step toward understanding the new language we are inventing to talk to our digital partners.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.