Naming the deliverable raises convention coverage in large language model outputs
This study demonstrates that explicitly using practitioner-specific names for deliverables (e.g., "RACI matrix" instead of a descriptive phrase) significantly improves the adherence of large language model outputs to professional conventions across various models and datasets.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the author. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of professional work, there is a quiet language that experts use to pack a complex set of rules into a single word. A project manager might ask for a "RACI matrix," a financial analyst might request a "term sheet," and a doctor might order a "discharge summary." To an outsider, these names are just labels, but to the practitioner, each name acts as a compressed file containing a specific structure, a required list of sections, and a strict format that must be followed. This is the vocabulary problem: the gap between what a person needs and the specific words an expert uses to describe it. For decades, information scientists have known that when people search for information using their own words instead of the field's official terms, they often miss the mark. Now, as artificial intelligence becomes a primary tool for drafting these professional documents, a new question has emerged. When a person asks an AI to create a document, does using the expert's specific name for that document help the machine produce a better result than simply describing what the document is supposed to do?
A researcher at a development corporation in Japan set out to test this idea with a series of controlled experiments involving large language models, the powerful AI systems that can write text, code, and reports. The study focused on a simple but critical distinction. In one scenario, a user would ask the AI to create a document using the professional name, such as "Create a Gantt chart." In another scenario, the same user would describe the document's purpose without using the name, such as "Create a schedule showing which tasks happen when." Both requests ask for the same thing, but the first carries the hidden weight of professional conventions, while the second leaves the AI to guess the format. The researcher asked the AI to generate dozens of different types of documents, from investment plans to medical notes, using both the professional name and the plain description. To see which version was better, the outputs were measured against a checklist of rules that define what a correct version of that document looks like.
The results were clear and consistent across every model tested. When the user provided the professional name, the AI included significantly more of the required conventions than when the user only described the task. The improvement was substantial, ranging from an increase of eleven to twenty-nine percentage points in how many rules the AI followed. This means that if a model following a plain description got about half of the rules right, adding the professional name pushed it to get significantly more of them right, though it did not reach perfection. The effect was strongest for documents that have a fixed, rigid structure, such as specific analytical methods or formal business templates. For these items, the name acted as a key that unlocked a set of instructions the AI had learned from human text but did not know to apply unless prompted. For more common documents, like a general press release, the difference was smaller, likely because the description of the purpose was already enough to guide the AI.
The study also explored whether the AI could help users who did not know the right name. In a second phase, the researcher asked the AI to first tell the user what the professional name for a document was, and then asked it to generate the document using that name. The AI was able to provide the correct name in roughly two-thirds of the cases in the smaller, curated set, but accuracy dropped to about 46% in the larger, rule-sampled set. When the user then used that name to ask for the document again, the quality of the output improved significantly, recovering about half to three-quarters of the benefit seen when the user knew the name from the start, though recovery was lower in the sampled set. However, the AI was not perfect; it sometimes suggested a real professional name for a different type of document, which would lead to a poor result. This finding suggests that while the AI can act as a bridge to the expert vocabulary, the user still needs to verify that the name is correct.
These findings challenge the idea that users need to learn a field's entire vocabulary to get good results from AI. Instead, the study suggests a simpler, more practical skill: knowing that these documents have specific names, asking the AI for those names when they are unknown, and then using them. The research indicates that the professional name is not just a label but a signal that tells the AI to switch into a specific mode of writing, activating a set of conventions that a plain description cannot trigger. This is particularly important for tasks where the format is as important as the content. The study concludes that for anyone using these tools, the ability to ask for and check the correct name for a deliverable is a fundamental skill, one that allows the AI to function as a true partner in professional work rather than just a text generator.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.