Resources and Evaluation in Computational Humour Generation: A Systematic Survey across Languages and Cultures

Making a computer tell a joke is one of the most deceptively difficult tasks in artificial intelligence. It requires more than just assembling words that follow the rules of grammar; the machine must understand the subtle, often invisible rules of human culture, timing, and expectation. A joke that works in one city might fall flat or even cause offense in another, and what feels funny to one person might seem confusing to someone else. For years, researchers have tried to build systems that can generate humor, moving from simple rule-based programs that swapped words to complex, modern systems that learn from vast amounts of text. But as these systems have grown more powerful, a new problem has emerged: while the computers are getting better at writing jokes, the foundation they stand on is becoming harder to see. We do not know if the data they learned from truly represents the people they are supposed to amuse, or if the tests used to judge them are actually measuring what we think they are.

A recent systematic survey by researchers at King Mongkut's University of Technology Thonburi takes a hard look at this hidden foundation. The team examined seventy-four studies published between 2020 and 2026, a period that saw a massive shift in how computers are taught to create humor. They did not just look at the jokes the machines produced; they traced the entire lifecycle of the research, from the raw data used to train the models to the methods used to decide if a joke was actually funny. Their goal was to determine whether the resources available to scientists are truly open for others to use, whether the data covers the wide variety of human cultures, and whether the way we test these systems is reliable. What they found was a field in rapid transition, where the technology has outpaced the transparency and rigor needed to support it.

The landscape of humor generation has changed dramatically in just a few years. In the past, researchers relied on specific, hand-crafted rules or small sets of data designed for a single type of joke, like a pun or a headline. Today, the field is dominated by large language models, which are massive computer programs trained on huge amounts of text from the internet. These models can generate text in many languages and styles, and they have appeared in seventy-three percent of the studies reviewed. Instead of writing a program to tell a specific joke, researchers now often give these models a simple instruction, or prompt, asking them to be funny. This flexibility has allowed for a surge in new experiments, with the number of studies rising sharply after 2023. However, this shift has created a new kind of opacity. Because the humor knowledge is now spread across the model's internal settings, the instructions given to it, and the data it retrieves during generation, it is often impossible to tell exactly where the humor comes from or what specific cultural context it is drawing upon.

One of the most striking findings of the survey is that despite the abundance of data, it is incredibly difficult to find, reuse, or verify. The researchers checked the availability of the datasets and resources created in these seventy-four studies and found that only twenty-eight percent were publicly available for others to use. Even more concerning, only one study explicitly named the legal license that governs how its data can be used. In the world of scientific research, this is like publishing a recipe but refusing to list the ingredients or the rules for sharing the dish. When the researchers looked at the few resources that were released, they found that none of them had a permanent, unchangeable address on the internet, known as a persistent identifier. This means that links to the data can break, and once a link is broken, the data is effectively lost. Without clear licenses and stable locations, other scientists cannot reproduce the results or build upon the work, which slows down progress and makes it hard to know if a system is truly effective or just lucky.

The survey also revealed a significant gap between the languages spoken by the people the systems are meant to serve and the languages used to build them. More than half of the studies focused exclusively on English, even though the researchers claimed their systems could work across different cultures. While there were studies involving Chinese, Spanish, Japanese, and other languages, many of these relied on translating English jokes rather than collecting humor that was naturally created by native speakers. This is a critical distinction because humor is deeply tied to culture; a joke that relies on a specific historical event or a local custom cannot simply be translated to work elsewhere. The researchers found that only four studies directly evaluated whether the generated humor was culturally appropriate, using raters who actually belonged to the target community. In most other cases, researchers assumed that if a native speaker found it funny, it was culturally safe, without actually testing for specific cultural sensitivities or traditions.

When it comes to judging whether a computer has successfully told a joke, the methods used are often inconsistent and sometimes misleading. The gold standard for measuring humor has always been human judgment, and the survey confirmed that human evaluation is still the most common method, used in eighty-four percent of the studies. However, the way these evaluations are conducted varies wildly. Some studies use a small group of students, while others use professional comedians or large groups of online workers. Fewer than forty percent of the studies reported using statistical tests to ensure their results were not just a fluke, and many did not explain how they decided if a joke was funny or not. In an effort to speed things up, some researchers have started using other computer models to act as judges, but the survey found that these automated judges often disagree with humans. They might rate a response as funny because it is novel or surprising, even if a human would find it confusing or offensive. This creates a risk that systems are being optimized to please other computers rather than to amuse real people.

The researchers concluded that the main obstacle in the field is no longer a lack of data, but rather the difficulty of finding, reusing, and verifying the data that already exists. The shift to powerful, general-purpose models has not removed the need for high-quality, well-documented resources; it has made those resources even more important. To move forward, the field needs to adopt better practices for sharing data, including clear licenses, permanent links, and detailed descriptions of how the data was collected and annotated. Most importantly, researchers need to ensure that their evaluations match the claims they make. If a system is supposed to be funny for a specific cultural group, it must be tested by people from that group, using criteria that reflect their values and traditions. Until these foundations are strengthened, the progress in making computers tell jokes will remain uncertain, with gains that may reflect the quirks of the testing methods rather than a true understanding of human humor.