In the world of artificial intelligence, large language models are often judged by a single, simple question: can it get the right answer? When a user asks for a solution to a math problem, a line of code, or a medical diagnosis, the standard test is whether the model produces one correct response. This approach treats the machine like a student taking a final exam, where only the top score matters. However, in many real-world situations, a single answer is not enough. A scientist designing a new drug might need to see ten different chemical structures that all work, so they can choose the one that is easiest to manufacture. A software engineer fixing a security hole might want several different ways to patch the code to see which is safest. In these scenarios, the value of the machine lies not in its ability to produce one perfect output, but in its ability to generate a rich set of distinct, useful options. The challenge for researchers has been how to measure this ability. Traditional tests often count how many times a model gets it right, or they simply pick the best answer from a list and ignore the rest. This misses the point entirely when the goal is to have a diverse collection of valid possibilities to choose from.

A team of researchers at the RIKEN Center for Computational Science in Japan has developed a new way to look at this problem. They created a specialized testing ground, which they call a benchmark, designed specifically to see how well an artificial intelligence can generate a variety of useful answers for a single task. Instead of asking, "Did it get the answer right?", their method asks, "How many different, genuinely useful answers did it find?" They built this test using five very different types of real-world problems, ranging from designing chemical molecules to fixing software bugs and searching for medical evidence. For each problem, they defined exactly what counts as a "useful" result and how to tell if two results are actually different. For instance, in molecule design, two answers might look different on the surface, but if they share the same core chemical structure, they count as the same useful outcome. The researchers then ran various models through these tasks, asking them to try multiple times, and counted how many unique, valid solutions they accumulated.

The results of this study reveal a surprising truth about how these machines work. The researchers found that the models that are best at producing a single, high-quality answer are not necessarily the ones that produce the best collection of diverse answers. In fact, the ranking of which model is "best" changes depending on how many attempts are allowed. A model that leads the pack when given just one try might fall behind when given twenty tries, while another model that starts slower might eventually uncover a wider range of useful solutions. This suggests that the settings used to run these models matter more than previously thought. The researchers discovered that increasing the "temperature," a setting that makes the model's output more random and varied, often led to a much wider collection of useful results, even though it sometimes lowered the quality of any single answer. Conversely, a feature called "thinking," which encourages the model to reason step-by-step before answering, improved the quality of individual answers but did not always help the model find more distinct solutions.

Perhaps most importantly, the study showed that simply looking at how different the answers look on the surface is not enough to tell if they are truly useful. A common way to judge diversity is to check if the words or characters in the answers are different from one another. The researchers found that this surface-level check is a poor predictor of whether the model has actually found new, valid solutions. A model could produce ten answers that look very different from each other but all represent the same underlying solution, failing to provide the variety a user actually needs. By using a system that checks the actual content and meaning of the answers against the specific rules of the task, the researchers could measure the true "coverage" of the results. They found that this deeper measure often led to different conclusions about which models and settings were performing best.

The team tested four different large language models across these five domains: designing molecules, repairing software repositories, finding bugs in code, diagnosing medical cases, and searching for evidence to support complex questions. They ran each model with different settings, such as varying the level of randomness and enabling or disabling the "thinking" process. They then measured how many distinct, valid outcomes each model produced as the number of attempts increased. The data showed that for most tasks, the model that performed best at the start was not the one that performed best after many attempts. For example, in the task of designing molecules, one model was the leader after a single try, but by the time twenty attempts were made, a different model had uncovered significantly more unique chemical structures. This pattern held true across the board, indicating that the best strategy for generating a single answer is often different from the best strategy for generating a set of answers.

The study also examined whether explicitly telling the model to avoid repeating itself would help. In some cases, researchers have tried to force diversity by asking the model to generate new ideas that are different from what it just said. The researchers found that this approach did not reliably work. While it helped in some specific tasks like searching for evidence, it actually reduced the number of useful solutions in others, such as repairing software. This suggests that simply asking for variety does not guarantee that the model will find new, valid ground. Instead, the way the model is configured, particularly the settings that control its randomness, plays a more critical role in determining whether it can explore a wide range of useful possibilities.

These findings change how we should think about evaluating artificial intelligence. For years, the focus has been on getting the single best answer. But as these machines are used for more complex tasks where having options is crucial, the old way of measuring success is insufficient. The researchers' work demonstrates that a finite set of candidate answers is a meaningful object of study in its own right. It is not just a collection of attempts leading to one winner; it is a landscape of possibilities that can be mapped and measured. By using their new method, which they call validated task coverage, they showed that we can directly evaluate how well a model explores this landscape. The results suggest that to get the most out of these tools, we may need to stop looking for the single best answer and start valuing the breadth of the options they can provide.

The implications of this research extend beyond just picking the right model. It suggests that the way we interact with these systems might need to change. If a user needs a variety of solutions, they might need to adjust the settings to encourage more exploration, even if it means accepting that individual answers might be slightly less perfect. The study provides a concrete way to test and compare these different approaches, moving beyond vague ideas of "creativity" or "diversity" to a clear, measurable count of useful outcomes. The researchers made their tools and data available to the public, allowing others to test new models and settings against this new standard. This opens the door for a more nuanced understanding of what these machines can do, shifting the focus from a single point of success to the full range of their potential.

In the end, the paper offers a quiet but significant shift in perspective. It argues that the value of an artificial intelligence is not just in its ability to be right, but in its ability to be useful in many different ways. By building a test that measures the accumulation of distinct, valid results, the researchers have provided a new lens through which to view these powerful tools. They have shown that the path to the best set of answers is not always the same as the path to the best single answer, and that the settings we choose can make a profound difference in the variety of solutions we receive. This work does not claim to have solved every problem in evaluating AI, but it provides a clear, reproducible method for asking a question that has been overlooked for too long: how many useful things can this machine actually find?