Capability, ranking dynamics, and inference pricing in the large language model ecosystem: evidence from linked public data
This paper integrates linked public data from leaderboards, human-preference panels, and pricing catalogs to characterize the large language model ecosystem, revealing that performance scales with parameter count and time, preference ratings exhibit rapid turnover and stability within models, and pricing is significantly driven by capability scores and open-weight status despite notable provider price variations.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the rapidly evolving world of artificial intelligence, a specific type of computer program known as a large language model has become a central tool for writing, reasoning, and solving problems. These systems are trained on vast amounts of text, learning to predict the next word in a sentence until they can generate coherent paragraphs, answer complex questions, and even write code. Because there are now thousands of these models, ranging from massive systems built by giant technology companies to smaller, open versions created by independent researchers, a need has arisen to understand how they compare. To do this, the community relies on two main things: standardized tests, where models answer a set of difficult questions to earn a score, and human preference arenas, where people vote on which of two model answers they like better. Alongside these performance metrics, a market has emerged where companies sell access to these models by the word, creating a complex landscape of prices and capabilities that is difficult to navigate.
A recent study by Peter Bago of Corvinus University of Budapest sought to bring order to this chaotic ecosystem by linking three massive, public sources of data. The researcher combined a record of thousands of test scores from open models, a long-term log of human preference rankings, and a comprehensive catalog of prices charged by different providers. By connecting these datasets, the study moved beyond looking only at the most famous models to examine the entire landscape, asking how performance relates to size, how rankings change over time, and how price is actually determined. The goal was not to declare a single winner, but to understand the underlying rules that govern how these digital tools are valued and how they evolve.
The first major discovery concerns the nature of the tests themselves. The study analyzed over 4,500 records of open models that had been tested on six different benchmarks, ranging from instruction following to advanced mathematics. The researchers found that these six different tests are not measuring six separate skills; instead, they are overwhelmingly measuring a single, dominant capability. When a model performs well on one test, it almost certainly performs well on the others. This common thread explains nearly three-quarters of the variation in scores across the entire ecosystem. However, the study also noted that these tests struggle at the very bottom of the performance scale. Many models score so low that they are indistinguishable from random guessing, suggesting that current tests are better at separating the best from the good than they are at distinguishing the very poor from the non-functional.
When looking at what drives a model to perform better, the study confirmed that size matters, but it is not the whole story. Larger models, measured by the number of parameters they contain, generally score higher. Specifically, a model that is ten times larger than another tends to score about 13.7 points higher on the average test. Yet, time is an equally powerful force. Even when comparing models of the same size and type, those submitted more recently perform significantly better. A model submitted one year after another, but built with the same amount of computing power, scores roughly 10.5 points higher. This suggests that the field is improving not just by building bigger machines, but by finding smarter ways to train them. The study also found that models designed for conversation or those created by merging existing models often outperform their raw, unmodified counterparts, indicating that the process of fine-tuning and combining is a key driver of current progress.
The human side of the ecosystem tells a different story about stability. The study tracked the top-ranked models in a human preference arena over more than a year, observing that leadership is incredibly fleeting. The title of the best model changed hands 19 times, with the average leader holding the top spot for only about 12 days. Despite this constant turnover, the gap between the very best and the tenth-best model remained surprisingly steady. The study also revealed a significant shift in the scoring system itself. On a specific date in May 2025, the default way scores were calculated changed, causing every model's rating to jump by approximately 18 points overnight. This meant that comparing scores from before and after that date without adjustment would be misleading, as the numbers were no longer on the same scale. Before this change, the ratings were stable, and models did not drift up or down simply by being listed repeatedly.
Perhaps the most complex finding relates to the price of using these models. The study examined nearly 2,700 different price listings from 82 different providers. It found that for the exact same model, prices can vary wildly depending on who is selling it. For models offered by three or more providers, the most expensive listing is typically 38 percent higher than the cheapest one, and for some models, the difference is even larger. This suggests that simply shopping around can save a buyer a significant amount of money. The study also built a model to understand what drives these prices. It found that capability is a major factor: a model that is one standard deviation better in human preference ratings commands a price that is more than three times higher. However, being an open model, where the code is publicly available, acts as a massive discount, reducing the price to less than one-fifth of what a proprietary model with similar skills would cost. Additionally, the study found that older models, those that have been listed for a longer time, tend to carry a price premium, suggesting that established models retain value even as newer, potentially better ones arrive.
Finally, the study checked how well the two main ways of measuring quality agree. When the researchers looked at the 69 models that appeared in both the test score records and the human preference rankings, they found a strong agreement. Models that scored well on the standardized tests were generally the same ones that humans preferred. This convergence suggests that while the tests and the human votes measure things slightly differently, they are pointing toward the same reality of capability. The study concludes that while the landscape is vast and the data is messy, clear patterns emerge when the sources are linked. The ecosystem is driven by a single dominant capability, improved by time and size, and priced in a way that rewards performance but heavily discounts open access. For anyone trying to understand or buy these tools, the lesson is that performance is predictable, leadership is temporary, and the price you pay depends as much on who is selling and when you buy as it does on what the model can actually do.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.