Indistinguishable from Chance: Auditing LLM-Generated Trading Strategies Against Exposure-Matched Nulls

In the world of investing, the most valuable skill is knowing when to buy and when to sell. This is called market timing. For decades, investors have tried to build machines that can predict these moments better than a human can, hoping to turn small advantages into large profits. The latest tool in this quest is the large language model, a type of artificial intelligence capable of writing complex computer code and explaining its logic in fluent sentences. These models can be asked to design a trading strategy, write the code to run it, and even produce a report showing how much money it would have made in the past. The promise is that anyone, from a professional fund manager to a curious individual, could ask a computer for a winning plan and receive a complete, ready-to-use solution. But there is a deep and well-known problem with looking at the past to predict the future. A strategy can appear brilliant on paper simply because it happened to match random patterns in the data, a mistake known as overfitting. When a strategy is built by a machine that has never seen the real world, the risk is that it is not finding a hidden pattern, but merely mimicking the shape of noise.

A team of researchers set out to test whether these artificial intelligence models actually possess any skill in timing the market, or if their impressive reports are just a convincing illusion. They did not simply ask the models to write code; they built a rigorous testing ground to see what happened when that code met real market data. The researchers asked five different AI models to create trading strategies for three major assets: the S&P 500 stock index, gold futures, and the currency pair of the euro and the US dollar. The models generated nearly two hundred distinct strategies, each with its own set of rules for when to buy and when to sell. To ensure the test was fair, the researchers ran every single strategy in a controlled digital environment that charged a small fee for every trade, mimicking the real costs of doing business. They then compared the results of these AI-generated plans against a simple benchmark: buying an asset and holding it without ever selling. They also compared the AI strategies against a clever control group. For every AI strategy, they created a "null" version by taking the exact same periods of time the AI decided to hold a position, but shuffling those periods around randomly. This test stripped away the AI's claimed ability to choose the right moment, leaving only the fact that it was holding the asset for a certain amount of time. If the AI was truly skilled, it should have outperformed this shuffled version.

The results were stark and consistent. Across the board, the strategies generated by the artificial intelligence failed to beat the simple act of holding the asset. On average, the AI strategies lost money, delivering a performance metric known as the Sharpe ratio that was negative. This means that after accounting for the risk taken and the costs of trading, the average investor would have been better off doing nothing than following the AI's advice. Only a tiny fraction of the strategies, less than eight percent, managed to outperform the simple buy-and-hold approach. When the researchers looked deeper, they found that the AI models were not finding any hidden timing skill. The strategies performed no better than the shuffled, random versions of themselves. In fact, the AI strategies were statistically indistinguishable from random chance. The few times a strategy appeared to do well, it was not because the model had learned to predict the market, but because the strategy happened to hold the asset more often when the market was rising. This is a bias toward being long, or holding the asset, rather than a skill in choosing the right moment. When the researchers removed this bias by comparing the strategies to their shuffled counterparts, the illusion of skill vanished completely.

Perhaps the most revealing part of the study occurred when the researchers asked the models to do something they cannot actually do: run a backtest and report the results without being given the data. The models were asked to write a strategy, calculate its performance, and tell the user the final numbers. None of the models could do this, because they do not have the ability to execute code and crunch numbers in real time. Instead, they produced a simulation of a report. Some models printed out a script that looked like it would calculate the numbers if someone else ran it. Others simply stated general rules about what a good result looks like. A few models went further and fabricated specific numbers, presenting a block of text that looked like a completed report but was entirely made up. This behavior highlights a dangerous gap between the appearance of competence and the reality of capability. The models are fluent in the language of finance and can mimic the structure of a professional analysis, but they cannot generate the evidence that analysis requires.

The study also examined whether using a larger, more powerful model or asking for a more robust strategy would change the outcome. It did not. While larger models wrote code that was more likely to run without errors, and specific prompts changed how often the strategies held an asset, neither factor improved the ability to time the market. The improvements were purely in the exposure, or how much time the strategy spent in the market, not in the quality of the decisions made. The researchers also found that their own testing process had to be constantly checked for errors. In one instance, a data error in their own system had made a random strategy look like a winner, a mistake that was only caught because they had built in controls to verify their own work. This self-audit reinforced the central lesson: a plausible-looking result is not a correct one until it has been checked against a standard that proves the test can actually find a winner if one exists.

For the investor, the takeaway is clear. The artificial intelligence models are not currently capable of generating trading strategies that contain genuine skill. They produce strategies that look complete and confident, but when tested against real data, they perform no better than random chance and often worse than simply holding an asset. The reports they generate, filled with specific numbers and performance charts, are hypotheses, not evidence. Until a strategy is re-run on documented data and compared against a fair benchmark that accounts for how much time it spends in the market, it should be treated as unproven. The cost of relying on these tools is not just a missed opportunity to beat the market, but the potential loss of the steady returns that come from passive investing. The models have mastered the art of sounding like an expert, but they have not yet learned how to be one.