EMRB: A Multi-Level Benchmark for Evaluating LLM Reasoning over Raw Electromagnetic Signals
This paper introduces EMRB, a multi-level benchmark comprising 200 problems across five difficulty levels that evaluates the ability of large language models to analyze raw electromagnetic I/Q data by writing and executing code, revealing significant performance gaps in complex system design tasks and proposing the ReconPilot framework to substantially improve reasoning accuracy.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the invisible air around us, a constant storm of radio waves carries our conversations, our music, and our data. To the human ear, this is silence, but to a radio receiver, it is a chaotic stream of raw electrical fluctuations. Engineers who work with these signals do not see neat charts or labeled boxes; they see a complex, unstructured wave that must be measured, cleaned, and understood before it can be useful. For decades, the task of making sense of this noise has required specialized human expertise and custom-built software. Now, a new generation of artificial intelligence, known as large language models, is being tested to see if it can perform this same work. These models are famous for writing code and answering questions, but they have never been asked to look at a raw, unprocessed radio signal and figure out what is happening inside it without any help.
A team of researchers has created a new test to answer this question, treating the analysis of radio waves as a rigorous challenge for artificial intelligence. They built a benchmark called EMRB, which presents computer programs with a raw recording of radio activity and asks them to write their own code to measure specific details, such as how strong a signal is or what kind of information it carries. The test is designed to be fair and precise, using signals that were generated by computers with known answers, so the researchers can check exactly how well the artificial intelligence performed. The results reveal a clear picture of current capabilities: while the most advanced models can handle simple measurements, they struggle significantly when the task requires them to track multiple signals at once or to design a new system based on what they find.
The researchers evaluated fourteen different artificial intelligence models, ranging from widely available open-source programs to powerful proprietary systems. They gave each model two hundred problems, organized into five levels of increasing difficulty. The easiest level asked the models to find basic facts, like the frequency or power of a single signal. As the levels grew harder, the models had to choose the right mathematical tools to separate overlapping signals, calculate complex communication metrics, and finally, design a complete system that could operate without interference. The models were not allowed to simply guess; they had to write and run Python code to extract the numbers from the raw data file, then interpret those numbers to answer the question.
The performance of the models varied widely, with scores ranging from about twenty-four percent to nearly seventy-nine percent. The strongest performers, which included the most advanced models available today, managed to solve the simpler problems with high accuracy, often scoring above eighty-five percent on basic measurements. However, as the tasks became more complex, the scores dropped sharply. When the models were asked to analyze a crowded spectrum with several signals mixed together, their performance fell to around fifty percent. The most difficult challenge, which required designing a new communication system based on the analysis, proved to be nearly impossible for the current generation of models, with the best score reaching only about forty percent.
The study found that the main failure was not a lack of knowledge about the formulas or the physics of radio waves. Instead, the models failed because they could not keep track of the different signals as they worked through a long chain of reasoning. When a model had to identify five different signals in a single recording, it often lost its place, mixing up the details of one signal with another. This caused errors to pile up, leading to incorrect conclusions even if the initial steps were correct. The researchers also discovered that the models did not benefit from simply trying harder or running more code; the most efficient models reached high scores with fewer attempts, while others wasted time on unproductive calculations.
To help the models succeed, the researchers proposed a new method called ReconPilot, which breaks the analysis into three distinct steps. First, a fixed, automated script scans the raw signal to create a simple map of where activity is happening, without trying to identify what the signals are. Second, the artificial intelligence uses this map to focus its attention and write code to solve the specific question. Third, the model reviews its own work to check for consistency before submitting an answer. This structured approach significantly improved the results. For some models, the new method raised the overall score by more than seventeen points, proving that giving the artificial intelligence a clear starting point and a way to check its work can compensate for some of its reasoning weaknesses.
Despite these improvements, the study concludes that current artificial intelligence is not yet ready to replace human engineers in the most complex tasks. The models can reliably measure simple signals, but they cannot yet be trusted to design new systems or manage crowded radio environments where multiple signals interfere with each other. The researchers emphasize that their test uses synthetic signals generated by computers, so the results do not yet prove how these models would perform on real-world radio recordings with hardware flaws or unexpected noise. However, the benchmark provides a clear, objective way to measure progress, showing that while artificial intelligence is getting better at understanding raw data, it still has a long way to go before it can reason through the full complexity of the electromagnetic spectrum.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.