Multitask Jet Analysis with Vision-Language Models: A Physics-Informed Four-Panel Representation
This paper demonstrates that physics-informed four-panel image representations of jets, when processed by adapted Vision-Language Models, enable efficient and accurate multitask analysis—including classification, attribute prediction, and anomaly localization—across simulated and real high-energy physics data through a unified instruction-conditioned interface.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
High-energy physics is the science of smashing particles together at speeds close to the speed of light to see what they are made of. When these collisions happen, they do not produce simple, clean results. Instead, they create sprays of new particles that fly out in all directions. Physicists call these sprays "jets." Inside every jet, there is a hidden story about the original particle that started the crash. Some jets come from common particles that fill the universe, while others come from rare, heavy particles that scientists are hunting for as clues to new laws of nature. The challenge is that these jets look very similar to the naked eye, even to the most advanced computers. To find the rare ones, scientists must look at the tiny details of how the particles are arranged, how heavy they are, and how far they travel before they stop.
For decades, researchers have built special computer programs just to read these jets. These programs are like custom-made keys designed to open one specific lock. They work well, but they are rigid. If a scientist wants to ask a different question about the same jet, they often have to build a new program from scratch. A new study asks a different question: Can we use a single, flexible computer brain that already knows how to look at pictures and read language to understand these jets? The researchers wanted to see if a general-purpose artificial intelligence, trained on everyday images like photos of cats or landscapes, could be taught to see the physics inside a particle collision without needing a custom-built brain for every single task.
The team started by turning the invisible data of a particle jet into a picture that a computer could see. They did not use a camera, of course. Instead, they took the list of every particle in a jet and arranged them into a grid, creating a square image. To make sure the computer could read the physics, they did not use colors like red or blue to mean "hot" or "cold." Instead, they used four distinct panels within the image to show four different kinds of information at once. One panel showed the flow of energy. A second panel showed what kind of particles were there, such as whether they were charged or neutral. A third panel showed how far the particles traveled before stopping, which is a clue about how long-lived they were. The fourth panel showed the electrical charge of the particles. This created a single, complex image that contained the entire story of the jet, ready for a computer to analyze.
They then took four different powerful artificial intelligence models, which are usually designed to understand photos and answer questions about them, and gave them a new job. They did not retrain these models from scratch. Instead, they made a small, efficient adjustment to the models, allowing them to learn the new language of particle physics while keeping their original knowledge intact. The researchers taught the models three different tasks using the same image. First, they asked the model to identify what kind of jet it was looking at, choosing from ten different types of particles. Second, they asked the model to describe specific features of the jet, such as how many branches of particles it had or if it contained heavy particles. Third, they tested the model's attention by swapping one of the four panels with a panel from a different jet and asking the model to spot the change.
The results showed that the general-purpose models could indeed learn to read these physics images. Before any training, the models were essentially guessing, getting the answer right only about one time in ten, which is what you would expect from random chance. But after seeing just a small fraction of the available data—about 200,000 jets, which is a tiny drop in the ocean of data collected by particle colliders—the models became highly accurate. One of the models reached an accuracy of over 75 percent in identifying the correct type of jet. More importantly, the same model could also describe the internal details of the jet with over 93 percent accuracy and spot the swapped panel with nearly 100 percent accuracy. This proved that a single, adaptable system could handle multiple complex questions about the same object without needing a separate tool for each job.
The researchers also discovered which parts of the image were most important for the computer to learn. They found that the panel showing how far the particles traveled was the most critical for telling the difference between common jets and those containing heavy, short-lived particles. They also found that the panel showing electrical charge was essential for distinguishing between two types of jets that look almost identical in every other way. When they tried to improve the model's ability to spot the rarest jets by showing it more examples of those specific types, they found a trade-off. Giving the model more examples of the rare jets improved its ability to find them, but it slightly reduced its overall accuracy on the other types. This suggests that while the model is powerful, it still needs a balanced view of the data to perform at its best.
Finally, the team tested if this learning could travel to new worlds. They took the models trained on simulated data and asked them to look at real data from actual particle collisions and a different set of simulated jets. Even with very little new training, the models adapted quickly. They maintained their high accuracy in describing the jets and spotting changes, proving that the way they learned to see the physics was not just memorizing the simulation but understanding the underlying patterns. This work suggests that the future of particle physics analysis might not require building a new, specialized computer for every new discovery. Instead, a single, flexible interface could allow scientists to ask different questions about the same data, using a common language to unlock the secrets hidden inside the spray of particles.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.