To understand how the brain works, scientists must first understand what the animal it controls is actually doing. For decades, the gold standard for measuring animal behavior was a human observer watching video recordings, pausing and rewinding to note every movement, from a mouse sniffing a corner to a fly striking another. This method is flexible and interpretable, but it is incredibly slow and prone to human error. In recent years, computers have begun to take over this task, but they have mostly relied on a two-step process that creates its own bottlenecks. First, a computer program must be trained to find specific points on an animal's body, like the tip of a nose or a joint in a leg, a process that requires thousands of hours of human labeling. Second, the computer uses those points to guess what behavior is happening. This approach is slow to set up, often fails when animals block each other from view, and is usually too sluggish to react in the moment. If a researcher wants to give a mouse a reward the instant it grooms itself, or stop an experiment the moment a specific movement occurs, the computer must be fast enough to see the action and decide what to do before the video even finishes recording.

A team of researchers at Carnegie Mellon University has developed a new system called B-REAL that cuts out the middleman entirely. Instead of trying to map the skeleton of an animal, this system looks directly at the raw video pixels and learns to recognize behavior instantly. The result is a tool that can classify what an animal is doing faster than the camera can take the next picture, even on standard computer hardware. This speed allows for a new kind of experiment where the animal's behavior can trigger immediate changes in its environment, such as delivering a drop of water or a burst of light, while the behavior is still happening. The researchers showed that this method works with far less training data than previous systems, can spot fleeting movements that last only a fraction of a second, and can be used to shape an animal's natural actions in real time.

The core of this new approach is a shift in how the computer processes video. Traditional methods treat a video as a series of still images, first finding body parts and then calculating how those parts move. B-REAL skips the body-part finding step. It takes a short sequence of video frames and feeds them directly into a neural network, a type of computer program designed to recognize patterns. To understand motion without tracking specific points, the system uses a technique that swaps small pieces of information between neighboring frames. Imagine a stack of photos where a few pixels from the photo above are swapped with the photo below; this allows the computer to see how things change over time without needing to know exactly where a paw or a wing is located. Because this method does not require the computer to learn a complex map of an animal's body, it avoids the common failures of older systems, such as losing track of an animal when it is partially hidden or when the lighting changes.

The researchers tested this system on two very different challenges: complex social interactions between mice and extremely fast, rare attacks between fruit flies. In the mouse experiments, the system learned to distinguish between attacking, investigating, and mounting with high accuracy after seeing only a minute of labeled video. This is a significant improvement over existing tools, which often require hours of labeled data to reach similar performance. When the team tested the system on fruit flies, the challenge was even greater. Some aggressive behaviors, like a fly charging at another, last for only a single frame of video, which is about 33 milliseconds. Most computer systems miss these entirely because they average out movement over longer periods. B-REAL, however, detected these split-second charges with high precision, outperforming other methods even when given very little training data. The system was able to identify these rare events because it learned directly from the visual patterns of the video rather than relying on a pre-defined set of body landmarks that might not capture such rapid, subtle shifts.

Perhaps the most compelling demonstration of the system's power was a closed-loop experiment where the computer acted as a teacher. The researchers set up a situation where mice received a reward of sweetened water only when the computer detected that they were grooming their heads. The system had to identify the grooming behavior and trigger the water dispenser within a fraction of a second. Because the detection was so fast and accurate, the mice quickly learned the connection. Over the course of the experiment, the rate at which the mice groomed themselves increased by 129 percent. The animals did not just groom more often; they also adjusted the duration of their grooming to match the computer's requirements, shifting toward slightly longer bouts to ensure they triggered the reward. This proves that the system is not just a passive observer but a capable partner in shaping natural behavior, capable of reinforcing specific actions as they happen.

To make the system even more efficient, the researchers explored a way to give the computer a head start without needing more labeled data. They used a large, pre-trained video model that had learned general concepts of motion from thousands of hours of unlabeled video. They transferred what this large model knew into the smaller, faster system. This process, known as distillation, allowed the system to learn from just 30 seconds of labeled video what would normally take five to ten minutes to learn from scratch. Remarkably, this pre-training improved the system's ability to recognize behaviors in completely new camera angles, such as a view from underneath the animal, even though the pre-training data only included overhead or side views. This suggests that the system learned a deep understanding of how animals move that applies across different settings, making it highly adaptable for future experiments.

The implications of this work extend beyond just speed and accuracy. By removing the need to track body parts, the system can observe behaviors that are difficult to define by posture alone, such as an animal interacting with a specific object or navigating a complex environment. It opens the door for experiments where the computer can respond to the animal's state in real time, creating a dynamic loop between the subject and its environment. While the current study focused on mice and flies, the principles apply broadly to any situation where understanding behavior in real time is crucial. The system runs on standard hardware, meaning it does not require expensive supercomputers to operate, making advanced behavioral analysis accessible to a wider range of laboratories. Ultimately, B-REAL transforms behavior from a static record of the past into a live signal that can be used to guide and understand the brain in the moment.