When One Modality Is Not Enough: Multimodal Sex and Life-Stage Classification of Red Deer from Aerial RGB-Thermal Video

Imagine you are a wildlife detective trying to solve a mystery in a giant, living puzzle. Your job isn't just to count how many animals are in a forest; you need to know who they are. Are they the big dads with antlers? The moms? Or the tiny, fast-growing babies? This is the daily challenge for wildlife managers who need to understand the "family tree" of a herd to keep it healthy. But here's the catch: forests are tricky. Trees cast shadows that hide animals, and the ground can look exactly like a deer's fur. To solve this, scientists use drones flying high above, carrying two different kinds of "eyes." One eye sees the world in color, like a human (RGB), and the other sees the world in heat, like a night-vision camera (Thermal). The problem is that neither eye is perfect on its own. The color eye gets confused in the shade, while the heat eye sees a bright, blurry blob that loses all the tiny details needed to tell a dad from a mom. This paper explores a clever trick: what if we don't have to choose between the two eyes, but instead make them work together as a super-team?

The researchers behind this study decided to test this idea on red deer, the majestic animals with the big antlers. They wanted to see if they could build a computer system that flies a drone over a forest, spots every single deer, and then figures out its species, gender, and age automatically. The big question was: Could combining the color and heat cameras create a "super-vision" that is better than either one alone, especially when the seasons change and the clues change with them?

The Two Eyes That See Different Worlds

Think of the drone's cameras like two detectives with very different skills. The Color Detective (RGB) is great at spotting details like the shape of antlers or the pattern of fur, but it gets easily blinded by shadows. If a deer is hiding under a tree, the Color Detective sees nothing but a dark patch of leaves. The Heat Detective (Thermal), on the other hand, sees heat. It doesn't care about shadows; a deer standing in the dark is a bright, glowing star to this detective. However, the Heat Detective is a bit clumsy with details. It sees the deer as a fuzzy, bright blob. It can't easily tell if that blob has antlers or not, and sometimes it gets confused by warm rocks or the ground.

The researchers found that these two detectives fail in opposite ways. In the summer, when a male deer's antlers are soft and covered in fuzzy "velvet," they are warm. The Heat Detective can see them glowing, but the Color Detective might miss them if the deer is in the shade. In the autumn, the antlers harden and cool down. Now, the Heat Detective sees nothing special about them, but the Color Detective can clearly see the hard, bony shape. If you only had one detective, you would miss half the clues depending on the time of year.

The Team-Up Strategy

Instead of trusting just one detective, the team built a pipeline where the two cameras work together at every single step. They didn't just mix the pictures together; they made the computers "talk" to each other using a special language called DINOv3 features. Think of this as a shared notebook where both detectives write down what they see, and the computer checks if their stories match.

Here is how their system works, step-by-step:

1. The Double-Check: The system only counts a deer as "real" if both cameras see it. If the Color camera sees a blob but the Heat camera sees nothing, the system assumes it's just a weird shadow or a rock, not a deer. This stops the system from counting fake animals.
2. The Filter: Not every picture of a deer is good. Some are blurry, some are covered by leaves, and some are just too far away. The system acts like a strict editor, throwing away the bad frames where the clues are missing. It doesn't throw away the deer; it just throws away the bad photos of that deer.
3. The Vote: For the deer that survived the filter, the system looks at all the good photos and takes a vote. If a male deer is seen in 10 photos, and 6 of them show clear antlers, the system votes "Male." Even if 4 photos look like a female because the antlers were hidden, the vote still wins.
4. The Size Check: To tell a baby deer (juvenile) from a mom (adult female), the system measures the deer's size. Since the drone flies at a consistent height, a baby deer will look much smaller in the photo than a grown-up. The system uses this size difference to spot the babies.

The Results: A Team That Wins

The team tested their system on four different drone flights over a forest in Austria. They had a secret "answer key" from people on the ground who knew exactly which deer were there and what they were.

The results were impressive. When they used just the Color camera or just the Heat camera, they got the gender right for about 20 out of 26 deer. But when they combined the two? They got it right for 25 out of 26 deer. That is a 96.0% success rate for identifying the species, and the fusion of the two sensors made the gender identification much more reliable across different seasons.

The system was particularly good at handling the "tricky" cases. For example, there were three male deer that both single cameras thought were females because the clues were hidden. But because the system looked at the combined evidence, it realized, "Wait, the Heat camera saw a warm spot here, and the Color camera saw a shape there," and correctly identified them as males.

Why This Matters

This paper shows that we don't have to choose between a color camera and a heat camera. By fusing them, we can create a census that is much smarter than a simple headcount. Instead of just saying, "There are 50 deer," a drone flight can now say, "There are 50 deer: 15 dads, 30 moms, and 5 babies."

This is a big deal for wildlife managers. Right now, getting this kind of detailed information often requires humans to manually look at photos, which takes a long time and can't be done for every flight. This new method automates the process, turning a simple count into a detailed report of the herd's health. It proves that when one way of seeing the world isn't enough, combining different perspectives can reveal the full picture, even when the animals are hiding in the shadows or the seasons are changing. The system isn't perfect—it still struggles a little with very small deer or when the drone flies too high—but it shows that with the right teamwork, we can read the story of a forest from the sky.