🧬 biology

Meow-Omni 1: A Multimodal Large Language Model for Feline Ethology

The paper introduces Meow-Omni 1, a novel open-source quad-modal large language model that fuses video, audio, physiological time-series, and text to achieve state-of-the-art feline intent recognition by addressing the challenge of semantic aliasing in animal behavior analysis.

Original authors: Jucheng Hu, Zhangquan Chen, Yulin Chen, Chengjie Hong, Liang Zhou, Tairan Wang, Sifei Li, Giulio Zhu, Feng Zhou, Yiheng Zeng, Suorong Yang, Dongzhan Zhou

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jucheng Hu, Zhangquan Chen, Yulin Chen, Chengjie Hong, Liang Zhou, Tairan Wang, Sifei Li, Giulio Zhu, Feng Zhou, Yiheng Zeng, Suorong Yang, Dongzhan Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Problem: The "Cat Whisperer" Dilemma

Imagine you are trying to guess what your cat is thinking. You see it purring. Is it happy? Or is it in pain and trying to soothe itself?

The Issue: The paper calls this "Semantic Aliasing." It's like a word having two completely opposite meanings depending on the context. A purr can mean "I love you" or "I am terrified," and looking at the cat's face (video) or listening to the sound (audio) often isn't enough to tell the difference.
The Old Way: Previous AI models were like detectives who only looked at photos or listened to recordings. They were "blind" to the cat's internal body signals. They could guess the action (e.g., "running"), but they couldn't guess the intent (e.g., "running because it's scared" vs. "running because it's playing").

The Solution: Meow-Omni 1

The researchers built Meow-Omni 1, a new type of AI brain designed specifically to understand cats. Think of it as a super-intern at a veterinary school who doesn't just watch and listen, but also has a direct line to the cat's heartbeat and movement sensors.

Here is how it works, using simple analogies:

1. The Four Senses (Quad-Modal)

Most AI models only have two or three "senses." Meow-Omni 1 has four, which the paper calls Quad-Modal:

Eyes (Video): Watching the cat move.
Ears (Audio): Listening to meows and purrs.
Voice (Text): Reading descriptions and asking questions.
The "Inner Sense" (Biological Time-Series): This is the magic ingredient. It reads high-speed data from sensors (like a smartwatch for cats) that track heart rate, acceleration, and body vibrations.
- Analogy: If a human detective sees a suspect running, they might think "he's late." But if they also have a heart monitor showing the suspect's heart is racing at 180 bpm, they realize, "He's being chased." Meow-Omni 1 uses this "inner sense" to solve the mystery of the purring cat.

2. The Brain Surgery (Architecture)

The researchers didn't build a brain from scratch; they performed "surgery" on an existing smart AI (MiniCPM-o).

The Transplant: They took a specialized "time-series encoder" (a tool good at reading sensor data from a different project called Intern-S1 Pro) and grafted it into the cat AI.
The Translator: They built a special "projection layer" that acts like a translator. It takes the raw, fast-paced numbers from the cat's sensors and turns them into a language the AI's brain can understand, just like a video game character translating a foreign language into English.

3. The Training (Learning to Think)

The AI didn't just memorize facts; it learned to reason.

The Dataset (Meow-10K): They fed the AI 10,831 examples of cats. Each example included a mix of video, sound, and sensor data, paired with a human expert's explanation of what the cat was actually feeling.
The Goal: Instead of just guessing "The cat is jumping," the AI was trained to guess the intent: "The cat is jumping because it is playing."
The Test (MeowBench): To see if it worked, they created a test called MeowBench. It's like a multiple-choice exam where the AI has to look at a cat's data and pick the right reason for its behavior from a list of options.

The Results: Did It Work?

Yes, and it was a big deal.

The Score: Meow-Omni 1 got 71.16% accuracy on the test.
The Competition: It beat the best existing AI models (which only looked at video or audio) by a significant margin. Even a very smart "all-rounder" AI that looked at video, audio, and sensor data as text descriptions only got 66.89%.
The Lesson: The paper proves that you can't just describe a cat's heartbeat in words; the AI needs to "feel" the raw data directly to understand the cat's true intent.

The "Hesitation" Feature (Uncertainty)

One of the coolest things the paper highlights is how the AI handles confusion.

The Scenario: Imagine a cat looks happy (video) but its sensors say it's in pain (biometrics).
Old AI: Would confidently pick one answer, probably the visual one, and be wrong.
Meow-Omni 1: When the signals conflict, the AI gets "uncertain." Its internal confidence score drops, and it essentially says, "I'm not sure, these clues don't match."
Why it matters: The paper suggests this is crucial for safety. If the AI is unsure, it can flag the situation for a human vet to check, rather than giving a dangerous wrong answer.

Summary

Meow-Omni 1 is the first AI that treats a cat's heartbeat and movement sensors as a primary language, not just background noise. By combining what the cat looks like, sounds like, and feels like internally, it can finally solve the puzzle of whether a purring cat is happy or in pain. The authors have released the code, the data, and the model for anyone to use, aiming to help vets and animal researchers understand non-verbal animals better.

Technical Summary: Meow-Omni 1

Problem Statement

The core challenge addressed by this work is semantic aliasing in computational ethology: the phenomenon where identical external signals (e.g., a cat's purr) correspond to fundamentally different internal states (e.g., contentment vs. pain) depending on the physiological context. Existing Multimodal Large Language Models (MLLMs) are largely "blind" to high-frequency biological time-series (TS) data. Consequently, they rely on superficial behavioral pattern matching (forecasting the next video frame or pitch contour) rather than genuine latent-state reasoning. Current architectures fail to natively ground visual and auditory signals in physiological reality, leading to ambiguity in intent inference.

Methodology

1. Architecture: Native Quad-Modal Integration

Meow-Omni 1 is the first open-source MLLM designed to natively fuse four data streams: text, video, audio, and biological time-series.

Backbone: Built upon the MiniCPM-o 4.5 architecture.
Model Surgery: The authors perform "transplantations" to accommodate feline biometrics:
- Vocabulary Expansion: Three control tokens (<|ts_start|>, <|ts_unit|>, <|ts_end|>) are added to the tokenizer.
- Specialized Encoders: A dedicated time-series encoder is integrated from Intern-S1 Pro, a scientific multimodal foundation model.
- Projection Layer: A custom linear projector maps extracted biological features into the LLM's joint embedding space, allowing the model to treat physiological data as native language tokens rather than disparate sensor readings.
Forward Pass: The model processes variable-length sequences containing any subset of modalities (V, A, TS, Text), interleaving TS embeddings with visual and linguistic tokens within a unified causal transformer block.

2. Theoretical Framework

The authors formalize animal intention ( $I$ ) using Pearl's structural causal models. Intention is defined not as a simple classification label, but as a latent biological and cognitive drive that maximizes the probability of a specific future action under a hypothetical "free-choice" intervention ( $do(E_{free})$ ). This approach decouples the animal's true intent from environmental constraints (e.g., a cage or handler presence) that might suppress or alter observable behavior.

3. Training Pipeline

The training follows a two-stage process to ensure modal stability:

Stage 1 (Projector Alignment): The TS projector is pre-trained on 383,853 labeled TS samples (using a Next-Behaviour Prediction strategy) while the LLM backbone and TS encoder remain frozen. This aligns biometric features with the linguistic latent space.
Stage 2 (Multimodal Specialization): The model is fine-tuned on the Meow-10K dataset (10,831 samples) using the LLM backbone only. This dataset includes diverse modality combinations (e.g., V+A, V+TS, A+TS, and full quad-modal) to teach the model to reason from any available subset of streams.

4. Datasets and Benchmarks

Meow-10K: A training dataset comprising 10,831 samples derived from multiple sources (accelerometer data, video clips, audio recordings, and synchronized pairs). It features natural-language query-response pairs and labels from a unified 30-class intention taxonomy.
MeowBench: A novel, expert-verified evaluation benchmark containing 527 high-fidelity samples. Since naturally synchronized quad-modal data is scarce, samples were synthesized by matching unimodal data (video-audio and TS) sharing the same intention label, verified by professional feline ethologists. The benchmark uses a Multiple Choice Question (MCQ) format to test discriminative accuracy.

Key Contributions

Meow-Omni 1 Architecture: The first native MLLM to unify visual, auditory, and biological time-series modalities for joint behavioral reasoning.
Meow-10K Dataset: A diverse, multi-source dataset of over 10,000 feline samples spanning varying modality combinations with natural-language descriptions.
MeowBench: The first benchmark specifically designed to evaluate MLLMs on cross-modal reasoning between biometrics, vision, and audio for intent decoding.
Open Ethological AI Framework: A complete open-source pipeline (model weights, training code, and data) demonstrating how foundation models can be adapted for non-human species.

Results

Evaluated on MeowBench, Meow-Omni 1 achieved a Top-1 accuracy of 71.16%, outperforming all baselines:

Unimodal Baselines: Significantly outperformed specialized acoustic (36.86%), video (61.95%), and TS (48.98%) models.
Multimodal Baselines: Surpassed the leading general-purpose omni-modal model, Qwen3.5-Omni-Plus (66.89% with V+A+TS), which treats non-linguistic signals with generic encoders.
Ablation Study: The inclusion of biological TS data provided the most significant performance gain. While Vision alone achieved 69.97%, adding TS (V+TS) boosted performance to 70.82%, and the full quad-modal integration reached 71.16%. This confirms that biological markers help resolve semantic aliasing inherent in purely visual or auditory displays.
Uncertainty Quantification: Using temperature sampling, the model demonstrated the ability to detect conflicting modalities. On congruent data, predictive entropy was low (1.28 bits), indicating high confidence. On adversarial conflict data (where visual cues suggested contentment but TS indicated pain), entropy rose sharply (3.15 bits), indicating the model correctly identified ambiguity rather than defaulting to the strongest modality.

Significance and Claims

The paper claims that Meow-Omni 1 establishes a scalable paradigm for inter-species intent understanding. Its primary significance lies in:

Resolving Semantic Aliasing: Demonstrating that true semantic understanding of animal behavior requires grounding observational data in physiological reality, moving beyond superficial pattern matching.
Clinical and Conservation Utility: Providing a practical tool for veterinary diagnostics (e.g., detecting occult pain) and wildlife conservation by bridging the sensory divide within foundation models.
Safety and Interpretability: Introducing a mechanism for uncertainty quantification that allows the system to flag ambiguous states, deferring to human experts in high-stakes scenarios.

The authors position this work as a foundational step toward next-generation veterinary diagnostics and ethological research, designed initially for domestic cats (Felis catus) but intended as a template for broader inter-species communication.