💻 computer science

Falcon Perception

The paper introduces Falcon Perception, a unified dense Transformer that replaces traditional modular encoder-decoder pipelines with a single early-fusion architecture to simultaneously handle perception and task modeling, demonstrating superior performance in mask prediction and OCR tasks through simplified design and specialized training signals.

Original authors: Aviraj Bevli, Sofian Chaybouti, Yasser Dahou, Hakim Hacid, Ngoc Dung Huynh, Phuc H. Le Khac, Sanath Narayan, Wamiq Reyaz Para, Ankit Singh

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aviraj Bevli, Sofian Chaybouti, Yasser Dahou, Hakim Hacid, Ngoc Dung Huynh, Phuc H. Le Khac, Sanath Narayan, Wamiq Reyaz Para, Ankit Singh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to look at a messy room, listen to your instructions, and then point out exactly where specific items are, even drawing a perfect outline around them.

Traditionally, engineers built robots with a "Two-Person Team" approach:

The Photographer (Encoder): A specialist who takes a picture of the room and creates a detailed map of everything they see.
The Detective (Decoder): A separate specialist who looks at that map and tries to guess what you asked for.

The problem? The Photographer and the Detective don't talk to each other until the very end. If the Photographer misses a tiny detail, the Detective can't fix it. If the Detective gets confused by a complex instruction (like "the red cup behind the blue book"), they have to guess based on a static map, not the live scene.

Falcon Perception is a new robot that says: "Why have two people? Let's have one super-smart brain that does everything at once."

Here is how it works, broken down with simple analogies:

1. The "All-in-One" Brain (Early Fusion)

Instead of a Photographer and a Detective, Falcon Perception is a single, unified brain.

How it works: It looks at the pixels of the image and the words of your sentence at the exact same time, mixing them together from the very first moment.
The Analogy: Imagine a chef who doesn't just chop vegetables (vision) and then add spices (text) later. Instead, the chef tastes the sauce while chopping the onions. The flavor of the onion changes how they chop, and the spice changes how they taste. This allows the robot to understand complex relationships, like "the cat under the table," much better than the old two-step method.

2. The "Chain of Perception" (Thinking Before Acting)

When you ask the robot to find something, it doesn't just guess the outline immediately. It follows a strict, logical order, like a detective solving a crime:

Where is it? (Coordinates)
How big is it? (Size)
What does it look like? (The Mask/Outline)

The Analogy: Think of drawing a portrait. You don't start by coloring the eyes. First, you draw the outline of the head (Where?), then you decide the face shape (Size?), and then you fill in the details (Mask?).
Why it helps: By forcing the robot to figure out "where" and "how big" before it tries to draw the outline, it stops getting confused. It knows exactly which object it is talking about before it tries to paint it.

3. The "Specialized Paintbrushes" (Lightweight Heads)

Even though the brain is one big unit, the robot has different "tools" for different jobs.

The Analogy: Imagine a Swiss Army Knife. The main handle is the brain (the Transformer), but it has a screwdriver for screws, a knife for cutting, and a corkscrew for bottles.
In the paper: The robot uses a "Fourier Feature" tool to measure coordinates (like a super-precise ruler) and a "Dot Product" tool to paint the mask (like a spray painter that instantly fills in the shape). This keeps the robot fast and efficient, even though it's doing complex math.

4. The "Crowded Room" Test (PBench)

The researchers created a new test called PBench to see if the robot could handle hard situations.

The Levels:
- Level 1: "Find the cup." (Easy)
- Level 2: "Find the cup with the star on it." (Attributes)
- Level 3: "Find the cup that says 'Starbucks'." (Reading text/OCR)
- Level 4: "Find the cup on the left of the book." (Spatial logic)
- Level 5: "Find the cup holding the spoon." (Relationships)
The Result: Old robots (like SAM 3) were great at Level 1 but got confused at Level 4 and 5. Falcon Perception excelled at the hard levels because its "brain" understood the relationship between the words and the image instantly.

5. The "Magic Trick" (Sampling)

Sometimes, the robot's first guess isn't perfect. But because it thinks in probabilities (like a human guessing), it has many possible answers in its head.

The Analogy: If you ask a human "Where is the cat?", they might say "Maybe on the sofa, maybe under the bed." If you let them check 8 different possibilities, they are almost guaranteed to find the right one.
The Result: The researchers found that if they let the robot generate 8 different guesses and pick the best one, its accuracy skyrocketed, beating even much larger, more expensive models.

6. The "Compact OCR" (Falcon OCR)

They also took this same "all-in-one brain" and shrunk it down to a tiny size (300 million parameters) to read text from documents.

The Analogy: Usually, reading a messy document requires a giant supercomputer. Falcon OCR is like a smartphone app that can read a messy, handwritten, or printed document just as well as the supercomputer, but it fits in your pocket and runs incredibly fast.

The Big Takeaway

The old way of building AI was like building a factory with many separate assembly lines. If one line broke, the whole thing stopped.

Falcon Perception is like a single, self-driving car. It sees the road, understands the traffic signs, and steers the wheel all at once. It's simpler, it's faster, and it handles complex, crowded, and confusing situations much better than the old "two-person team" robots.

In short: They proved you don't need a complicated, multi-step machine to understand the world. You just need one smart brain that learns to see and speak at the same time.

1. Problem Statement

Current perception-centric systems for open-vocabulary segmentation and OCR typically rely on a modular encoder-decoder pipeline. In this architecture, a vision backbone extracts features, which are then passed to a separate decoder or late-fusion module for task prediction. The authors argue that this separation creates bottlenecks:

Limited Interaction: It restricts deep interaction between visual and textual modalities, often confining them to late-stage cross-attention.
Complexity: It necessitates task-specific mechanisms (e.g., query matching, post-processing) that complicate scaling.
Dense Prediction Challenges: Handling dense outputs (e.g., hundreds of masks in crowded scenes) with autoregressive decoding is computationally expensive and prone to drift or duplication.

The central question posed is: Can a single early-fusion stack perform both perception and task modeling at scale without an encoder-decoder split?

2. Methodology: Falcon Perception

Falcon Perception introduces a unified dense Transformer that processes image patches and text tokens in a shared parameter space from the first layer.

A. Architecture

Unified Early-Fusion Stack: The model uses a single Transformer backbone ( $f_\theta$ ) for both visual and textual inputs.
Hybrid Attention Mask:
- Image Tokens: Attend bidirectionally to all other image tokens to build global visual context (encoder-like behavior).
- Text/Task Tokens: Attend causally to the full visual prefix and preceding text tokens (decoder-like behavior).
- This allows the same weights to function as a bidirectional vision encoder and an autoregressive language decoder.
Chain-of-Perception (CoP): To balance expressivity and inference speed, the model decomposes instance segmentation into a structured sequence:
1. <coord>: Predicts center coordinates.
2. <size>: Predicts object dimensions.
3. <seg>: Predicts the segmentation mask.
- This coarse-to-fine curriculum forces the model to resolve spatial ambiguity before committing to pixel-level details.

B. Specialized Heads & Decoding

While the backbone is shared, the output heads are specialized to handle dense continuous outputs efficiently:

Coordinate & Size Heads: Instead of standard tokenization, the model uses Fourier Feature mappings to project continuous coordinates into a high-dimensional space. This overcomes spectral bias and improves spatial precision.
Segmentation Head: The <seg> token acts as a query. The mask is generated via a dot product between the <seg> token's hidden state and upsampled visual features.
- Content-Aware Upsampling: A cross-attention module upsamples features using the high-resolution input image as a guide, allowing the model to "paint" semantic features onto the pixel grid without heavy mask encoders or Hungarian matching.

C. Training Strategy

Multi-Teacher Distillation: Weights are initialized by distilling knowledge from DINOv3 (for local features) and SigLIP2 (for language-aligned features) to provide a robust visual foundation.
Data Curation: A massive dataset of 54M images with 195M positive expressions and 488M negatives, curated via hierarchical clustering, VLM listing, and human verification loops to ensure semantic granularity.
Three-Stage Training:
1. In-Context Listing: Learning to list scene inventory (building global context).
2. Task Alignment: Masking inter-query attention to force independent grounding for each prompt.
3. Long-Context Finetuning: Adapting to extreme density (up to 600 instances per image).
Optimization: Uses the Muon optimizer for specialized heads and Gram Loss to preserve visual feature structure during training.

3. Key Contributions

Falcon Perception Model: A unified, early-fusion Transformer that eliminates the encoder-decoder split, achieving state-of-the-art performance in open-vocabulary segmentation with a simpler architecture.
Chain-of-Perception Interface: A novel autoregressive sequence format (<coord> $\to$ <size> $\to$ <seg>) that stabilizes training and enables efficient, parallel mask generation.
PBench (Perception Benchmark): A new benchmark designed to evaluate compositional capabilities rather than just overall accuracy. It categorizes prompts into five complexity levels:
- Level 0: General objects.
- Level 1: Attributes (color, size).
- Level 2: OCR-based identification.
- Level 3: Spatial relationships (left/right, depth).
- Level 4: Relational grounding (holding, wearing).
- Dense Stress Test: Evaluates performance on crowded scenes with hundreds of instances.
Falcon OCR: An extension of the architecture to a compact 300M-parameter model for document understanding, achieving strong OCR performance without the need for massive parameter counts.

4. Results

A. Open-Vocabulary Segmentation (SA-Co Benchmark)

Falcon Perception achieves a Macro-F1 of 68.0, outperforming SAM 3 (62.3) and other state-of-the-art models like OWLv2 and DINO-X.
It shows significant gains in specific domains (e.g., Food & Drink: 70.3 vs 58.1; Sports: 75.2 vs 71.2).
Limitation: It lags slightly in "presence calibration" (MCC) compared to SAM 3 due to the lack of a fixed-query empty-class mechanism, though this is being addressed via reinforcement learning.

B. PBench Performance

Falcon Perception demonstrates superior robustness as prompts become more complex.
Level 3 (Spatial): +21.9 point gain over SAM 3.
Level 2 (OCR): +13.4 point gain over SAM 3.
Dense Scenes: It significantly outperforms large VLMs (e.g., Qwen3-VL-30B) in crowded environments (72.6 vs 8.9), proving its ability to handle long-context generation without hallucination or instance merging.

C. Falcon OCR Results

olmOCR: Achieves 80.3% accuracy, matching or exceeding much larger models like Gemini 3 Pro and PaddleOCR VL 1.5.
OmniDocBench: Achieves an Overall score of 88.64, ranking highly among top-tier systems despite being only 300M parameters.
Efficiency: Processes ~3000 tokens/second on a single GPU, making it highly suitable for large-scale document digitization.

D. Sampling & Initialization

Pass@k: Stochastic decoding (sampling) significantly boosts performance. On SA-Co, Pass@8 improves cgF1 from 34.7 to 54.3, matching SAM 3.
Distillation: Models initialized via multi-teacher distillation significantly outperform those trained from scratch, highlighting the necessity of strong visual feature initialization.

5. Significance and Impact

Simplicity over Complexity: The paper challenges the prevailing trend of adding modular components for every new capability. It demonstrates that a single, well-architected early-fusion Transformer can handle diverse perception tasks (segmentation, OCR, spatial reasoning) effectively.
Scalability: The architecture offers a clean scaling path. Improvements can be driven by data quality, compute, and training signals rather than architectural redesigns.
Efficiency: By unifying perception and language, Falcon Perception achieves high performance with smaller parameter counts (e.g., the 300M OCR model) compared to massive generalist VLMs.
Future Direction: The work suggests that Reinforcement Learning (RL) and better sampling strategies (like Pass@k) are the next frontiers for refining autoregressive perception models, allowing them to select the best solution from a rich probability distribution.

In summary, Falcon Perception represents a paradigm shift towards unified, early-fusion dense transformers, proving that a single stack can effectively replace complex modular pipelines for high-precision visual perception tasks.