🤖 AI

MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation

MPerS is a novel remote sensing scene segmentation framework that leverages dynamic Mixture-of-Experts prompts to generate high-quality captions from multiple MLLMs, which are then adaptively integrated via a Dynamic MixExperts module and Linguistic Query Guided Attention to guide DINOv3-extracted visual features for superior segmentation performance.

Original authors: Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a high-resolution aerial photo of a city from a plane. To a computer, this is just a grid of millions of colored dots. To a human, it's a neighborhood with houses, trees, cars, and roads. The challenge for computers is to not just "see" the dots, but to understand exactly where every single object is and what it is. This is called semantic segmentation.

The paper introduces a new method called MPerS (Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation). Here is how it works, explained simply:

1. The Problem: The "Blind" Computer and the "Hallucinating" AI

Traditionally, computers try to guess what is in a satellite image just by looking at the pixels. It's like trying to identify a fruit in a dark room just by feeling its shape; you might guess an apple, but you could be wrong.

To help, researchers started using AI that can "talk" (Large Language Models or MLLMs) to describe the image. However, if you just ask a standard AI, "What's in this picture?", it might give you a vague or even wrong answer (like saying "cars are parked in a tree"). If the computer's description is wrong, its map of the city will be wrong, too.

2. The Solution: A Panel of Expert Describers

The authors realized that to get the best map, you need the best description. Instead of asking one AI for a simple description, MPerS acts like a panel of three different expert detectives (using models like LLaVA, ChatGPT, and Qwen).

The Multi-Perspective Prompts: Instead of just asking "What do you see?", the system asks these experts three specific types of questions:
1. What is there? (List the categories like cars, trees, buildings).
2. How much is there? (Estimate the percentages, e.g., "40% is concrete, 10% is trees").
3. Where is it? (Describe the relationships, e.g., "The cars are near the buildings").
The Quality Check: The system doesn't just trust the AI blindly. It has a "fact-checker" step. If the AI says something that doesn't match the image (like cars in a tree), the system rejects that description and asks the AI to try again until it gets a high-quality, accurate caption.

3. The "MixExperts" Gating Network: The Smart Manager

Now, the system has three different descriptions from three different AI experts. Which one should the computer listen to?

Imagine a restaurant manager (the Gating Network) standing in front of three chefs.

Chef A is great at describing buildings.
Chef B is great at spotting small cars.
Chef C is great at describing trees.

The manager doesn't just pick one chef. Instead, the manager looks at the specific part of the image being analyzed. If the computer is looking at a parking lot, the manager says, "Listen mostly to Chef B." If it's looking at a forest, "Listen mostly to Chef C." This dynamic mix ensures the computer gets the best possible description for every part of the image.

4. The "Guided Attention": The Flashlight

Once the computer has the perfect description, it uses a special tool called Linguistic Query Guided Attention.

Think of the visual features (the pixels) as a dark room full of objects. The text description acts like a flashlight.

If the text says, "There is a red car on the left," the flashlight shines brightly on the left side of the image, telling the computer, "Look here! This is a car!"
This helps the computer ignore the background noise and focus precisely on the objects mentioned in the text, drawing a much cleaner and more accurate map.

5. The Result: A Perfect Map

The system combines the visual "eyes" (using a powerful vision model called DINOv3) with the "flashlight" of the text descriptions.

The paper tested this on three different real-world datasets (Vaihingen, Potsdam, and SynDrone), which are like different neighborhoods with varying densities of houses and trees.

The Outcome: MPerS created more accurate maps than previous state-of-the-art methods.
Why it matters: It was particularly good at finding small, tricky objects like individual cars or small patches of vegetation, which other methods often miss or confuse.

Summary Analogy

If traditional computer vision is like a person trying to draw a map from a blurry photo, MPerS is like giving that person a high-definition photo and a team of expert tour guides who point out exactly where every street and building is, while a smart manager ensures the guides are only talking about the parts of the map the person is currently drawing. The result is a perfect, detailed map.

Technical Summary: MPerS – Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation

1. Problem Statement

Semantic segmentation in remote sensing (RS) scenes is critical for various interpretation applications but faces significant challenges due to the high resolution, extensive spatial coverage, and dense distribution of land-cover objects. Traditional approaches relying solely on single-modality imagery often require costly, time-consuming pixel-level manual annotations and suffer from limited perceptual capabilities. While multimodal methods integrating textual semantic information have emerged, existing studies predominantly focus on architectural optimizations for fusing text and visual features. They largely neglect two critical aspects:

The generation of high-quality RS captions: Simple prompts often lead to hallucinated elements or a lack of fine-grained spatial details, resulting in erroneous perceptual understanding.
The investigation of caption effectiveness: There is insufficient research on how to select and integrate the most effective textual semantics from multiple Large Language Model (LLM) sources to guide downstream segmentation tasks.

2. Methodology

The authors propose MPerS (Dynamic MLLM Mixture-of-Experts Perception-Guided Remote Sensing Scene Segmentation), a framework designed to leverage diverse MLLM experts to perceive RS scenes and guide visual segmentation. The workflow consists of three main stages:

A. High-Quality RS Caption Generation

To overcome the limitations of simple descriptive prompts, the authors design a multi-perspective prompting strategy. Instead of generic inquiries, they analyze the RS scene to generate three specific types of prompts for MLLMs:

Existing Category Analysis: Identifying specific land-cover classes present.
Proportion Analysis: Estimating the percentage coverage of each class.
Positional Relationship Analysis: Describing spatial relationships and locations.

These prompts are fed into multiple MLLMs (specifically LLaVA, ChatGPT, and Qwen) to generate diverse caption sets. A Caption Sentences Check Strategy is employed to validate effectiveness:

A similarity matrix is constructed between the generated captions and the corresponding RS images using CLIP encoders.
A threshold ( $\tau = 0.55$ ) is set; captions scoring below this are regenerated.
Validated captions must contain three key elements: object number, category, and location relationships.

B. Dynamic MixExperts Text Encoder (DMTE)

Recognizing that different MLLMs possess varying domain expertise, MPerS employs a Dynamic MixExperts mechanism to adaptively integrate textual semantics.

Multiple sets of text tokens are generated from the validated captions of different MLLMs.
A Gating Network dynamically calculates routing weights ( $G_m$ ) for each expert based on the current scene context.
The final text token ( $T_{MixExperts}$ ) is a weighted sum of the expert tokens, ensuring that the most effective textual semantic information for the specific segmentation task is prioritized.

C. Linguistic Query Guided Attention (LQGA)

To fuse textual and visual features effectively, a Linguistic Query Guided Attention module is introduced.

Visual Encoding: The framework uses DINOv3 (frozen) as the primary vision encoder, supplemented by a Lite Detail Prior Encoder (LDPE) based on CNN blocks to capture RS-specific detail priors.
Fusion Mechanism: The $T_{MixExperts}$ tokens act as Queries ( $Q_{text}$ ), while visual features serve as Keys ( $K_{vision}$ ) and Values ( $V_{vision}$ ).
This attention mechanism generates text-guided weights to modulate visual features, ensuring that textual semantics guide the visual perception without losing original visual information.
The fused features are passed through a multi-stage skip-connection decoder (U-Net architecture) to produce the final segmentation map.

3. Key Contributions

The paper outlines three primary contributions:

MPerS Framework: The first work to investigate the effectiveness of text-guided captions generated by different MLLMs specifically for aiding multimodal dense RS-Scene segmentation.
Dynamic MixExperts Text Encoder: A novel module that leverages multiple MLLMs (LLaVA, ChatGPT, Qwen) to extract and dynamically route the most effective textual semantic information, rather than relying on a single static text source.
Linguistic Query Guided Attention: A fusion mechanism designed to fully utilize textual information to guide visual features, ensuring precise segmentation in complex, dense scenes.

4. Experimental Results

The method was evaluated on three public RS datasets: Vaihingen, Potsdam, and SynDrone.

Performance Metrics: MPerS achieved state-of-the-art performance across all datasets.
- On Vaihingen, using DINOv3-LVD-0.3B weights, MPerS achieved an mIoU of 77.10% and mF1 of 86.79%, outperforming the second-best single-modality method (RS3Mamba) by 3.02% in mIoU and the latest text-guided method (SegCLIP) by 4.87% in mIoU.
- On Potsdam, MPerS achieved an mIoU of 81.04% and mF1 of 89.32%, showing significant gains in "Low vegetation" and "Tree" categories.
- On SynDrone, the method demonstrated robustness in segmenting foreground objects with few samples and small sizes (e.g., "Person" and "Obstacle"), achieving an mIoU of 72.17%.
Ablation Studies:
- Caption Quality: Multi-perspective prompts significantly outperformed simple inquiry prompts.
- MixExperts: The dynamic gating mechanism was shown to be more effective than using a single MLLM's caption.
- Components: The inclusion of the Lite Detail Prior Encoder (LDPE) and Linguistic Query Guided Attention (LQGA) provided consistent improvements in Overall Accuracy (OA), mIoU, and mF1.
Efficiency: While the inclusion of DINOv3 and CLIP increased inference time compared to lightweight CNNs (55.52ms vs. ~14-29ms for others), the authors note the speed remains acceptable for RS tasks.

5. Significance and Claims

The authors claim that MPerS provides a new solution for exploring how multimodal large models can accurately perceive RS scenes. The significance of the work lies in:

Shifting Focus from Architecture to Perception: Moving beyond architectural fusion to address the quality and effectiveness of the textual input itself.
Modular Design: The framework is modular, allowing components like CLIP, DINOv3, and the MixExperts to be replaced or iteratively updated.
Practical Application: The method demonstrates that high-quality, multi-perspective textual guidance can significantly enhance the segmentation of dense, complex land-cover objects, offering a viable path for practical RS interpretation without relying solely on massive manual annotation efforts.

The paper concludes that accurate perception and fusion of multimodal data, driven by effective caption generation strategies, better represent real-world scenes and are highly applicable to practical RS interpretation.