Frequency and Edge-Guided Segment Anything Model for Remote Sensing Image Semantic Segmentation
This paper proposes FE-SAM, a scalable framework for remote sensing image semantic segmentation that enhances the Segment Anything Model by introducing a Frequency-Modulated Adapter to adaptively decompose frequency features and an EGRefiner to integrate multi-scale edge information, thereby overcoming feature adaptation limitations and boundary ambiguity to achieve state-of-the-art performance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Satellites and aircraft constantly capture vast, detailed photographs of our planet, revealing everything from sprawling cities to remote forests. Making sense of these images requires a process called semantic segmentation, where a computer is taught to identify and outline every distinct object it sees, such as a specific building, a stretch of road, or a patch of trees. For years, scientists have relied on specialized computer programs to perform this task, but these tools often struggle when faced with the unique complexity of the Earth's surface. A major breakthrough in computer vision recently introduced a powerful, general-purpose model trained on millions of everyday photographs. While this model is incredibly good at recognizing objects in standard pictures, it falters when applied to aerial views because the textures, patterns, and lighting of the ground look very different from the street-level scenes it was originally taught to understand.
Researchers have been working to bridge this gap, trying to adapt these powerful general models to the specific needs of remote sensing without having to retrain them from scratch. The challenge lies in the fact that the ground is a chaotic mix of smooth fields, jagged rooftops, and winding roads, each with its own visual signature. A standard model might confuse a flat roof with a paved road or miss the sharp corners of a building entirely, blurring the edges that are crucial for accurate mapping. To solve this, a team of scientists has developed a new approach that teaches the model to "listen" to the hidden frequencies within an image and to pay closer attention to the sharp lines that define the world around us.
The researchers, led by Feng Gao and his colleagues, created a system they call FE-SAM, which stands for Frequency and Edge-guided Segment Anything Model. Their work focuses on two specific problems that have plagued previous attempts to use these general models for aerial photography. First, the models often fail to adapt to the diverse types of land cover found in remote sensing images. Second, they tend to lose the fine details at the boundaries of objects, resulting in fuzzy outlines that make it hard to distinguish where one object ends and another begins. To fix this, the team designed two new components that work alongside the existing model. The first is a frequency modulator, which acts like a sophisticated filter. Instead of treating an image as just a flat picture, this component breaks it down into its underlying frequency patterns. It recognizes that some parts of an image, like a dense city block, are full of rapid, high-frequency changes, while others, like a large field, are smoother and dominated by low-frequency patterns. By analyzing the energy of these patterns, the system can automatically decide which parts of the image need more attention and adjust its understanding accordingly, ensuring it treats a forest differently from a highway.
The second component is an edge-guided refiner, which acts as a sharpener for the final result. Because the main model processes images in large chunks to save time, it often loses the tiny, precise details needed to draw a perfect line around a car or a house. The new refiner takes the original image and looks for the sharp edges and structural lines that the main model might have missed. It then feeds this extra information back into the system, effectively telling the model, "Look here, the edge is actually this sharp, not that blurry." This allows the system to reconstruct the fine details of the landscape, ensuring that the boundaries of buildings, roads, and trees are drawn with high precision.
The team tested their new system on three different sets of real-world satellite and aerial images, covering a variety of landscapes from urban centers to rural areas. The results showed that their method consistently outperformed the best existing techniques. On the test datasets, the new system achieved higher accuracy in identifying and outlining objects than any other method currently available. It was particularly successful at handling difficult situations, such as distinguishing between a road and a building roof that look similar in color, or separating individual cars from the background noise. The researchers also found that their system was efficient, adding only a small amount of extra computing power to the process while delivering a significant boost in performance. By combining a deep understanding of the image's frequency patterns with a keen eye for structural edges, the team has created a tool that can map the Earth's surface with a level of detail and accuracy that was previously difficult to achieve. This advancement suggests that we can now rely on these powerful computer vision tools to provide clearer, more reliable maps for everything from urban planning to disaster monitoring, turning complex aerial data into actionable information with greater confidence.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.