PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation

1. Problem Statement

The paper addresses the critical challenge of deploying promptable visual segmentation (specifically models based on the Segment Anything Model, or SAM) on extreme edge devices and in-sensor computing platforms.

• Constraints: Traditional SAM architectures rely on heavy Transformer backbones with quadratic complexity, massive memory footprints (hundreds of MBs to GBs), and complex operators (e.g., Softmax, LayerNorm) that are unsupported by low-power, in-sensor AI cores.
• Hardware Limitations: The target hardware, the Sony IMX500, has a strict on-chip SRAM budget (<8 MB), a limited operator set, and requires integer-only arithmetic for real-time inference. Existing lightweight SAM variants (e.g., TinySAM, EdgeSAM) still exceed these memory budgets or rely on unsupported operators.
• Latency & Privacy: Cloud-based processing introduces latency and privacy risks. The goal is to achieve real-time, on-device segmentation without offloading data.

2. Methodology

The authors propose PicoSAM3, a lightweight, promptable segmentation model specifically designed for in-sensor execution. The methodology consists of four key pillars:

A. Implicit Prompt Encoding via Centered Cropping

To overcome the IMX500's limitation of accepting only RGB inputs (no explicit point/box coordinates as network inputs), the authors developed an implicit prompting strategy:

• Training: Instead of feeding bounding box coordinates into the network, the input image is dynamically cropped around the object of interest (Region of Interest - ROI) with 10% padding.
• Mechanism: The crop is resized to a fixed 96×96 resolution. The network learns that the target object is always centered in the input, effectively treating the crop center as an implicit point prompt and the crop boundaries as an implicit box prompt.
• Inference: This aligns perfectly with the IMX500's hardware ROI capabilities, where the sensor crops the region before inference, eliminating the need for external prompt encoding modules.

B. Hardware-Aware Architecture

PicoSAM3 replaces the heavy Transformer blocks of SAM with a dense Convolutional Neural Network (CNN) architecture (U-Net style) optimized for the IMX500:

• Backbone: Uses depthwise separable convolutions with four stages.
• Enhancements:
  • Dilated Bottleneck: Expands the receptive field without additional downsampling.
  • Efficient Channel Attention (ECA): Adds lightweight channel recalibration to improve feature selectivity.
  • Refinement Head: A depthwise convolution layer for sharper boundary delineation.
• Size: The model contains only 1.37 million parameters (5.26 MB FP32), reducing to 1.31 MB after INT8 quantization.

C. Knowledge Distillation from SAM3

To bridge the performance gap between the tiny student network and large foundation models, a two-stage knowledge distillation pipeline is employed:

• Teacher: Uses SAM3 (and SAM2) as the teacher model.
• Process: The teacher generates soft mask logits based on bounding box prompts. These are pre-computed and cached to avoid running the heavy teacher during training.
• Loss Function: A composite loss combines:
  • Teacher Loss: MSE and Dice loss against the teacher's soft predictions (temperature-scaled).
  • Ground Truth Loss: Balanced BCE and Dice loss against binary masks.
  • Area Preservation: A penalty term to prevent the model from predicting trivially small masks.
• Adaptive Weighting: The balance between teacher and ground truth supervision is dynamically adjusted based on the teacher's confidence score.

D. Quantization and Deployment

• Post-Training Quantization (PTQ): The model is quantized to INT8 using Sony's Model Compression Toolkit (MCT). The CNN architecture's inherent robustness allows for near-lossless quantization.
• Deployment: The quantized ONNX model is compiled to the IMX500's .rpk format. The system runs fully in-sensor, utilizing the sensor's DSP for inference and ROI cropping.

3. Key Contributions

1. First In-Sensor Promptable Segmentation: Demonstrates the feasibility of running a promptable segmentation model directly on the Sony IMX500 sensor with <8 MB memory and <1.4 M parameters.
2. Novel ROI-Based Prompting: Introduces an implicit prompting strategy via centered cropping that eliminates the need for explicit prompt channels, enabling seamless integration with hardware ROI features.
3. State-of-the-Art Efficiency: Achieves superior performance compared to existing edge baselines (TinySAM, EdgeSAM, LiteSAM) while operating at a fraction of the complexity.
4. Real-Time Performance: Achieves 11.82 ms end-to-end latency on the IMX500 (approx. 84 FPS), enabling real-time applications.

4. Experimental Results

The model was evaluated on COCO and LVIS datasets:

• Accuracy:
  • COCO mIoU: 65.45% (PicoSAM3) vs. 53.6% (SAM-H) and 51.6% (FastSAM).
  • LVIS mIoU: 64.01% (PicoSAM3).
  • Improvement: Outperforms the previous PicoSAM2 by +13.55 mIoU on COCO and +19.11 mIoU on LVIS.
• Efficiency:
  • Model Size: 1.31 MB (INT8), a 4x reduction from the FP32 version and significantly smaller than baselines (e.g., TinySAM is ~37 MB).
  • Latency: 11.82 ms on IMX500 DSP, a 17% speedup over the quantized PicoSAM2.
• Quantization Impact: INT8 quantization resulted in negligible accuracy degradation (<0.2% mIoU loss).
• Ablation Studies:
  • Distillation from SAM3 provided the highest gains (+14.5% mIoU over supervised training).
  • Switching from centered-point prompts to ROI-based cropping was the most critical factor for low-resolution performance.

5. Significance

• Paradigm Shift: PicoSAM3 proves that high-quality, promptable segmentation does not require cloud offloading or heavy GPUs. It can be executed entirely within the sensor package.
• Privacy & Latency: By processing data locally on the sensor, it eliminates bandwidth bottlenecks and privacy concerns associated with transmitting raw video streams.
• Scalability: The approach offers a blueprint for deploying complex vision tasks on battery-operated devices (e.g., smart glasses, IoT sensors, wearable AR) where power and memory are strictly constrained.
• Hardware-Software Co-Design: The work highlights the necessity of designing algorithms that respect specific hardware constraints (operator sets, memory hierarchy) rather than simply compressing existing large models.