Technical Summary: FOVI – A Biologically-Inspired Foveated Interface for Deep Vision Models

1. Problem Statement

Processing visual data at native high resolutions presents severe computational challenges for deep learning models. While standard computer vision often relies on low-resolution inputs (e.g., 224x224), many real-world applications—such as humanoid robotics, autonomous driving, and broad scene processing—require sensitivity to large fields of view (FOV) at high resolutions.

For transformer-based architectures (ViTs) utilizing self-attention, increasing image resolution imposes doubly quadratic costs:

1. Pixel Count: The number of pixels increases quadratically with side length.
2. Attention Complexity: The self-attention mechanism scales quadratically with the number of patches ($O(N^2)$), as each patch attends to all others.

Existing solutions often simplify the problem by downsampling, which discards critical high-frequency detail. Conversely, prior attempts at foveated sensing (log-polar models or warped Cartesian approaches) often result in locally anisotropic sampling, producing non-biologically plausible receptive field shapes and undesirable warping. There is a lack of a general-purpose, biologically plausible foveated interface that can be adopted across diverse architectures (CNNs and ViTs) while maintaining efficient, isotropic sampling.

2. Methodology

The authors propose FOVI (Foveated Vision Interface), a system inspired by the human retina and primary visual cortex (V1). The core innovation is reformating a variable-resolution, retina-like sensor array into a uniformly dense, V1-like sensor manifold.

2.1. The Sensor Manifold and Sampling

FOVI utilizes the mathematical model of retino-cortical mapping (Rovamo & Virsu, 1984).

• Cortical Magnification Function (CMF): Sampling density is determined by $M(r) = \frac{1}{r+a}$, where $r$ is eccentricity and $a$ is a parameter controlling the strength of foveation. As $a \to \infty$, sampling becomes uniform; smaller $a$ values indicate stronger foveation.
• Isotropic Sampling: The system integrates the CMF to derive a "cortical" dimension. It then samples this dimension uniformly and calculates corresponding retinal radii. Angular samples are determined to enforce local isotropy (ensuring the distance between neighboring angles equals the distance between neighboring radii at any point).
• Manifold Representation: The resulting points form a 3D sensor manifold. By cutting this manifold along the vertical meridian and flattening it, the system creates a 2D representation that mirrors the retinotopic organization of human V1 hemispheres.

2.2. kNN-Convolution and Kernel Mapping

To perform standard convolutional operations on this non-uniform manifold, the authors introduce kNN-convolution:

• Receptive Fields: Defined as $k$-nearest-neighborhoods (kNNs) around output units on the sensor manifold.
• Kernel Mapping: A novel technique maps a learned reference kernel (trained on a standard Cartesian grid) into each kNN neighborhood.
  • Step 1: Define polar neighborhood coordinates relative to the output unit.
  • Step 2: Transform these to Cartesian coordinates to align with the reference kernel's frame.
  • Step 3: Spatially sample the reference kernel using these coordinates.
• Resolution: The reference kernel can be set at a higher resolution ($s = 2\sqrt{k}$) than the neighborhood size to better accommodate idiosyncratic spatial positions, acting as an anti-aliasing mechanism.

2.3. Architectural Implementations

The paper demonstrates two distinct implementations:

1. FOVI-CNNs: A hierarchical architecture (AlexNet-like) where each layer's activation map is a sensor manifold. Downsampling is performed via max-pooling, and kNNs are defined for the subsequent layer.
2. FOVI-ViTs: A foveated adaptation of the DINOv3 foundation model.
   • Patchification: Patches are defined as kNNs over the sensor array.
   • Adaptation: The pre-trained DINOv3 model is adapted using Low-Rank Adaptation (LoRA) applied to the patch embedding layer and the first half of the transformer layers. This allows the model to integrate the new sensor interface while retaining pre-trained visual features and minimizing overfitting.

3. Key Contributions

1. FOVI Interface: A new foveated vision interface that samples visual space according to a retino-cortical mapping model, producing a uniformly dense sensor manifold.
2. kNN-Convolution: A novel convolutional mechanism with kernel mapping that enables standard deep learning operations (weight sharing, orientation alignment) on foveated, non-Euclidean inputs.
3. Biological Plausibility: The architecture naturally produces receptive fields (RFs) that scale linearly with eccentricity, matching the spatial characteristics of primate neural RFs (unlike log-polar or warped Cartesian methods which produce anisotropic RFs).
4. Foundation Model Adaptation: A successful protocol for adapting large-scale pre-trained ViTs (DINOv3) to foveated inputs using LoRA, achieving high performance with reduced computational budgets.

4. Results

4.1. FOVI-CNN Performance

• Receptive Field Properties: The model exhibits RF sizes that increase linearly with eccentricity, matching human V1-V3 data. This dependence is abolished in non-foveated variants.
• Classification Accuracy: On ImageNet-1K, models with intermediate foveation ($a=0.5$) outperformed both strong foveation and non-foveated (uniform) controls when constrained to a limited "pixel budget" (sampling 64x64 pixels from a 256x256 image).
• Fixation Strategy: Performance improves significantly with the number of fixations, saturating around 20 fixations.

4.2. FOVI-ViT Performance

• Efficiency vs. Accuracy: The FOVI-ViT-H+ variant (using 64 patches, ~1/16th the pixels of a 224x224 uniform baseline) achieved 84% top-1 accuracy on ImageNet-1K with a single fixation. This represents >96% of the accuracy of the full-resolution ViT-H+ baseline while using approximately 1/3 of the GFLOPS.
• Comparison to Baselines: FOVI models significantly outperformed:
  • Uniform downsampling to 64x64.
  • "Weak" FOVI (near-uniform sampling).
  • Log-polar foveated models (which struggled due to distribution shifts from pre-training).
• Scalability: Analysis shows that while non-attention operations scale roughly quadratically ($O(m^{1.76})$), attention operations scale as $O(m^4)$. FOVI effectively mitigates the $O(m^4)$ bottleneck by reducing the effective sequence length for high-resolution inputs.

5. Significance and Claims

The paper claims that FOVI provides a general-purpose, biologically plausible solution to the computational bottleneck of high-resolution vision.

• Efficiency: It opens pathways for efficient active sensing in high-resolution, egocentric vision tasks (e.g., robotics, autonomous driving) by drastically reducing pixel counts and computational cost without sacrificing performance.
• Flexibility: Unlike previous methods restricted to specific architectures or suffering from anisotropic artifacts, FOVI can be integrated into both CNNs and Transformers (including foundation models).
• Biological Fidelity: It is the first approach to accurately model the locally isotropic and eccentricity-dependent nature of human foveation in a way that is compatible with end-to-end deep learning.
• Future Potential: The authors suggest FOVI is particularly valuable for scenarios requiring high-resolution processing where computational constraints are critical, and for computational modeling of human peripheral vision phenomena (e.g., crowding, saccade guidance).

The authors remain modest, noting limitations such as the computational overhead of kNN convolution compared to standard 2D convolution and the current evaluation being limited to image classification rather than detection or segmentation. They emphasize that the primary contribution is the design of the interface and its integration mechanism.