Technical Summary: RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

1. Problem Statement

Pre-trained Vision Foundation Models (VFMs), such as the DINO, CLIP, and SigLIP families, have become central to modern computer vision due to their powerful semantic representations and generalization capabilities. However, these models typically output low-resolution feature maps due to patchification or aggressive pooling operations. This limitation hinders their effectiveness in dense prediction tasks requiring fine-grained, pixel-level reasoning, such as semantic segmentation, depth estimation, and 3D reconstruction.

Existing feature upsampling approaches face significant trade-offs:

• Traditional Interpolation: Methods like bilinear interpolation are computationally efficient but often degrade semantic fidelity, leading to boundary blurring and loss of fine details.
• Learned Upsampling: Recent task-agnostic methods (e.g., LiFT, FeatUp, LoftUp, JAFAR) improve performance but often require retraining for specific VFMs or incur heavy computational overhead at inference time.
• Universal Approaches: While recent work like AnyUp attempts to create encoder-agnostic frameworks, it relies on heavy network architectures, resulting in high parameter counts and slow inference speeds, limiting scalability and efficiency.

2. Methodology

The authors propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic framework that reconstructs high-resolution feature maps at arbitrary resolutions by lifting feature reconstruction into a geometry-aware ray domain. The architecture consists of four core components:

2.1 Spatially Decoupled Guidance Encoder

Motivated by the observation that standard $3 \times 3$ convolutional kernels in existing upsamplers exhibit anisotropic weight distributions (favoring corners over the center), RaysUp introduces a lightweight encoder that decomposes feature extraction into orthogonal directional components.

• Structure: The encoder splits the guidance feature generation into four parallel branches: a $1 \times 1$ convolution (center), $1 \times 3$ (horizontal), $3 \times 1$ (vertical), and a $2 \times 2$ dilated convolution (diagonal/corners).
• Efficiency: This design reduces parameter overhead by approximately 69.4% compared to standard $3 \times 3$ convolutions while enhancing directional interpretability and structural disentanglement.
• Output: It generates direction-aware guidance features ($F_g$) that capture multi-scale spatial semantics more effectively than raw RGB differences.

2.2 Any-Resolution Cross-Attention Mechanism

To support arbitrary target resolutions without retraining, RaysUp employs a resolution-decoupled cross-attention mechanism.

• Queries ($Q_g$): Generated at the target high-resolution ($H_{any} \times W_{any}$) via adaptive average pooling of the guidance features.
• Keys ($K_g$): Generated at the backbone low-resolution ($H_{lr} \times W_{lr}$) via adaptive average pooling.
• Values ($V$): Directly taken from the low-resolution VFM features.
• Function: This allows high-resolution queries to aggregate information from semantically aligned low-resolution features without explicit interpolation, enabling flexible cross-resolution propagation.

2.3 Ray Positional Encoding (RayPE)

To overcome the limitations of 2D local neighborhoods and inject 3D geometric priors, RaysUp introduces RayPE, inspired by Neural Radiance Fields (NeRF).

• Concept: Instead of relying on 2D Euclidean proximity, each pixel is associated with a 3D camera ray defined by its origin ($o$) and normalized direction ($d$).
• Encoding: The ray parameters are encoded using 6D Plücker coordinates. A multi-band harmonic phase encoding maps these coordinates into a high-dimensional rotational space.
• Modulation: Similar to Rotary Positional Embedding (RoPE), RayPE applies rotational modulation to the query and key features based on the 3D ray geometry. This ensures that feature aggregation respects angular consistency and underlying 3D structures, preventing interactions between geometrically unrelated rays.

2.4 Geometry-Aware Neighborhood Cross-Attention

The final reconstruction step utilizes a learnable bilateral aggregation operator.

• Mechanism: Instead of global attention, each high-resolution query aggregates information from a local $k \times k$ neighborhood in the low-resolution feature space.
• Dilation: To bridge resolution gaps, the neighborhood is defined with dilation factors based on the upsampling ratio.
• Benefit: This ensures attention is performed among rays with similar orientations, preserving geometric consistency while reducing computational complexity from global $O(H_{any}W_{any} \cdot H_{lr}W_{lr})$ to local $O(H_{any}W_{any} \cdot k^2)$.

3. Key Contributions

1. RaysUp Framework: An ultra-lightweight (0.14M parameters), task-agnostic, and VFM-agnostic framework capable of upsampling backbone features to arbitrary resolutions.
2. Spatially Decoupled Guidance Encoder: A novel encoder design that enhances anisotropic modeling and reduces parameters by ~69% compared to standard convolutions.
3. Ray Positional Encoding (RayPE): A method that injects implicit 3D geometric priors via 6D Plücker ray coordinates, extending classical Joint Bilateral Upsampling (JBU) into a geometry-aware reconstruction process that mitigates structural drift and boundary blurring.
4. Efficiency and Performance: The framework achieves state-of-the-art performance across dense prediction tasks while using only ~16% of the parameters of the leading baseline (AnyUp) and delivering approximately 7× faster inference.

4. Experimental Results

Extensive experiments were conducted on diverse datasets (COCO-Stuff, ADE20K, Pascal-VOC, Cityscapes, NYUv2, DAVIS) and various VFMs (DINOv2, DINOv3, SigLIP2, PE Spatial).

• Task-Agnostic Performance: RaysUp achieved the best or second-best performance across most metrics in semantic segmentation, depth estimation, surface normal estimation, video object segmentation, and open-vocabulary segmentation. Notably, it showed superior geometric consistency in depth and normal estimation tasks.
• VFM-Agnostic Generalization: When tested on unseen VFMs (including DINOv3 and SigLIP2) and different model scales (ViT-S, ViT-M, ViT-L), RaysUp consistently outperformed AnyUp, demonstrating robust generalization without retraining.
• Efficiency:
  • Parameters: 0.14M (vs. 0.87M for AnyUp).
  • Inference Speed: At 224×224 resolution, RaysUp achieves 55 FPS compared to 11 FPS for AnyUp.
  • Scalability: RaysUp remains runnable at extreme resolutions (2K×2K) where other methods (LoftUp, JAFAR, AnyUp) encounter Out-Of-Memory (OOM) errors or significant slowdowns.
• Ablation Studies: Experiments confirmed the necessity of the Spatially Decoupled Guidance Encoder and RayPE. Removing RayPE resulted in a significant performance drop, validating the importance of 3D geometric priors.

5. Significance and Claims

The paper claims that RaysUp establishes a new standard for the accuracy-efficiency trade-off in universal feature upsampling. By revisiting the classical Joint Bilateral Upsampling paradigm and reformulating it with geometry-aware ray representations, RaysUp successfully balances high semantic fidelity, geometric consistency, and computational efficiency.

The authors position RaysUp as a practical and scalable solution for modern computer vision pipelines, enabling the use of frozen or fine-tuned VFMs in downstream tasks that require high-resolution, pixel-level understanding without the computational burden of heavy upsampling networks. The work highlights that integrating 3D geometric priors (via ray representations) into 2D feature upsampling is a critical step toward improving structural fidelity and reducing artifacts in dense prediction tasks.