Technical Summary: Learning to Adaptively Allocate Gaussians for Arbitrary-Scale Image Super-Resolution

1. Problem Definition

Traditional Super-Resolution (SR) methods are typically designed for fixed integer scale factors (e.g., $\times 2, \times 4$). However, real-world applications such as VFX compositing, custom-resolution rendering, and foveated VR require upscaling to arbitrary, non-integer scales. Existing Arbitrary-Scale SR (ASR) approaches face two primary limitations:

1. Inefficiency of Continuous Representation: Implicit Neural Representations (INRs) decode continuous coordinates via MLPs, but exhaustive pixel-by-pixel querying creates severe computational bottlenecks at high resolutions.
2. Uniform Resource Allocation: Recent adaptations of 2D Gaussian Splatting (GS) for ASR (e.g., GaussianSR, GSASR) apply a uniform densification strategy. They assign the same Gaussian budget to flat backgrounds as they do to intricate textures. This ignores the inherent adaptability of Gaussians, leading to unnecessary computational overhead and memory usage while failing to concentrate resources where structural complexity demands it.

Furthermore, transitioning GS into a feed-forward ASR model is non-trivial. Standard GS optimization relies on high-resolution ground-truth gradients to drive primitive growth. In a feed-forward ASR setting, this high-resolution target is unavailable during inference; the network must autonomously predict the necessary Gaussian densification directly from the low-resolution (LR) input.

2. Methodology: QuADA-GS

The authors propose QuADA-GS (Quadtree Adaptive Gaussian Splatting), a framework that dynamically allocates Gaussian primitives based on local image complexity and the target scale. The architecture consists of four main stages:

2.1 Feature Extraction and Scale-Conditioned Refinement

The input LR image is processed by an encoder-agnostic backbone to extract features. To handle arbitrary scales, the network employs window-based tiling with resolution-invariant Gaussian embeddings. These embeddings are refined through a dual-stage attention mechanism:

• Scale-Conditioned Cross-Attention: Modulates embeddings using a learned scale embedding ($e_s$) derived from the inverse target scale ($1/s$) and gathers local image context via Window Cross-Attention.
• Gaussian Self-Attention: Ensures spatial coherence and prevents overlapping artifacts by communicating internally via Window Self-Attention with shifted-window partitioning.

2.2 Neural Routing Architecture (Dynamic Budget Allocation)

Instead of uniform upsampling, a Differentiable Neural Routing Architecture analyzes the refined latent features to determine the optimal expansion factor for each pixel.

• Discrete Routing: The router predicts a probability distribution over three upsampling levels ($L_0, L_1, L_2$), corresponding to splitting a feature into 4, 16, or 64 children (Gaussians).
• Differentiability: The Gumbel-Softmax trick and Straight-Through Estimator (STE) are used to sample discrete routing decisions while maintaining gradient flow.
• Budget Constraints: To prevent over-allocation (predicting maximum density everywhere) or under-allocation, the model utilizes a Structure Tensor computed from the ground truth during training. This generates a "structure map" representing local complexity (edges vs. textures). A budget-matching loss ($L_{budget}$) forces the router's total predicted cost to align with the global structural complexity of the target image, granting the router spatial autonomy to distribute the budget where it minimizes rendering error.

2.3 Hierarchical Pointer Convolution (HPC)

The dynamic routing creates a variable-density, non-grid topology (resembling a quadtree), which breaks standard grid convolutions. To enable efficient spatial communication across these varying densities without the memory overhead of dense operations or the complexity of sparse hash tables, the authors introduce Hierarchical Pointer Convolution (HPC).

• Pointer Grid: A lightweight, dense index grid stores scalar identifiers for features at their current hierarchical level.
• O(1) Lookup: HPC retrieves topological neighbors by sampling the Pointer Grid at the boundaries of a feature's footprint using adaptive offsets. This allows the system to gather neighbors regardless of their resolution level in constant time.
• Sparse Update: Features are updated residually by fusing gathered neighbors, enabling seamless coordination between coarse background features and fine details.

2.4 Gaussian Prediction and Rasterization

The coordinated features are decoded into 2D Gaussian primitives (position, covariance, opacity, color).

• Level-Aware Modulation: A shared MLP predicts Gaussian parameters, but physical bounds (standard deviations) are decoupled from the MLP output via level-aware modulation factors ($m_{l_i}$) to stabilize training across different expansion levels.
• Continuous Rasterization: The primitives are rasterized directly in continuous normalized space, allowing for scale-agnostic rendering of arbitrary output resolutions.
• Loss Functions: The total loss combines rendering error, the budget-matching loss, and a Routing Quality Loss ($L_{rout}$). $L_{rout}$ penalizes the router for allocating coarse levels to regions with high spatial reconstruction error, ensuring fine details are allocated where necessary.

3. Key Contributions

1. Neural Routing Architecture: A module that dynamically defines a 2D Gaussian budget and allocates it spatially based on input complexity, replacing rigid uniform densification with a quadtree-inspired topology.
2. Hierarchical Pointer Convolution (HPC): A novel operator achieving $O(1)$ complexity for topological neighbor lookup. It facilitates efficient feature coordination across non-grid topologies, bypassing the memory bottlenecks of dense or traditional sparse operators.
3. State-of-the-Art ASR Performance: QuADA-GS achieves superior texture reconstruction and perceptual fidelity while maintaining a lean memory footprint compared to dense competitors. It enables scaling to massive Gaussian counts that would cause out-of-memory failures in uniform-grid models.

4. Experimental Results

The model was trained on the DIV2K dataset and evaluated on standard benchmarks (Urban100, Set5, Set14, etc.) across scales from $\times 2$ to $\times 30$.

• Reconstruction Quality: QuADA-GS sets a new state-of-the-art in PSNR, SSIM, LPIPS, and DISTS metrics, particularly on the structure-rich Urban100 dataset. It outperforms implicit methods (LIIF, Meta-SR) and other GS-based approaches (GaussianSR, GSASR).
• Extreme Scales: Even at extreme upsampling factors ($\times 16, \times 24, \times 30$), where input signals are severely degraded, QuADA-GS remains the most competitive GS-based method, extracting superior structural information compared to implicit baselines.
• Efficiency:
  • Speed: By employing a "dense index, sparse compute" paradigm, QuADA-GS achieves approximately 61% faster inference and 35% memory reduction compared to dense upsampling baselines.
  • Comparison: While it inherently requires more VRAM than rigid-grid GSASR (due to higher total Gaussian counts for better fidelity), it is significantly faster than implicit methods which scale poorly with target resolution.
• Ablation Studies:
  • Routing: Learned Neural Routing outperforms heuristic structure maps.
  • HPC: Removing HPC blocks severely degrades quality by preventing spatial coordination.
  • Modulation: Level-aware modulation is critical for stable training across different expansion scales.
  • Backbone: RDN is shown to be more effective than EDSR for extracting the necessary high-frequency features.

5. Significance and Claims

The paper claims that QuADA-GS successfully bridges the gap between the continuous representation capabilities of Gaussian Splatting and the efficiency requirements of arbitrary-scale super-resolution. By moving away from uniform processing, the model mimics human visual attention, concentrating computational resources strictly where structural complexity demands it.

The authors emphasize that their approach breaks the rigidity of fixed-grid ASR methods. The introduction of the Hierarchical Pointer Convolution is highlighted as a critical enabler, allowing for efficient spatial communication in variable-density topologies without prohibitive memory costs. While the method improves speed and memory usage significantly over dense baselines, the authors modestly note that mitigating the absolute memory overhead remains a focus for future work, as the quadtree topology still allocates a higher total number of Gaussians than uniform grids to ensure superior fidelity.