NOIR: Neural Operator mapping for Implicit Representations

1. Problem Statement

Current medical image computing relies heavily on discrete grid-based deep learning (e.g., CNNs, Transformers) where images are represented as fixed-resolution pixels or voxels. This paradigm faces several critical limitations:

• Resolution Sensitivity: Models trained on specific grid sizes often degrade significantly when tested on images with different resolutions, voxel spacings, or interpolation artifacts.
• Discretization Artifacts: Treating inherently continuous physical signals (like MRI or CT scans) as discrete grids introduces aliasing errors and limits the model's ability to generalize across heterogeneous acquisition protocols.
• Lack of Continuity: Standard models cannot perform true "function-to-function" transformations; they map grid-to-grid, making resolution-independent inference difficult.

The paper challenges this paradigm, proposing that medical imaging tasks should be reframed as operator learning problems between continuous function spaces rather than discrete grid prediction tasks.

2. Methodology: The NOIR Framework

NOIR (Neural Operator mapping for Implicit Representations) decomposes the problem into two main stages: Continuous Function Representation and Neural Operator Mapping.

A. Continuous Function Representation (Implicit Neural Representations - INRs)

Instead of processing raw pixel grids, NOIR first embeds discrete input ($f^d$) and output ($g^d$) signals into continuous function spaces ($\mathcal{F}$ and $\mathcal{G}$) using Implicit Neural Representations.

• Architecture: It uses a shared dataset-level INR network $\phi(x; \theta)$ modulated by signal-specific parameters $\gamma_i$.
• Meta-Learning Strategy: To avoid training a separate INR for every image (which is computationally expensive), NOIR employs a meta-learning (auto-decoder) approach:
  • A hypernetwork $M_\psi$ predicts modulation parameters $\gamma_i$ from a low-dimensional latent vector $z_i$.
  • Training: An inner loop optimizes the latent vector $z_i$ for a specific sample to minimize the reconstruction error against the discrete signal. An outer loop updates the shared weights $\theta$ and hypernetwork $M_\psi$.
  • Inference: For a new signal, the shared weights are frozen, and only the latent vector $z^*$ is optimized via a few gradient steps to fit the continuous function.
• Result: This creates a resolution-independent continuous approximation of the medical image, where the latent vector $z$ encodes the signal's content regardless of the input discretization.

B. Neural Operator (NO) Mapping

Once the input and output signals are represented as continuous functions (encoded by their latent vectors $z_{in}$ and $z_{out}$), the core task becomes learning a mapping between these latent spaces.

• Operator Learning: A Neural Operator $T$ (implemented as a Multi-Layer Perceptron with residual connections) learns the mapping $z_{out} = T(z_{in})$.
• Function-to-Function Transformation: Since $z_{in}$ and $z_{out}$ represent continuous functions, the operator $T$ effectively learns a transformation between function spaces $\mathcal{F} \to \mathcal{G}$.
• Task Agnosticism: This framework applies to various downstream tasks (segmentation, completion, translation, synthesis) simply by changing the target domain's INR, while the operator learns the transformation between the latent manifolds.

C. Theoretical Guarantee: $\epsilon$-ReNO

The authors prove that NOIR satisfies the properties of a Representation Equivalent Neural Operator ($\epsilon$-ReNO). This means the operator produces consistent outputs regardless of the input discretization resolution, theoretically bounding the aliasing error. This is achieved because the operator acts on the latent representation (which is resolution-invariant) rather than the raw grid.

3. Key Contributions

1. Paradigm Shift: Challenges the standard grid-based deep learning assumption in medical imaging, proposing a continuous function-space formulation.
2. Novel Framework: Introduces NOIR, which combines Implicit Neural Representations (INRs) with Neural Operators (NOs) to enable resolution-independent medical image processing.
3. Versatility: Demonstrates applicability across diverse 2D and 3D tasks:
   • Segmentation: Binary (Chest X-ray) and Multi-label (Brain MRI).
   • Shape Completion: Reconstructing missing skull structures (3D).
   • Image Translation: MRI modality conversion (PD to T2).
   • Image Synthesis: Generating ultrasound images from label maps.
4. Theoretical Validation: Empirically validates that NOIR acts as an $\epsilon$-ReNO, maintaining performance stability across unseen resolutions where grid-based methods fail.

4. Experimental Results

The framework was evaluated on public datasets (Shenzhen, OASIS-4, SkullBreak, fastMRI) and an in-house ultrasound dataset, comparing against U-Net, ViT, Attention U-Net, FNO, and DDPM.

• Resolution Robustness:
  • Grid-based methods (U-Net, ViT, AttU-Net): Showed significant performance degradation (e.g., Dice score drops) when tested at resolutions lower than their training resolution.
  • NOIR: Maintained consistent performance across 10%, 25%, and 100% resolutions. For example, on the OASIS-4 dataset, NOIR outperformed all baselines at low resolutions and remained competitive at full resolution.
  • FNO: While more stable than CNNs, FNO still exhibited aliasing errors and performance drops at low resolutions compared to NOIR.
• Task Performance:
  • Segmentation: Achieved competitive Dice scores (e.g., 0.94 on Shenzhen) while being robust to resolution changes.
  • Shape Completion: Successfully reconstructed missing skull anatomy, though fine-grained details in complex regions (orbital) were slightly less precise than top grid-based methods.
  • Synthesis/Translation: Outperformed all baselines in PSNR for ultrasound synthesis and maintained consistent quality across resolutions for MRI translation.
• Efficiency: Inference time for NOIR (including latent optimization) was generally under 1 second, significantly faster than diffusion models (DDPM) which took ~5 seconds.

5. Significance and Future Work

• Clinical Relevance: NOIR addresses a critical clinical need for resolution independence. Medical images vary widely in acquisition protocols (different scanners, voxel sizes). NOIR allows for stable inference without retraining or complex interpolation, ensuring consistency across heterogeneous data.
• Theoretical Advancement: By satisfying the $\epsilon$-ReNO property, NOIR bridges the gap between theoretical operator learning and practical medical imaging, offering a mathematically sound alternative to grid-based approximations.
• Limitations & Future Directions:
  • Supervision: Currently requires paired data (supervised learning). Future work aims to extend NOIR to unsupervised tasks like deformable image registration.
  • Latent Dimensionality: High-dimensional latent spaces (e.g., 4096 for fastMRI) increase the complexity of the operator learning problem, requiring careful co-design of the INR and the Operator.
  • Fine Details: While robust, the method sometimes struggles with very fine-grained geometric details in shape completion tasks.

In conclusion, NOIR represents a significant step toward continuous, resolution-agnostic medical image computing, leveraging the strengths of implicit representations and neural operators to overcome the inherent limitations of discrete grid-based deep learning.