Rapid patient-specific neural networks for intraoperative X-ray to volume registration

Technical Summary: Rapid Patient-Specific Neural Networks for Intraoperative X-ray to Volume Registration

Problem Statement
Image-guided interventions and surgical robotics rely on the precise alignment of 3D preoperative volumes (e.g., CT, MRI) with 2D intraoperative X-ray images (e.g., fluoroscopy). However, existing 2D/3D registration methods face significant limitations in clinical deployment. Traditional intensity-based optimizers are sensitive to initialization errors, often converging to incorrect solutions if the starting pose is even a few centimeters off. Conversely, deep learning approaches typically require extensive manually labeled datasets and are constrained to the specific anatomies on which they were trained. These supervised methods struggle to generalize across diverse patient populations, pathologies, and non-standard intraoperative views, leading to out-of-distribution failures. Furthermore, the scarcity of ground-truth C-arm poses in clinical records creates a data bottleneck for training supervised models.

Methodology: The xvr Framework
The authors introduce xvr, a self-supervised framework designed to train patient-specific neural networks for rapid 2D/3D rigid registration. The approach integrates physics-based simulation with gradient-based optimization to eliminate the need for manual annotation.

1. Differentiable Synthetic Data Synthesis:
   xvr utilizes a differentiable X-ray renderer to generate synthetic training data directly from a patient's own preoperative 3D volume. By simulating C-arm poses (rotations $\alpha, \beta, \gamma$ and translations $x, y, z$) and rendering synthetic X-rays via ray casting (using trilinear interpolation or Siddon's method), the system generates unlimited training pairs with known ground-truth poses. This process also supports the rendering of specific anatomical structures if 3D label maps are available.

2. Foundation Model and Transfer Learning:
   To address the long training times associated with patient-specific models, the authors pretrain a foundation model on a corpus of over 2,000 whole-body CT and MR volumes. This model learns a population-level representation of human anatomy and C-arm pose distributions.

   • Architecture: The network uses a ResNet34 backbone to regress the C-arm pose. To handle the manifold nature of rigid transforms ($SE(3)$), the model regresses the quaternion adjugate, a continuous representation that resolves discontinuities inherent in Euler angles and standard quaternions.
   • Adaptation: For a new patient, the foundation model is finetuned on synthetic X-rays generated from that specific patient's preoperative scan. This reduces the training time from hours (required for training from scratch) to approximately 5 minutes.

3. Robustness to Domain Shifts:
   To handle variable intraoperative acquisition parameters (e.g., C-arm panning, field-of-view changes), xvr employs a geometric resampling strategy. Intraoperative X-rays are resampled to match the canonical intrinsic parameters used during training before being fed to the network. During the refinement stage, the system renders synthetic X-rays using the original, high-resolution intrinsic parameters of the clinical image.

4. Iterative Pose Refinement:
   While the neural network provides an accurate initial pose estimate, the system employs a gradient-based optimizer to refine this estimate. Using the differentiable renderer, the optimizer maximizes the similarity between the real intraoperative X-ray and the synthetic X-ray. The authors combine multiscale normalized cross-correlation (mNCC) for robustness in large error regions and gradient normalized cross-correlation (gradNCC) for precision in depth estimation, achieving millimeter-level accuracy.

Key Contributions

• Self-Supervised Patient-Specific Training: A framework that generates ground-truth training data from a patient's own scan, removing the dependency on manual landmark annotation or large curated datasets.
• Foundation Model for Rapid Adaptation: A pre-trained model capable of adapting to any anatomical region (pelvis, femur, neurovasculature, skull) in just 5 minutes of finetuning, achieving accuracy comparable to training from scratch for 12 hours.
• Differentiable Rendering Pipeline: A high-speed, differentiable X-ray renderer capable of generating thousands of synthetic images per minute, enabling both scalable pretraining and real-time refinement.
• Handling of Variable Intrinsics: A geometric resampling technique that allows a single network to handle diverse C-arm acquisition parameters without retraining.

Results
The authors evaluated xvr on five datasets spanning multiple hospitals, patient populations, and anatomical structures, including the DeepFluoro (pelvic cadaveric), Femur (cadaveric), Ljubljana (neurovascular), and internal clinical datasets (neonatal Vein of Galen malformations and adult neurosurgery).

• Accuracy: xvr significantly outperformed existing methods. On the DeepFluoro dataset, the finetuned model achieved a median Target Registration Error (mTRE) of 1.1 mm, compared to 352.2 mm for fixed initialization and 82.1 mm for supervised landmark regression.
• Generalization: The framework demonstrated robustness to extreme out-of-distribution shifts. It successfully registered intraoperative DSAs of neonatal patients with Vein of Galen malformations (a pathology absent from the adult-only pretraining corpus) after 5 minutes of finetuning, achieving a median mTRE of 2.3 mm.
• Efficiency: The system achieves high accuracy in seconds. The initial pose estimation by the finetuned network takes seconds, and the subsequent iterative refinement adds only 1–8 seconds of latency.
• Comparison: In the largest evaluation of 2D/3D registration on real fluoroscopy to date, xvr improved upon the accuracy of existing methods by an order of magnitude. It consistently achieved sub-millimeter success rates (SMSR) where other methods failed, particularly in non-standard views and complex anatomical scenarios.

Significance and Claims
The paper claims that xvr makes "pan-anatomical 2D/3D rigid registration accessible to broad clinical and research communities." By combining differentiable rendering with rapid pre-trained model finetuning, the framework addresses the data bottleneck and generalization issues that have hindered the adoption of AI in image-guided surgery. The authors emphasize that their approach enables reliable, accurate, and anatomically generic registration across diverse medical specialties without disrupting existing clinical workflows. The framework is released as open-source software to facilitate further research and clinical integration. The authors modestly note that while their evaluation is the largest and most diverse to date, larger prospective studies are required to fully validate robustness across all potential clinical scenarios.