AENEAS Project: First real-time intraoperative application of machine vision-based anatomical guidance in neurosurgery

The AENEAS project demonstrates the first successful intraoperative deployment of a real-time, machine vision-based anatomical guidance system in neurosurgery, achieving reliable landmark detection with low latency to enhance surgical orientation and training.

The Gist

Imagine you are navigating a dense, foggy forest at night. You know there are specific landmarks—a giant oak tree, a flowing stream, a rocky cliff—that you need to find to stay safe and reach your destination. But in the dark, it's easy to get turned around or miss a crucial sign. Now, imagine you have a magical pair of glasses that instantly highlights these landmarks in bright neon colors the moment you look at them, whispering, "Tree here! Stream there!"

This is essentially what the AENEAS Project did for brain surgeons, but instead of a forest, they were navigating the incredibly delicate and complex "forest" inside a human nose and skull.

Here is the story of their breakthrough in simple terms:

The Problem: Navigating in the Dark

For years, surgeons performing a specific type of brain surgery (called endoscopic endonasal surgery) have had to rely entirely on their own eyes and memory to identify tiny, critical structures like nerves and blood vessels. It's like trying to solve a puzzle while wearing blinders; if you miss a piece, the consequences can be serious. While computers had gotten good at recognizing these structures after the surgery was over (like reviewing a photo album later), no one had successfully built a system that could do it while the surgery was happening in real-time.

The Solution: A Super-Sharp Digital Co-Pilot

The AENEAS team built a "digital co-pilot" for surgeons. Here is how they made it work:

1. The Brain (The AI): They taught a computer program (using a smart system called YOLOv8) to recognize anatomical structures. Think of this like showing a child thousands of pictures of cars, dogs, and trees until they can instantly spot a dog in a crowd.
2. The Speed Boost (The Engine): To make sure this "brain" could think fast enough to keep up with a moving camera, they didn't just leave it in a slow computer lab. They translated it into a high-speed language (ONNX) and gave it a turbocharger (NVIDIA TensorRT). This is like taking a regular car engine and swapping it for a Formula 1 engine so it can race without stalling.
3. The Delivery (The Glasses): They put this super-fast brain into a specialized medical computer kit (NVIDIA Clara AGX) and connected it to the surgical camera. Now, as the surgeon looks through the camera, the computer instantly draws boxes and labels on the screen, saying, "That's the optic nerve," or "That's the pituitary gland."

The Results: Fast and Accurate

When they tested this system during actual surgeries, the results were impressive:

• Accuracy: The system correctly identified the important landmarks about 56% of the time (a score of 0.56). In the world of complex medical AI, getting it right more than half the time on the very first try is a huge win.
• Speed: The system was incredibly fast. From the moment the camera saw an image to the moment the label appeared on the screen, it took less than 48 milliseconds. To put that in perspective, a human blink takes about 300 milliseconds. The computer was faster than a blink, meaning the surgeon never had to wait for the computer to catch up.

Why This Matters

This isn't just a cool gadget; it's a game-changer.

• Less Stress: It acts like a GPS for the brain, reducing the mental load on the surgeon so they can focus on the delicate work of cutting and fixing.
• Training: For young surgeons learning the ropes, it's like having a master teacher standing right next to them, pointing out the important bits in real-time.
• The Future: This is the first time a system like this has worked in a live operating room. It proves that we are moving toward a future where Artificial Intelligence is a standard, helpful partner in the operating room, making surgeries safer and more precise for everyone.

In short, the AENEAS project turned a "magic idea" into a working reality, giving surgeons a pair of super-vision glasses that never blink and always know where they are.

Technical Summary

1. Problem Statement

While offline machine learning models have demonstrated promising performance in recognizing anatomical structures for neurosurgery, a critical gap remains: no system has yet been successfully deployed for real-time, intraoperative use during live operations. Existing solutions lack the low-latency processing and robust integration required to function as a live guidance tool during complex procedures like endoscopic endonasal surgery. The challenge lies in bridging the divide between high-accuracy offline detection and the strict real-time constraints of a clinical operating room.

2. Methodology

The AENEAS project developed a complete pipeline for real-time anatomical guidance, moving from model training to clinical deployment:

• Model Architecture: The core detection system utilizes YOLOv8 (developed by Ultralytics), chosen for its balance of speed and accuracy in object detection tasks.
• Training Environment: The model was trained within the PyTorch framework using a dataset of intraoperative video frames.
• Optimization Pipeline: To ensure real-time performance on edge hardware, the trained PyTorch model was exported to ONNX (Open Neural Network Exchange) format. It was subsequently optimized using the NVIDIA TensorRT engine to maximize inference speed.
• Deployment System: The optimized model was integrated into the NVIDIA Holoscan SDK, a framework designed for real-time medical data processing. The hardware platform was an NVIDIA Clara AGX developer kit, an edge AI computing device suitable for surgical environments.
• Evaluation Metrics:
  • Accuracy: Measured using Average Precision at an Intersection-over-Union (IoU) threshold of 0.5 (AP50), comparing the model's output against manually labeled video references.
  • Latency: End-to-end delay was measured from the moment of frame acquisition to the display of the annotated output on the surgeon's screen.

3. Key Contributions

• First Clinical Deployment: This work represents the first reported intraoperative deployment of a real-time machine vision system specifically for anatomical guidance in neurosurgery.
• End-to-End Pipeline: The authors successfully demonstrated a full workflow from model training (PyTorch) to hardware optimization (TensorRT) and real-time execution (Holoscan/Clara AGX).
• Continuous Guidance: Unlike previous static or post-operative analysis tools, this system provides continuous, automated anatomical guidance directly from the surgical field during the procedure.

4. Results

The system demonstrated feasibility and reliability in a live surgical context:

• Detection Accuracy: The model achieved a mean AP50 of 0.56. This indicates reliable detection of the most critical anatomical landmarks within the transsphenoidal corridor, despite the challenging visual environment of endoscopic surgery.
• Real-Time Performance: The system maintained a low mean end-to-end latency of 47.81 ms (with a median of 46.57 ms). This sub-50ms latency is crucial for ensuring that the augmented reality overlay remains synchronized with the surgeon's movements, avoiding disorientation caused by lag.

5. Significance and Impact

The AENEAS project marks a pivotal transition in neurosurgical technology:

• Feasibility Proof: It proves that "clinical-grade" real-time AI assistance is technically feasible within the constraints of a live neurosurgical workflow.
• Clinical Benefits: By providing automated anatomical orientation, the system aims to reduce the cognitive load on surgeons, allowing them to focus more on the surgical task rather than spatial navigation.
• Training and Future Integration: The tool offers a powerful resource for surgical training and represents a foundational step toward integrating AI support into routine neurosurgical practices, potentially improving patient safety and surgical outcomes.