🔬 physics

Deep learning-enhanced Lagrangian 3D Tracking of motile microorganisms

This paper presents a deep learning-enhanced Lagrangian 3D tracking method that overcomes the limitations of traditional fluorescence-based approaches by enabling faster, more accurate, and versatile long-term observation of motile microorganisms in complex media, including non-fluorescent species like magnetotactic bacteria.

Original authors: Thierry Darnige, Daniel Midtvedt, Renaud Baillou, Benjamin Perez Estay, Changsong Wu, Alex Le Guen, Giovanni Volpe, Eric Clement

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Thierry Darnige, Daniel Midtvedt, Renaud Baillou, Benjamin Perez Estay, Changsong Wu, Alex Le Guen, Giovanni Volpe, Eric Clement

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to follow a single, tiny, hyper-active firefly in a dark, crowded forest. You want to watch it for hours to see exactly how it flies, where it goes, and what it does.

The Problem:
In the past, scientists had a "flashlight" (a microscope) to see these tiny creatures (microorganisms). But the flashlight had two big problems:

The Spotlight was too small: The firefly would dart out of the light's view in a split second, and the scientist would lose it.
The Flashlight was too hot: To keep the firefly in the light, the beam had to be very bright. But this "hot light" would eventually blind the firefly (photobleaching) or even burn it (photodamage), changing how it behaved or killing it. Also, some fireflies simply can't be seen with a bright flashlight at all (like the magnetotactic bacteria, which are sensitive to light).

Traditional methods tried to solve this by using a robot arm to move the microscope, keeping the firefly in the center. But the robot's "brain" (the old algorithm) was a bit clumsy. It would guess where the firefly was based on how blurry it looked. If the forest was foggy or crowded (like in mucus or a swarm of bacteria), the robot would get confused, lose focus, and the firefly would disappear from the screen.

The Solution: The "AI Spotter"
This paper introduces a new, super-smart way to track these tiny swimmers. Instead of a clumsy robot guessing, they gave the microscope a Deep Learning Brain (a type of Artificial Intelligence).

Here is how it works, using a few analogies:

1. The "Zero-Crossing" GPS vs. The "Guessing Game"

The Old Way: Imagine trying to find the peak of a hill in the dark. You walk up, feel the ground, and if it feels flat, you think, "Maybe I'm at the top?" But you might be on a plateau or a small bump. You keep guessing and adjusting, sometimes stepping the wrong way. This is what the old algorithms did; they tried to find the "minimum blur" to guess the focus.
The New Way: The AI acts like a GPS that knows exactly where "Level Ground" is. It doesn't just guess; it looks at the image and instantly knows: "You are 5 microns too high" or "You are 2 microns too low." It uses a "zero-crossing" method, meaning it knows exactly when it has hit the perfect spot, allowing it to move the microscope with surgical precision.

2. The "Night Vision" Goggles (Brightfield Microscopy)

Usually, to track these tiny things, scientists had to paint them with glowing paint (fluorescence). But as we said, the paint fades, and the light hurts the creature.

The AI Magic: Because the AI is so good at recognizing shapes, it doesn't need the glowing paint. It can track the bacteria using Brightfield Microscopy (just normal light, like looking through a clear window).
The Magnetotactic Bacteria: Some bacteria have tiny internal magnets. If you try to paint them with fluorescent dye, the process kills their magnetism. The AI allows scientists to watch these "magnetic swimmers" in their natural state, without touching them or hurting them, just by looking at their shadows.

3. The "Crowded Party" and the "Sticky Slime"

The paper tested this AI in two very hard environments:

The Turbulent Bath: Imagine a swimming pool filled with thousands of other swimmers, all moving chaotically. It's hard to follow one person. The AI, however, was trained to ignore the crowd and lock onto the specific "tracer" particle, even when it was surrounded by optical noise.
The Mucus Maze: Imagine trying to follow a swimmer through thick, sticky slime (like the mucus in our guts). It's cloudy and uneven. Old microscopes get lost here. The AI, trained specifically on "slime" images, could see through the fog and track the bacteria as it got stuck and unstuck, revealing how it navigates this sticky world.

Why This Matters

This isn't just about taking better pictures. It's about listening to the story of the microorganism.

Long-term Stories: Because the AI doesn't blind the bacteria, we can watch them for hours, not just seconds. We can see rare events, like a bacteria changing its mind or getting trapped.
Real Behavior: Since we aren't hurting them with bright lights or toxic dyes, what we see is how they really behave in nature.
New Discoveries: By tracking these "magnetic swimmers" and "slime-navigators," scientists can finally understand how bacteria cause infections, how they move in our bodies, and how they interact with their environment in ways we couldn't see before.

In Summary:
The researchers built a microscope with a super-smart AI brain that acts like a perfect, tireless cameraman. It can follow a tiny, invisible swimmer through a chaotic, crowded, and foggy world for hours, without ever blinding or hurting the swimmer. This opens the door to understanding the secret lives of the microscopic world in ways we never thought possible.

1. Problem Statement

Understanding the behavior of individual motile microorganisms (e.g., bacteria, protozoa) in their natural environments is crucial for microbiology, ecology, and medicine. However, current microscopy techniques face significant limitations:

Limited Observation Time: Traditional tracking methods often lose the subject quickly as it moves out of the limited field of view.
Photodamage and Photobleaching: Current Lagrangian tracking (where the microscope stage moves to follow the particle) relies on fluorescence microscopy. The feedback loop uses fluorescence intensity to maintain focus. Long-term exposure causes photobleaching (signal loss) and photodamage (altering cell viability and motility).
Labeling Constraints: Not all microorganisms can be fluorescently labeled without altering their physiology. For example, magnetotactic bacteria require low-oxygen environments for magnetosome formation, which conflicts with the high-oxygen conditions often needed for GFP expression.
Optical Complexity: In crowded environments (e.g., biofilms, mucus, active bacterial suspensions), background noise and optical distortions cause traditional algorithms to fail, leading to "spurious jumps" in position data and loss of tracking.
Algorithmic Limitations: Standard feedback algorithms often rely on "minimum searching" (finding the sharpest image based on size or intensity). This approach is prone to errors near the focal plane and struggles to determine the direction of defocus (above or below), leading to instability.

2. Methodology

The authors propose a novel Deep Learning-enhanced Lagrangian 3D Tracking system that replaces traditional feedback algorithms with a Convolutional Neural Network (CNN).

System Architecture:
- Hardware: An inverted microscope (Zeiss-Observer) equipped with a 3-axis motorized stage (Applied Scientific Instrumentation) and a high-speed camera (Hamamatsu ORCA-Flash 4.0).
- Feedback Loop: Instead of optimizing a metric (like intensity), the system captures an image, processes it through a pre-trained CNN, and receives a prediction of the 3D displacement ( $\Delta X, \Delta Y, \Delta Z$ ) required to center the particle and return it to the focal plane.
- Real-time Implementation: The CNN is trained in Python (using TensorFlow and the DeepTrack2 library) but executed in real-time via a C-based Dynamic Link Library (DLL) called by LabVIEW software. This allows prediction rates exceeding 200 Hz, though the system is currently limited to 80 Hz by camera readout and stage response times.
Deep Learning Model:
- Architecture: A CNN with 4 max-pooling layers separated by convolutional layers, followed by a dense regression head (2 hidden layers, 16 neurons each).
- Input/Output: The input is a cropped image of the particle (defocused or in focus). The output is the predicted shift from the center and the focal plane.
- Training Strategy:
  - Data Acquisition: Training sets are generated by scanning immobile particles (or bacteria immobilized via ethanol, heavy water, or surface fixation) vertically ( $\Delta Z$ ) to capture defocus patterns.
  - Augmentation: Data is augmented with random rotations and translations to ensure robustness against horizontal motion.
  - Asymmetry Enhancement: To improve the "zero-crossing" capability (distinguishing above vs. below focus), the authors intentionally introduced optical asymmetry (e.g., misaligning the correction ring of the objective lens) to make defocus patterns distinct above and below the focal plane.
Microscopy Modalities:
- The system is trained to work with both Fluorescence and Brightfield microscopy.
- Brightfield capability is a critical innovation, allowing tracking of non-fluorescent organisms (like magnetotactic bacteria) and avoiding photodamage.

3. Key Contributions

Paradigm Shift in Feedback: Moving from "minimum searching" (optimizing a signal) to a "zero-crossing" method (predicting position relative to the focal plane). This eliminates ambiguity in the direction of movement near the focal plane.
Versatility Across Modalities: The method successfully tracks organisms using Brightfield microscopy, enabling the study of magnetotactic bacteria and other organisms that cannot be fluorescently labeled without functional degradation.
Robustness in Complex Media: The deep learning model, trained on specific environmental data, maintains tracking accuracy in optically complex environments (turbid bacterial suspensions, mucus) where traditional algorithms fail due to noise and scattering.
Long-term, Non-Destructive Tracking: By utilizing brightfield and optimized illumination protocols, the method minimizes photodamage, allowing for observation times exceeding 400 seconds (and up to an hour in some cases) without altering cell behavior.

4. Key Results

Accuracy and Stability:
- Tracking of Brownian motion in colloidal particles showed the elimination of "spurious jumps" in the Z-axis, which were prevalent in standard tracking.
- The prediction error for the focal position was reduced to < 0.5 µm, with a linear response near the focal plane.
Fluorescent E. coli Tracking:
- Demonstrated long-term tracking of fluorescent E. coli.
- Highlighted the issue of photodamage in fluorescence: high-intensity blue light caused a steady decline in swimming velocity.
- Showed that Brightfield tracking (using yellow light) maintained stable swimming velocities over time, proving the absence of photodamage.
Magnetotactic Bacteria (MSR-1):
- Successfully tracked MSR-1 bacteria in 3D under a magnetic field using brightfield microscopy.
- Revealed complex kinematics, including "run reversals" and a "fat-tail" distribution in run-times, providing new insights into magnetotaxis mechanisms.
Complex Media Applications:
- Active Turbulence: Tracked passive tracers in dense, turbulent bacterial suspensions. The system calculated the Mean Squared Displacement (MSD), revealing a ballistic-to-diffusive crossover and quantifying local polar order parameters ( $P \approx 0.2$ ).
- Mucus: Tracked bacteria in porcine intestinal mucus. The system detected transient trapping events (stopping times) and quantified diffusion coefficients ( $D \approx 50 \mu m^2/s$ ), data previously inaccessible due to optical heterogeneity.

5. Significance and Impact

Ecological and Biological Insights: This method enables the study of individual phenotypic traits over extended periods in realistic, complex environments. It bridges the gap between single-cell behavior and population-level dynamics.
Medical and Environmental Applications:
- Infection Mechanisms: Understanding how bacteria navigate mucus layers is vital for developing treatments for mucosal infections.
- Drug Delivery: Insights into transport in active fluids can improve targeted drug delivery systems.
- Diagnostics: The robustness in complex media suggests potential for monitoring pathogens in bodily fluids.
Future Directions: The framework is adaptable to other imaging modalities (phase-contrast, DIC) and can be integrated with microfluidics for synthetic biology. The authors suggest that adaptive, real-time model updates could further enhance tracking in dynamically changing environments.

In conclusion, this work presents a transformative tool for microbiology, replacing fragile, fluorescence-dependent tracking with a robust, deep learning-driven system capable of observing life at the micro-scale in its most natural and challenging states.