⚡ electrical engineering

Video to All-in-focus Image Reconstruction Algorithm for Automated Microscopic Urinalysis

This paper proposes an automated microscopic urinalysis pipeline that reconstructs all-in-focus images from short, manually focused videos to enable efficient deep learning-based detection and classification of urine sediments, thereby eliminating the time-consuming process of capturing multiple discrete focal-plane images.

Original authors: Chinmay Nema, Hari Om Aggrawal, Dipam Goswami, Rajiv Gupta, Vinti Agarwal

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chinmay Nema, Hari Om Aggrawal, Dipam Goswami, Rajiv Gupta, Vinti Agarwal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to take a picture of a busy city street, but your camera lens can only focus on one specific layer at a time. If you focus on the people in the foreground, the buildings behind them turn into a blurry mess. If you focus on the buildings, the people vanish into a fuzzy haze. This is a common headache for photographers, but it's an even bigger problem for doctors trying to look at tiny cells under a microscope. Urine samples aren't flat like a photograph; they are more like a 3D sandwich with layers of cells floating at different heights. When a doctor looks through a standard microscope, they can only see the "slice" of the sandwich that is perfectly in focus. The rest is a blur, making it hard to spot important clues like infections or kidney issues.

To solve this, doctors usually have to play a tedious game of "focus, snap, refocus, snap" many times, taking dozens of still photos at different heights and hoping they catch every cell clearly. It's slow, expensive, and requires special, high-tech equipment that many small clinics or villages simply can't afford. But what if, instead of taking a bunch of still photos, you just held the camera and slowly turned the focus knob while recording a video? You'd capture every layer of the "sandwich" as it comes into and goes out of focus. The big question is: Can a computer look at that messy video and stitch it back together into one perfect, crystal-clear picture where every single cell is sharp? This is the puzzle a team of researchers set out to solve, aiming to make microscopic urine analysis faster, cheaper, and easier for everyone.

The researchers behind this study propose a clever workaround for the "blurry sandwich" problem. Instead of using expensive, automatic focusing machines, they suggest a low-tech approach: a lab technician simply records a short video (lasting between 2 and 14 seconds) of a urine sample while manually turning the microscope's focus knob up and down. As the focus shifts, different cells come into sharp view at different moments. The team then uses a new computer algorithm to act like a digital editor. This algorithm slices the video into tiny square patches, figures out which patch from which frame is the sharpest, and stitches them all together to create a single "all-in-focus" image. Think of it like a chef who tastes every layer of a cake separately and then magically combines the best-tasting bite from each layer into one perfect, super-cake.

Once this perfect image is reconstructed, a smart computer program (a deep learning model) scans it to count and identify different types of cells, such as red blood cells, pus cells, and epithelial cells. The team tested this method on 14 videos recorded by a trained technician in a standard lab. The results were promising: the system successfully detected about 70% of the cells present in the videos. For identifying pus cells and epithelial cells, the computer was incredibly accurate, getting the right answer about 95% of the time when it said it found a cell. However, the system struggled a bit more with red blood cells, sometimes mistaking blurry bits of other cells for them, which lowered its accuracy for that specific type.

The paper explicitly rules out the idea that this method works perfectly for every single scenario without tweaking. The researchers found that if cells are moving around too much (like in one of their test videos), the "stitching" becomes harder, and the results are less certain. They also note that their current method doesn't automatically track moving cells; it assumes the cells stay still while the focus changes. While the approach is a strong proof-of-concept that works well for stationary samples, the authors admit it is not yet a fully solved, perfect system for all moving targets. They suggest that future work will need to focus on tracking moving cells and making the software even more robust.

Ultimately, this study suggests that a simple video recording, combined with a smart reconstruction algorithm, can replace expensive autofocus systems. This could be a game-changer for smaller labs or rural areas where experienced technicians are scarce and budgets are tight. By turning a slow, manual, multi-step process into a quick video and a computer's "magic stitch," the researchers hope to make high-quality urine analysis accessible to more people, potentially catching health issues like kidney infections or urinary tract infections earlier and more reliably.

Technical Summary: Video to All-in-Focus Image Reconstruction for Automated Microscopic Urinalysis

Problem Statement
Microscopic urinalysis is a critical diagnostic tool for detecting urinary tract infections (UTIs), renal failure, and other conditions. While automated analyzers exist, they are often expensive, standalone systems with built-in autofocus mechanisms, making them inaccessible for small laboratories or regions with a shortage of experienced staff. A fundamental challenge in manual and automated microscopy is the multi-layered structure of urine samples on glass slides. Because the depth of field of a lens is limited, cells at different depths cannot be simultaneously in focus within a single static image. Traditionally, this requires acquiring multiple images at different focal planes and performing fine focus adjustments, a process that is time-consuming and prone to human error.

Methodology
The authors propose a cost-effective pipeline that replaces the acquisition of multiple static images with a single video recording. The methodology consists of three primary stages:

Video Acquisition: Instead of an autofocus system, a trained technician manually adjusts the focus of a standard microscope lens while recording a video (typically 2–14 seconds) using a smartphone or camera adapter attached to the eyepiece. This captures a "blur-sharp-blur" cycle where different cells come into focus at different times.
Patch-Based All-in-Focus Reconstruction: The core contribution is a novel algorithm to synthesize a single, sharp, all-in-focus image from the video frames.
- Patch-wise Sharpness: Rather than analyzing pixel-wise sharpness, the algorithm divides frames into square patches. It calculates the sharpness of each patch using the Frobenius norm of the Hessian of image intensity, which the authors found superior to gradient or Laplacian measures for urine images.
- Initial Depth Map: For each patch, the algorithm identifies the frame index where the sharpness measure is maximal, creating an initial depth map.
- Depth Map Smoothing: The initial depth map is often noisy, leading to discontinuities in the reconstructed image. The authors introduce a smoothing step that enforces local consistency. If the sharpness difference between a patch and its neighbors falls below a threshold $\tau$ , the neighbor inherits the frame index of the central patch. This process repeats until the number of unique frame indices stabilizes.
- Image Synthesis: The final all-in-focus image is constructed by stitching together the sharpest patches from their respective optimal frames.
Deep Learning Detection: The reconstructed all-in-focus image is fed into a RetinaNet deep neural network (DNN) trained on a public dataset of urine sediments to detect and classify cells (specifically Red Blood Cells, Pus cells, and Epithelial cells).

Key Contributions

Video-Based Acquisition Strategy: The paper introduces a workflow that utilizes manual focus variation in a video format to capture the full depth of a sample, eliminating the need for expensive autofocus hardware.
Novel Reconstruction Algorithm: The authors present a patch-based focus stacking algorithm specifically tuned for urine microscopy. It addresses the noise and discontinuity issues common in standard depth map estimation by utilizing Hessian-based sharpness measures and a specific local smoothing heuristic.
Proof of Concept: The study validates this pipeline on 14 real-world urine sample videos acquired in a diagnostic lab environment, demonstrating the feasibility of low-cost automation.

Results
The system was evaluated on 14 videos containing various sample types: multi-layered (ML), single-layered (SL), and videos with moving cells (MV).

Reconstruction Quality: The algorithm successfully generated sharp, all-in-focus images where cells at different depths were clearly visible, resolving the discontinuity issues seen in initial reconstructions.
Detection Performance: Using the reconstructed images, the RetinaNet model achieved the following performance metrics at a confidence level of 0.6:
- Pus Cells: 96% precision, 71% recall, 0.81 F1 score.
- Epithelial Cells: 94% precision, 77% recall, 0.85 F1 score.
- Red Blood Cells (RBCs): 58% precision, 71% recall, 0.64 F1 score.
Limitations: The authors note that the DNN struggles with RBCs, often misclassifying out-of-focus small cells or epithelial nuclei as RBCs due to visual similarities. Additionally, the model has difficulty detecting pus cells when they appear in dense clusters. The algorithm's robustness on videos with moving cells (MV) was observed but deemed uncertain.

Significance and Claims
The paper positions this work as a practical, low-cost alternative to expensive commercial urine analyzers, specifically targeting small laboratories and rural settings where experienced technicians are scarce. The authors claim that their video-based approach, combined with the novel reconstruction algorithm, achieves a positive predictive value of up to 95% and a true positive rate of 70% for key sediment types. They emphasize that the system is user-friendly, requiring no special training for lab technicians beyond recording a focus-varied video. The study concludes that video-based urinalysis is a promising area for research, with future work planned to include cell tracking across frames and the development of an open-source web application for clinical use.

Technical Summary: Video to All-in-Focus Image Reconstruction for Automated Microscopic Urinalysis

More like this