Train–Test Leakage in Pothole Detection Benchmarks: Evidence from a Video-Disjoint Re-Split of BharatPotHole
This study reveals that the widely used BharatPotHole dataset suffers from severe train–test leakage due to duplicated frames and video overlap, which artificially inflates detection performance metrics by up to 15.2% and demonstrates that a corrected, video-disjoint split is essential for obtaining realistic estimates of pothole detection capabilities on new roads.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Roads are the arteries of modern life, but they are constantly under attack from the very vehicles that travel on them. Among the most common and damaging injuries are potholes, deep craters that form when water seeps into cracks and freezes, or when heavy traffic simply wears away the pavement. These holes are more than just an inconvenience; they damage vehicles, increase the risk of accidents for motorcyclists, and force cities to spend vast sums on repairs. For decades, finding these defects has been a slow, manual job, with inspectors driving slowly and marking spots on a map. Today, technology offers a faster alternative: cameras mounted on cars or dashboards can record video, and computer programs can scan that footage to spot potholes automatically. This promise of automated inspection relies heavily on public datasets, collections of thousands of images that researchers use to teach computers what a pothole looks like. If a computer performs well on these test images, agencies assume it will work well on real roads. But this assumption only holds true if the computer is being tested on roads it has never seen before.
A new study by P. Pathmanaban Pugazhendi investigates whether this assumption is actually true for a popular dataset called BharatPotHole, which contains thousands of images of Indian roads. The research focuses on a subtle but critical flaw known as data leakage. In the world of machine learning, a computer learns by studying a training set of images and then proving its knowledge on a separate test set. For the test to be fair, the test images must be completely independent of the training images. However, because road data often comes from video, consecutive frames are nearly identical. If a researcher randomly shuffles these frames and puts some in the training set and others in the test set, the computer might simply be recognizing a frame it has already seen, rather than learning to identify a pothole on a new road. This study audited the BharatPotHole dataset to see if this kind of leakage had inflated the reported success rates of pothole detectors.
The researchers began by examining every single one of the 7,074 images released in the dataset. They used a method of digital fingerprinting to check for exact copies and near-identical duplicates. The audit revealed a startling amount of repetition. Of the total images, 86 were exact duplicates, and another 2,769 were simply export copies of other frames, often created by software tools that generate variations of an image. In total, 40.4% of the released images were not unique scenes at all. More importantly, the study looked at how the dataset was divided into training and testing groups. In the published split, nearly 99.4% of the test images came from the same video recordings as the training images. This means that for almost every test image, the computer had already seen the exact same road, the same lighting, and the same pothole just a few seconds earlier in the training footage.
To understand how much this overlap boosted the scores, the researchers created a new, stricter version of the dataset. They removed all the duplicate images and then split the remaining unique videos so that no video appeared in both the training and testing groups. This ensured that the computer was being tested on entirely new roads it had never encountered. They then trained a standard pothole-detection model on both the original, leaky version and this new, clean version. When the model was tested on the original, leaky set, it achieved a score of 0.619. However, when the exact same model was tested on the clean set of new roads, its score dropped significantly to 0.467. This gap of 0.152 represents a 33% drop in performance, suggesting that the high scores reported in previous studies were largely due to the computer recognizing familiar scenes rather than truly detecting defects.
The researchers went further to prove that this drop was caused specifically by the leakage and not by some other factor. They took the model trained on the clean data, which had a very low score on new roads, and secretly replaced some of the test images with copies of images it had already seen during training. As soon as they introduced these familiar images, the model's score jumped dramatically, rising from 0.133 to 0.494 when all test images were replaced with training copies. In contrast, when they replaced test images with copies of other unseen images, the score remained low. This experiment confirmed that the presence of near-identical training images was the sole reason for the inflated performance. Furthermore, when they tested the model on a completely different dataset from another region, the model trained on the leaky data performed worse than the model trained on the clean data, proving that the high scores on the original dataset did not translate to real-world ability.
The study concludes that the way road-damage datasets are currently split is misleading. By randomly shuffling video frames, researchers have been creating a situation where computers are tested on data they have already memorized, leading to overly optimistic results. The author argues that for automated inspection systems to be reliable, datasets must be split by video or route, ensuring that the test roads are completely separate from the training roads. They also recommend that future benchmarks remove duplicate images and report performance on external datasets from different locations. Without these changes, the scores used to evaluate pothole detectors may be measuring a computer's memory rather than its ability to see the road ahead. The findings serve as a necessary correction, reminding the scientific community that a high score on a benchmark is only meaningful if the test is truly a test of new knowledge.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.