💻 computer science

Deadline-Aware Hardening of Real-Time Object Detection Against Candidate-Inflation Latency Attacks

This paper proposes a retraining-free, deployment-selectable mechanism that caps the number of candidates entering non-maximum suppression to a deadline-calibrated bound, thereby mitigating candidate-inflation latency attacks and ensuring real-time deadline integrity across diverse hardware and detector architectures while revealing that bounding suppression alone is necessary but insufficient due to significant decoding overhead.

Original authors: Salah Gontara, Selem Trabelsi, Khaled Ben Khalifa

Published 2026-09-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Salah Gontara, Selem Trabelsi, Khaled Ben Khalifa

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of autonomous vehicles and security cameras, seeing is not enough; seeing in time is everything. A computer vision system designed to spot pedestrians or traffic signs must do more than simply identify them correctly. It must deliver that identification before the next moment arrives. If a car traveling at highway speeds receives a warning about a hazard a fraction of a second too late, the result is not merely a slower response, but a potential catastrophe. This requirement creates a strict deadline for every single image the system processes. If the computer takes too long to finish its work on one picture, the pipeline falls behind, and the output becomes stale, describing a scene that has already passed.

For years, researchers have focused on making these systems faster and more accurate, often measuring success by average speed. However, in a real-time system, the average can be misleading. A system might be incredibly fast most of the time but occasionally freeze for a long duration. In a safety-critical application, that single slow moment is a failure. Furthermore, these systems are not just vulnerable to random glitches; they can be targeted by attackers who do not try to trick the computer into seeing the wrong object, but rather into working so hard that it runs out of time. This paper explores a specific type of attack where an adversary subtly alters an image to force the computer to generate an overwhelming number of potential detections, causing it to miss its deadline. The researchers then propose a simple, practical way to stop this without needing to retrain the computer's brain.

The core of the problem lies in how these detectors work. When a camera captures an image, the software scans it and produces a massive list of potential objects, each with a confidence score. To turn this chaotic list into a clean set of final detections, the system uses a process called non-maximum suppression. Imagine a crowded room where many people are shouting out the same name; this process filters out the duplicates and keeps only the loudest, most confident voices. Under normal conditions, this filtering is quick. However, an attacker can craft an image that tricks the system into generating tens of thousands of potential objects instead of a few dozen. The filtering process then has to compare every single one of these thousands of candidates against every other one. This creates a computational explosion. The more candidates the attacker forces the system to consider, the longer the filtering takes, eventually causing the system to miss its deadline and fail to deliver a result in time.

The researchers tested this threat on a real-time object detection system running on powerful hardware, specifically designed to handle video streams at thirty frames per second. They found that a standard, unmodified system could be easily overwhelmed. When they fed the system images designed to trigger this overload, the time it took to filter the candidates jumped from a fraction of a millisecond to hundreds of milliseconds. Even on the fastest hardware they tested, the system failed to meet the deadline for the filtering stage if the number of candidates was left unchecked. However, the study confirmed that simply using faster hardware or a more efficient software version of the filtering process was not enough to solve the problem on its own. While these improvements made the system faster, they did not stop the attacker from controlling the workload. The attacker could still force the system to do so much work that even the fastest machine would stumble if no limit was placed on the input.

To solve this, the researchers introduced a strict limit on the number of candidates allowed to enter the filtering stage. Instead of letting the system process every single potential object the image generated, they capped the number at a specific, manageable level. If the system produced more candidates than this limit, it simply selected the most promising ones and discarded the rest before the heavy filtering began. This approach acts as a safety valve, ensuring that the amount of work the system must perform never exceeds a known, safe maximum. The researchers carefully measured the cost of this safety measure. They found that by limiting the candidates to one thousand and twenty-four, the system could handle the filtering stage well within the deadline for that specific stage, reducing the latency to just 4.03 ms. However, the study revealed a critical nuance: even with this cap in place, the defended requests still missed the overall end-to-end deadline. This was not solely due to the attack, but because other bottlenecks, such as the time required to decode the image itself, consumed the remaining time budget. In fact, the researchers found that clean images without any attack also missed the overall deadline 92.7% of the time when using lossless formats, indicating that the decoding process was a major bottleneck regardless of the attack. The trade-off for this protection was an almost imperceptible drop in accuracy, measured at a tiny fraction of a percent, which is negligible for practical use.

The study went further to ensure this solution was robust across different scenarios. They tested the method on two different types of camera sensors and with different software backends, including those running on standard computers and those running on smaller, energy-efficient edge devices. In every case, the limit held firm for the filtering stage, preventing the attacker from inflating the workload beyond the cap. Even when the hardware was under stress from heat or when the system was running on a less powerful board, the capped approach prevented the filtering stage from stalling. However, the researchers emphasized that while the cap successfully controlled the filtering stage, it did not guarantee that the entire pipeline would meet the deadline. They found that once the filtering was controlled, the next bottleneck was often the time it took to decode the image itself. This means that while limiting the candidates is a necessary step to protect the system from this specific attack, it is not a complete cure-all; the entire pipeline must be monitored to ensure the deadline is met.

The authors argue that this method of setting a hard limit on the workload is a crucial step for deploying real-time vision systems in the real world. It shifts the control of the workload from the attacker back to the system administrator. By defining a maximum number of candidates based on the system's speed and the required deadline, a deployment can guarantee that it will never be forced to do more work than it can handle during the filtering stage. The paper concludes that while faster hardware and better algorithms are helpful, they are not sufficient on their own. A real-time system needs a hard boundary on the work it is asked to do. Without such a boundary, an attacker can always find a way to overwhelm the system. With it, the system remains reliable in its processing of the filtering stage, delivering its results on time for that specific component, even when the world around it tries to break it, though the overall system deadline depends on managing all other stages as well.

Technical Summary: Deadline-Aware Hardening of Real-Time Object Detection Against Candidate-Inflation Latency Attacks

Problem Statement

Real-time object detection systems are typically evaluated based on mean latency or frame rate. However, in streaming perception pipelines, availability is defined by whether a response meets a specific deadline (e.g., the frame period of the camera). A single missed deadline renders the output stale and useless for the next frame.

The paper addresses Candidate-Inflation Latency Attacks, a class of adversarial inputs designed not to cause misclassification, but to maximize the computational cost of the Non-Maximum Suppression (NMS) stage. These attacks manipulate the input image to generate an excessive number of candidate bounding boxes that survive the confidence threshold. Since NMS complexity is often quadratic ( $O(N^2)$ ) relative to the number of candidates, these attacks can drive latency far beyond acceptable limits, causing deadline misses.

Prior evaluations suggest that realistic hardware, optimized representations, and faster backends reduce the severity of these attacks. However, the authors argue that reduced severity is not a bound; the workload remains attacker-controlled. Furthermore, previous studies often fail to jointly report deadline-relative tail latency and the deadline-miss ratio (DMR) for a defended serving path, nor do they account for implementation dependencies.

Methodology

System Model and Threat

The study utilizes a real-time detection pipeline (YOLOv8x-P2) deployed on an edge-optimized serving path (TensorRT fp16 on RTX 4090 GPUs). The threat model assumes an adversary perturbs an input image within an $L_\infty$ budget ( $\epsilon$ ) to maximize the number of candidates ( $N_\tau$ ) surviving the confidence threshold before NMS. The attack is deterministic and input-complexity based, distinct from request flooding.

The evaluation distinguishes between three scopes:

Client Scope: End-to-end latency including network, queueing, and serialization.
Worker Scope: Inference and post-processing time on the server.
Suppression Scope: The time taken specifically by the NMS backend.

Defense Mechanism: Bounded Candidate Admission

The core proposal is to admit at most $K$ candidates to the NMS stage.

Selection: A standard Top- $K$ selection operation is applied to the candidates surviving the confidence threshold.
Calibration: The value of $K$ is not arbitrary; it is calibrated against a declared deadline ( $D$ ) and an acceptable accuracy tolerance ( $\delta_A$ ).
Implementation: The selection uses a lexicographic key (negated scores, then index) to break ties, ensuring stability. This operation is $O(N)$ and adds no pairwise IoU comparisons, effectively capping the quadratic term of NMS.

Experimental Setup

The authors employ a rigorous validation strategy:

Backends: They replay identical attacked candidate arrays through three distinct suppression backends: NumPy (CPU), torchvision (CPU), and torchvision (CUDA).
Corpora: Experiments use VisDrone2019-DET (aerial) and BDD100K (driving).
Metrics: Performance is measured using the 95th percentile latency ( $p95$ ) and the Deadline-Miss Ratio (DMR).
Controls: The study includes "matched controls" where clean images are re-encoded to lossless formats to isolate the impact of image representation from the attack itself.

Key Contributions

Deadline-Oriented Evaluation: The paper scores attacks using client-scope $p95$ latency and DMR, keeping client, worker, and suppression scopes separate to identify bottlenecks accurately.
Implementation-Aware Characterization: By replaying identical attacked data across three backends, the authors separate the contribution of the implementation from the attack itself, demonstrating that while faster backends reduce latency, they do not eliminate the deadline violation if the candidate count is unbounded.
Bounded Candidate Admission: The authors propose using Top- $K$ selection as a deployment-selected bound on attacker-controlled work. This defense requires no retraining and no weight changes, making it applicable to already promoted artifacts.
Deadline-Calibrated Selection Rule: A rule is provided to select $K$ based on measured tail latency and accuracy tolerance, allowing deployments with different frame periods to determine their own bounds.
End-to-End Validation: The study validates the defended artifact against a matched control, expressing requirements as a deadline-aware condition for model promotion.

Results

Attack Severity and Backend Dependence

Uncapped Latency: Without a bound, the attack causes massive latency spikes. On the deployed NumPy path, the $p95$ latency for NMS jumps from 1.16 ms (clean) to 635.35 ms (attacked). Even on the fastest backend (torchvision CUDA), the latency reaches 49.08 ms, exceeding the 33.3 ms (30 Hz) deadline.
Implementation Impact: The attack is strongly implementation-dependent. The same attacked data yields significantly different latencies across backends, confirming that "faster" backends alone are insufficient defense.

Effectiveness of the Cap ( $K=1024$ )

Latency Reduction: Admitting at most $K=1024$ candidates reduces the attacked NMS $p95$ to approximately 4.28 ms across all backends, well within the 33.3 ms budget.
Accuracy Cost: The accuracy cost is measured rather than assumed. For the evaluated detector, the cost is 0.00064 AP@[.50:.95] on the validation set and 0.000188 AP@[.50:.95] on the held-out test set. This contradicts prior assumptions that such limits would significantly compromise model integrity.
Generality: The bound holds across two detector geometries (different anchor counts), three backends, and a thermally throttled edge board (Raspberry Pi 4B).

The "Sufficiency" Boundary

A critical finding is that bounding NMS work is necessary but not sufficient for end-to-end deadline integrity.

Even with the NMS cap in place, the defended requests still miss the end-to-end deadline (33.3 ms) when served through the full pipeline.
Root Cause Analysis: Per-stage instrumentation reveals that the largest attack-associated increment is not in NMS, but in image decoding and representation. The attack forces the system to process larger, lossless payloads, increasing decode time.
Clean vs. Attacked: Surprisingly, clean lossless images (without any adversary) also miss the deadline in the tested pipeline configuration, indicating that the bottleneck is the delivery format and decoding, not just the adversarial input.

Significance and Claims

The paper claims that bounding candidate admission is the only mechanism that explicitly limits the quantity of work an attacker can control, regardless of the backend implementation.

Rejection of "Reduced Severity" as Defense: The authors argue that while realistic deployments (faster hardware, quantization) blunt the attack, they do not provide a hard bound. An attacker can still force a deadline miss if the candidate count is unbounded.
Promotion Criterion: The authors propose a new acceptance criterion for model promotion (Eq. 7) that includes deadline evidence. Under this criterion, the evaluated artifact fails to be promoted at a 30-Hz deadline because the end-to-end pipeline (specifically decoding and representation) cannot meet the deadline, even with the NMS defense.
Modest Scope: The paper does not claim to solve all real-time latency issues. It explicitly states that bounding suppression work is a necessary step for deadline integrity but that the remaining latency path (decoding, network, queueing) must also be measured and controlled. The defense is specific to the NMS stage and does not protect against other forms of latency attacks or system overloads.

In conclusion, the paper establishes that while implementation choices matter, a deployment-selected, deadline-calibrated cap on candidate admission is required to bound attacker-controlled work. However, achieving end-to-end real-time guarantees requires a holistic view of the pipeline, as the attack's impact can shift to other stages (like decoding) once NMS is hardened.