💻 computer science

CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

The paper introduces CoRE, a weakly supervised coarse-to-fine framework that learns to identify the specific temporal moments and scene entities supporting risk predictions in driving videos by distilling graded effects from structured interventions on a coarse video-level predictor, thereby enabling fine-grained evidence localization without requiring costly fine-grained annotations.

Original authors: Kaiser Hamid, Can Cui, Nade Liang

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaiser Hamid, Can Cui, Nade Liang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every day, millions of drivers navigate a world of moving cars, pedestrians, and changing weather, constantly making split-second judgments about safety. To an outside observer, a video of a drive might look like a simple stream of images, but to a computer trying to understand risk, it is a complex puzzle. The core challenge is that while we can easily label an entire video as "risky" or "safe," explaining exactly why is much harder. We know a situation is dangerous, but we often cannot pinpoint the precise moment a hazard appeared or identify exactly which car or person contributed to that danger. This gap between a general feeling of risk and the specific evidence that supports it has long limited how well computers can learn to drive safely. Researchers have struggled to teach machines to look at a video and say not just "this is dangerous," but "this specific moment with that specific object is what makes it dangerous," especially when they only have the general label to start with.

A team of researchers has developed a new approach called CoRE that bridges this gap without needing expensive, detailed instructions. Instead of asking humans to draw boxes around every dangerous moment or label every risky object in thousands of videos, CoRE learns to find those details on its own by observing how a computer's judgment changes when parts of the video are altered. The process begins by training a computer to give a single risk score for an entire video clip, using only the broad labels humans provide. Once this computer is trained, it is frozen, meaning its brain is locked in place. The researchers then systematically test this frozen computer by temporarily hiding or blurring small sections of the video or specific moving objects, one by one. They watch closely to see how much the computer's risk score drops when a particular piece of the scene is removed. If hiding a specific car causes the risk score to plummet, that car was a major driver of the danger. If hiding a patch of time has no effect, that moment was likely irrelevant.

These measured changes become the teacher for a second, smarter computer. This second computer, the student, is shown the original, unaltered videos and asked to predict exactly which moments and objects are responsible for the risk, based entirely on the lessons learned from the first computer's reactions. The student learns to mimic the pattern of influence it saw during the testing phase, effectively distilling the complex "what-if" experiments into a direct ability to spot evidence. The result is a system that can look at a raw video and immediately highlight the specific seconds and the specific vehicles that support a risk prediction, all without ever having been shown a single example of a human-drawn risk interval.

The researchers tested this method on three different types of video data to see if it held up in various situations. First, they used a dataset of driving clips where humans had only provided a general sense of how risky the scene felt, with no details about time or objects. In this setting, the system successfully learned to identify the specific moments that drove the risk perception, outperforming previous methods that relied on less direct clues. Next, they applied the technique to a dataset of driving videos that did contain precise, human-labeled timestamps for traffic anomalies, which the system had never seen during training. Here, CoRE proved its accuracy by locating these dangerous events with a high degree of precision, matching the independent human labels even though it was trained only on the coarse, video-level labels. Finally, the team tested the system on a completely different type of video: surveillance footage of crimes, which has nothing to do with driving. The method worked just as well, finding the specific moments of abnormal behavior in crowded streets and empty hallways.

The findings suggest that a computer's initial, broad judgment contains a hidden map of the evidence that supports it. By measuring how that judgment shifts when the input is tweaked, the system can reconstruct the fine-grained details of the scene. This approach does not require the computer to understand the physics of a crash or the intent of a driver; it simply learns which parts of the visual input are necessary for the prediction to hold true. The study demonstrates that coarse supervision, once thought too vague to be useful for detailed analysis, can actually be leveraged to recover precise temporal and object-level support. This means that in the future, safety systems could potentially learn from vast amounts of unlabeled or loosely labeled video data, identifying the exact causes of risk without the prohibitive cost of detailed human annotation. The work confirms that the path to understanding complex visual events does not always require more data, but rather a smarter way of asking the computer what it is actually looking at.

Technical Summary of CoRE: Weakly Supervised Coarse-to-Fine Risk Evidence Learning in Driving Videos

1. Problem Formulation

The paper addresses the challenge of learning fine-grained prediction support (temporal and entity-level) from coarse video-level supervision in driving scenarios. While driving videos are often annotated with overall risk scores or event labels (e.g., "high risk" or "anomaly"), the specific evidence driving these judgments—when a risk emerges and which entities contribute—is typically unannotated. Acquiring such fine-grained temporal and entity-level annotations is prohibitively expensive.

Existing weakly supervised methods often rely on classifier activations or instance scores to infer localization. However, the authors argue that these scores may highlight only the most discriminative snippets rather than the full evidence supporting a prediction. Furthermore, attention weights do not necessarily provide faithful explanations of model decisions. The core problem is to recover the specific temporal regions and entity tracks that support a coarse prediction without requiring explicit fine-grained labels during training.

2. Methodology: CoRE Framework

The authors propose CoRE (Coarse-to-Fine Risk Evidence), a weakly supervised framework based on prediction-effect distillation. The method operates in two main stages:

A. Teacher Training (Coarse Predictor)
A coarse video-level predictor (the "Teacher") is trained using standard video-level supervision (e.g., a single risk score per clip). This model consists of a visual encoder, a temporal encoder, and a scoring head. Once trained, the Teacher is frozen. It serves solely as a source for measuring prediction changes; it is not used for inference in the final system.

B. Prediction-Effect Target Construction
Instead of learning directly from coarse labels, CoRE generates graded weak targets by measuring how the frozen Teacher's prediction changes when specific candidates are intervened upon.

Candidates: The system considers two types of evidence: temporal regions (snippets of the video) and entity tracks (tracked objects).
Intervention: For each candidate, a predefined intervention (e.g., masking or perturbing the candidate) is applied to the input video while keeping the rest unchanged.
Effect Measurement: The prediction effect is calculated as the positive difference between the Teacher's response on the original video and the intervened video: $d = [s_T(X) - s_T(I(X, c))]_+$ .
Target Generation: These measured effects are converted into graded support targets ( $p$ ) that reflect the relative importance of each candidate. This process creates a "soft" supervision signal that captures the full distribution of evidence, not just the single most salient snippet.

C. Student Training (Direct Support Prediction)
A separate "Student" network is trained to predict temporal and entity support directly from the original, unmodified video.

The Student learns to match the intervention-derived targets using a distillation loss.
Crucially, the Student never observes the interventions or the Teacher during inference.
The final training objective combines the support distillation loss with the original weak bag-level objective to ensure the Student also maintains accurate coarse prediction capabilities.

D. Inference
At inference time, the Teacher and all intervention operations are discarded. The Student processes the video in a single forward pass, outputting the coarse response along with the fine-grained temporal and entity support scores.

3. Key Contributions

The paper claims three primary contributions:

Problem Formulation: It introduces the first formulation for learning both temporal and entity-level perceived-risk support from clip-level subjective judgments alone, addressing the gap between coarse supervision and fine-grained evidence.
CoRE Framework: It proposes a prediction-effect distillation mechanism that converts the response of a frozen coarse predictor under structured interventions into graded targets for direct support prediction.
Empirical Validation: It validates the approach across three distinct settings:
- RISEE: Testing support learning from subjective risk judgments without temporal/entity labels.
- DoTA: Evaluating weakly supervised traffic-anomaly localization against independent temporal annotations.
- UCF-Crime: Demonstrating generalization to a standard non-driving anomaly detection benchmark.

4. Experimental Results

The authors evaluate CoRE against state-of-the-art weakly supervised baselines (e.g., Attention MIL, RTFM, MGFN) across the three datasets:

RISEE (Perceived Risk): CoRE achieves the lowest Mean Absolute Error (MAE) and highest Spearman correlation for clip-level risk prediction. More significantly, for temporal support, CoRE shows a 0.320 selected prediction drop and a 0.542 correlation with measured candidate effects, vastly outperforming baselines like Attention MIL, which showed negative correlations. This indicates CoRE successfully identifies the actual evidence driving the prediction.
DoTA (Temporal Localization): CoRE achieves state-of-the-art results on frame AUC, AP, and F1 scores under both ResNet-50 and CLIP ViT feature banks. For instance, with ResNet-50, it improves Frame AUC from 0.642 (best baseline) to 0.735 and F1@0.5 from 0.237 to 0.374.
UCF-Crime (Generalization): CoRE achieves 85.68% Frame AUC, outperforming controlled reproductions of RTFM (84.30%) and MGFN (82.79%), demonstrating that the coarse-to-fine mechanism extends beyond driving contexts.

Ablation Studies: Removing the prediction-effect supervision causes the most significant performance degradation (AUC dropping from 0.744 to 0.419 on DoTA), confirming that the intervention-based targets are the core driver of the method's success.

5. Significance and Claims

The paper concludes that coarse video predictions can provide useful supervision for recovering the fine-grained evidence supporting them without requiring corresponding fine-grained labels.

The authors emphasize that:

The learned support reflects prediction dependence (how much the model relies on a specific evidence) rather than physical causality.
The method avoids the pitfalls of attention-based explanations, which may not be faithful.
The framework is general, applicable to both temporal localization and entity-level relevance, and effective across different domains (driving and surveillance).

The work demonstrates that by leveraging the sensitivity of a coarse predictor to structured interventions, it is possible to distill a direct, fine-grained predictor that operates efficiently at inference without the need for costly annotations or complex intervention procedures during deployment.