🤖 AI

ESTANet: Efficient Online Error Detection in Procedural Videos via Prediction Inconsistency

ESTANet is a lightweight, real-time framework for online error detection in procedural videos that leverages prediction inconsistencies among standard and error-sensitive action detectors with varying temporal contexts to achieve state-of-the-art performance without complex architectural designs.

Original authors: Shih-Po Lee, Reza Ghoddoosian, Faizan Siddiqui, Enna Sachdeva, Behzad Dariush

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shih-Po Lee, Reza Ghoddoosian, Faizan Siddiqui, Enna Sachdeva, Behzad Dariush

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning to cook a new recipe or assemble a piece of furniture, and you have a helpful AI assistant watching over your shoulder. This assistant's job is to spot mistakes the moment they happen and tell you, "Hey, you put the salt in before the water!" so you can fix it immediately.

The paper introduces a new system called ESTANet (Error-Sensitive and Temporally-vArying Network) that acts as this assistant. Its main goal is to catch errors in real-time while you are doing a task, using a very clever and simple trick instead of building a giant, complicated brain.

Here is how it works, broken down into simple concepts:

The Core Idea: The "Panel of Judges"

Most AI systems try to be perfect by building one super-smart model. ESTANet takes a different approach. Instead of one judge, it sets up a panel of four different "judges" (action detectors) to watch what you are doing.

These four judges are split into two teams:

The "Steady" Judges: These two are trained to be calm and consistent. They look at what you are doing and say, "Okay, you are chopping onions." They are good at spotting what is supposed to happen.
The "Nervous" Judges: These two are trained to be extra sensitive and jittery. They are designed to get confused or change their minds easily when something goes wrong. If you accidentally drop the knife, these judges might suddenly shout, "Wait, that's not right!"

The Secret Sauce: Different "Time Windows"

To make the system even better, the paper gives these judges different "time windows" to look at, like looking through binoculars of different zoom levels.

The "Short-Sighted" View (Small Window): Some judges only look at the last few seconds of your video. They are great at catching immediate, physical mistakes (like dropping an egg).
The "Long-Sighted" View (Large Window): Other judges look back at a longer history of what you did. They are great at catching order mistakes (like trying to put the roof on a house before you've built the walls).

How It Catches Mistakes

The magic happens when the judges disagree.

Scenario A (You are doing it right): All four judges agree. The Steady ones say "Chopping onions," and the Nervous ones also say "Chopping onions." No alarm is raised.
Scenario B (You make a mistake):
- If you drop the knife, the Nervous judges freak out and say "Error!" while the Steady ones are confused.
- If you skip a step (like forgetting to boil water), the "Long-Sighted" judges realize the sequence is wrong and disagree with the "Short-Sighted" judges who only see the current action.

The system uses a majority vote. If at least three out of the four judges (or pairs of judges) disagree with each other, the system flags the moment as an error. It's like a committee deciding, "Okay, three of us think something is wrong here, so we're going to sound the alarm."

Why This Is Special

The paper claims this method is special for three reasons:

It's Fast and Light: It doesn't need a massive, heavy computer to run. It's like a lightweight app that can run on wearable devices (like smart glasses) without slowing them down.
It Learns from "Good" Videos Only: Usually, to teach an AI what a mistake looks like, you need to show it thousands of videos of people failing. ESTANet is smart enough to learn what a mistake looks like just by watching videos of people doing things correctly. It learns that "if the judges start disagreeing, something is wrong."
It Catches Two Types of Errors: It can spot both physical mistakes (dropping things, adding the wrong ingredient) and order mistakes (doing step 5 before step 2).

The Results

The authors tested this system on three different datasets involving cooking, assembling furniture, and setting up tents. They found that ESTANet was better at spotting errors in real-time than other top methods, while staying fast enough to be used in real-world situations.

In short, ESTANet is a lightweight, real-time error detector that works by asking a small team of AI "judges" with different perspectives to vote on whether you are making a mistake. If they start arguing, you know it's time to fix your action.

Technical Summary: ESTANet

Problem Statement
The paper addresses the challenge of real-time, online error detection in procedural videos. Unlike offline methods that process entire videos, online systems must operate causally using only past and current frames, a requirement for practical deployment on wearable devices (e.g., Microsoft HoloLens, Apple Vision Pro) that assist users in daily tasks. The system must detect two distinct types of errors:

Execution Errors: Actions that are interrupted, performed incorrectly, or are extraneous (e.g., dropping a knife, adding sugar instead of honey).
Procedural Errors: Deviations in the sequence, such as omitting, repeating, or misordering steps (e.g., tightening table legs before inserting a cross support).

Existing online methods suffer from limitations: they often detect only the first error, overlook nuanced execution errors, rely on heavy Large Language Models (LLMs) that hinder real-time performance, or require annotated error-free and error-containing videos during training.

Methodology: ESTANet
The authors propose ESTANet (Error-Sensitive and Temporally-vArying Network), a lightweight framework that detects errors by exploiting prediction inconsistencies among a small ensemble of action detectors. The core hypothesis is that action detectors with different sensitivities and temporal contexts will naturally diverge in their predictions when an error occurs, while remaining consistent on correct executions.

The framework consists of four detectors divided into two categories:

Standard vs. Error-Sensitive Detectors (Capturing Execution Errors):
- Standard Detectors: These use a simple architecture consisting of a single-layer Gated Recurrent Unit (GRU) followed by a linear classification head. They are trained to produce stable predictions.
- Error-Sensitive Detectors: These incorporate a Temporal-Aware Dynamic (TAD) module. The TAD module generates dynamic weights and biases for the affine transformation of frame features before they enter the GRU. These parameters are input-dependent and temporal-aware, making the detector highly sensitive to out-of-distribution data (i.e., execution errors), leading to inconsistent predictions compared to the standard detectors.
Temporally-Varying Design (Capturing Procedural Errors):
- The system trains detectors with different temporal window sizes: a small window ( $s$ ) and a large window ( $l$ ).
- Strategy for Window Selection: Instead of exhaustive hyperparameter search, the authors propose a robust, dataset-specific strategy:
  - $s$ is set to the shortest mean duration of any action in the dataset, forcing the detector to rely on partial context.
  - $l$ is calculated to cover $\beta$ (set to 2) complete preceding actions for approximately $\theta$ (set to 50%) of the actions in the dataset.
- This ensures that when a procedural error (e.g., a missing step) occurs, the detector with the large window (relying on long-range dependencies) will predict differently than the detector with the small window (relying on short-range context).

Inference and Decision Logic
During inference, the system processes frames causally. It generates four action predictions per frame:

$y^{s-st}$ and $y^{s-lg}$ : Standard detectors with small and large windows.
$y^{e-st}$ and $y^{e-lg}$ : Error-sensitive detectors with small and large windows.

The system constructs four comparison pairs to check for agreement:

$(y^{s-st}, y^{e-st})$ and $(y^{s-lg}, y^{e-lg})$ : Detect execution errors by comparing standard vs. error-sensitive predictions.
$(y^{s-st}, y^{s-lg})$ and $(y^{e-st}, y^{e-lg})$ : Detect procedural errors by comparing predictions across different temporal windows.

A frame is flagged as an error ( $z_t = 1$ ) if at least three out of the four pairs show a mismatch (majority voting). The model is trained exclusively on error-free videos using a combination of cross-entropy loss and a smoothing loss to ensure temporal consistency.

Key Contributions

Novel Perspective: The paper shifts the focus from designing complex architectures to leveraging the intrinsic prediction behaviors of action detectors. It demonstrates that inconsistencies among simple detectors are a powerful signal for error detection.
Lightweight Architecture: ESTANet avoids heavy components like LLMs or complex graph learning, utilizing only GRUs and a lightweight TAD module, ensuring real-time efficiency suitable for edge devices.
Robust Window Selection Strategy: The paper introduces an automated, dataset-agnostic method to determine optimal temporal window sizes ( $s$ and $l$ ), removing the need for time-consuming heuristic searches.
Unified Detection: The framework simultaneously addresses both execution and procedural errors without requiring annotated error data during training.

Experimental Results
The authors evaluated ESTANet on three datasets: EgoPER (containing both error types), Assembly-101-O, and EPIC-Tent-O (focusing on procedural errors).

EgoPER: ESTANet achieved state-of-the-art performance with F1@10 of 47.2, F1@25 of 37.8, and F1@50 of 21.6, outperforming baselines like PREGO and DTGL.
Assembly-101-O: The method achieved an Avg-F1 of 59.6, surpassing DTGL (53.5) and PREGO (32.5).
EPIC-Tent-O: ESTANet achieved an Avg-F1 of 70.2, significantly outperforming DTGL (46.5) and PREGO (29.4).
Efficiency: The model maintains real-time efficiency due to its lightweight design, avoiding the latency issues associated with LLM-based approaches.

Significance and Claims
The paper claims that ESTANet provides a practical and powerful solution for online error detection by leveraging the intrinsic properties of action detectors rather than increasing architectural complexity. The authors emphasize that their approach aligns with real-world constraints where normal (error-free) videos are abundant, but annotated error data is scarce. By achieving state-of-the-art performance with a simple ensemble of GRUs, the work suggests that prediction inconsistency is a fundamental and underutilized signal for detecting deviations in procedural tasks.