💻 computer science

OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planning

This paper introduces Cross-Video Scene Procedure Planning (CVSPP), a new task requiring models to retrieve relevant video evidence and plan actions simultaneously, and proposes One-Step Evidence Fusion (OSEF), a method that avoids hard selection by fusing all candidate evidence into a soft lattice for planning, achieving state-of-the-art performance on a newly established eleven-source benchmark.

Original authors: Zhentong Ye, Lei Zhang, Sijia Zhou, Yingda Yu, Yuehan Shi, Jiaqi Xuan, Shuaiwu Dong, Guanchao Tong, Meimei Zhang, Bin Li

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhentong Ye, Lei Zhang, Sijia Zhou, Yingda Yu, Yuehan Shi, Jiaqi Xuan, Shuaiwu Dong, Guanchao Tong, Meimei Zhang, Bin Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make a perfect cup of coffee. You don't just want the robot to know the steps; you want it to watch a video, figure out which specific video shows the right way to do it, find the exact moment the barista pours the water, and then write down the instructions itself. This is the world of Video Scene Procedure Planning. It's a branch of artificial intelligence where computers try to understand how to turn a starting scene (like raw ingredients) into a goal scene (like a finished meal) by watching videos.

For a long time, researchers gave the computer the "correct" video and the exact start and end times. It was like handing the robot the right page of a cookbook and saying, "Read this." But in the real world, if you ask a search engine for "how to make coffee," it doesn't give you one perfect video; it gives you a list of ten different videos. Some are great, some are terrible, and some show the same steps but filmed in a different kitchen. The big challenge is: How do you teach a computer to pick the right video from the messy list, find the right part of that video, and then plan the steps, all without getting confused by the bad options? This paper tackles that messy, real-world problem.

The Problem: The "Wrong Recipe" Trap

The authors introduce a new, tougher game called Cross-Video Scene Procedure Planning (CVSPP). Think of it like a cooking show challenge where the host gives you a description of a dish ("Start with raw potatoes, end with golden fries") and a stack of four different cooking videos. Your job is to:

Pick the one video that actually shows how to make fries.
Find the specific clip within that video where the frying happens.
Write down the step-by-step recipe.

The tricky part is that all four videos might be about "making food," and they might even look very similar. If you pick the wrong video (maybe the one showing how to bake a cake) or the wrong clip (maybe the part where they are washing the potatoes, not frying them), your recipe will be nonsense.

Previous methods tried to solve this by playing "guess and check." They would pick one video, say, "Okay, this is the one!" and then try to plan the steps. But if they picked the wrong video at the start, the whole plan collapsed. It's like trying to follow a map to Paris when you accidentally picked a map of London; no matter how well you read the map, you'll never get to the Eiffel Tower. The authors argue that making this "hard choice" too early is a fatal mistake.

The Solution: OSEF (One-Step Evidence Fusion)

To fix this, the team built a new system called OSEF (One-Step Evidence Fusion). Instead of picking a single video and throwing away the rest, OSEF keeps all the videos in the mix until the very last second.

Imagine you are a detective trying to solve a mystery. Instead of arresting the first suspect you see, you keep all four suspects in the room. You look at the clues (the "start" and "goal" description) and ask: "How well does Suspect A fit the story? How about Suspect B?" You don't just pick one; you create a "scoreboard" for every single moment in every single video.

Then, OSEF uses a special "translator" (called a token-global adapter) to feed this entire scoreboard to the planning brain. It's like giving the chef a giant, transparent sheet that shows every possible step from every video, weighted by how likely it is to be correct. The chef can then look at the whole picture and decide, "Ah, the frying step in Video 1 looks best, but the chopping step in Video 2 is clearer."

By keeping all the options "soft" and available, the system avoids the trap of locking onto the wrong video too early. It allows the planning process to correct itself, even if the initial guess about which video is best was slightly off.

What They Found

The authors tested their new method against nine other planning families on a massive benchmark they built, which included 11 different sources of instructional videos (like cooking, DIY, and cleaning). They created a "scoreboard" with 14 different test scenarios.

Here is what the numbers say:

The Big Win: On the six most reliable test scenarios (called "native cells"), OSEF ranked first in every single one.
The Improvement: On four specific tests where the videos were very similar (same-task), OSEF improved the success rate of getting the exact right video and the exact right plan by 2.9 to 10.7 percentage points compared to the previous best methods.
The Secret Sauce: The authors found that the biggest boost didn't come from fancy new math for picking the video, but from the interface that kept all the video options open for the planner to see. It was the "token-global" approach that made the difference.
The Limits: The system still struggles when the videos are very short or the labels are messy. In five of the converted test scenarios, the system performed no better than just guessing the most common answer (the "majority-sequence floor"), suggesting that for some types of messy data, the problem is still very hard.

Why It Matters

This paper shows that when we ask computers to learn from a messy pile of real-world videos, we can't force them to make a single, hard choice immediately. Instead, we need to let them hold all the possibilities in their mind at once, letting the final decision emerge from the whole picture.

The authors are careful to note that while this is a significant step forward, it's not a magic wand. The system still relies on pre-processed video features and deterministic queries, and it can still get confused if the videos are too similar or the data is too noisy. However, by proving that "keeping your options open" works better than "picking a winner early," they've given AI researchers a new, more robust way to teach machines how to learn from the chaotic, wonderful mess of human instruction videos.

Technical Summary: OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planning

1. Problem Definition: Cross-Video Scene Procedure Planning (CVSPP)

The paper addresses a gap in existing Video Scene Procedure Planning (VSPP). Traditional VSPP systems assume the correct evidence video and temporal window are provided as input to predict an action sequence transforming a start scene to a goal scene. In practical scenarios, however, the correct evidence is not available; a system must first retrieve the correct video from a candidate set and localize the relevant temporal window before planning.

The authors introduce Cross-Video Scene Procedure Planning (CVSPP) as a benchmarked task where a model receives:

A legal query: An answer-redacted text description of a start–goal transition (e.g., "start: raw potatoes; goal: golden fries served") without revealing the video ID, timestamps, or action sequence.
A candidate set ( $K$ videos), each divided into $M$ temporal cells.

The model must simultaneously:

Retrieve the correct supporting video.
Localize the relevant temporal window within that video.
Plan the ordered closed-set action sequence bridging the start and goal states.

The task is defined by two coupled challenges:

Same-task Evidence Ambiguity: Multiple candidate videos may demonstrate the same nominal procedure with nearly identical stages and neighboring windows, making task-name-only conditioning insufficient.
Retrieval-to-Plan Error Propagation: In standard "retrieve-then-plan" cascades, an early hard selection of the wrong video or window passes an incorrect scene chain to the planner, causing the planning loss to fail irrecoverably.

2. Methodology: One-Step Evidence Fusion (OSEF)

To address these challenges, the authors propose One-Step Evidence Fusion (OSEF), a framework that delays irreversible evidence commitment. Instead of selecting a single video and cropping a window before planning, OSEF scores all candidate cells and feeds the full soft evidence representation to the planner in a single differentiable path.

The architecture consists of three main components:

Query-Conditioned Evidence Lattice:
- The model scores every cell $(k, m)$ in the $K \times M$ candidate lattice against the legal query $q$ .
- A parameter-free symmetric lift constructs valid closed intervals $(i \le j)$ to form a lattice of scores $H_\theta(q, k, i, j)$ .
- The argmax of this lattice determines the reported video and localized span for evaluation, but the planner path retains all $K \times M$ cells.
Token-Global Evidence-to-Plan (E2P) Adapter:
- This component prevents the loss of cell-level signals that occur in standard pooling.
- It flattens the entire $K \times M$ lattice (without cropping) and processes it through a single-layer Transformer encoder.
- $2T$ learned endpoint queries (representing the start and goal states for each of the $T$ steps) cross-attend to all cells in the lattice.
- This produces a scene-state chain tensor ( $B \times T \times 2 \times d_v$ ) where every candidate's evidence remains available to the planner, even if the hard argmax selection is incorrect.
Goal-Conditioned Planner:
- A three-layer Transformer maps the state pairs to per-step action logits.
- The legal query conditions the planner via two mechanisms: a FiLM-style affine scale/shift over state tokens and as a distinct prepended token for attention.
- A shared head decodes the $T$ steps.

Training Objective:
The model employs Joint-Cascade Supervision. The loss ( $L_{JC}$ ) combines the cross-entropy loss of the plan generated from the soft global chain ( $\hat{S}$ ) with a weighted sum of losses from plans generated by individual candidate chains ( $S_k$ ), weighted by the retrieval probability $p_k$ . This allows the planning loss to reshape the evidence representation.

3. Benchmark Construction

The authors constructed a benchmark from 11 instructional video sources (3 native planning benchmarks: COIN, CrossTask, NIV; and 8 converted sources like YouCook2, Ego4D, etc.).

Legal Query Audit: A "fail-closed" gate ensures queries contain no ground-truth leakage (no video IDs, timestamps, or action sequences).
Negative Roles: The benchmark includes typed negative roles to test specific failure modes: Wrong Video (WV), Wrong Window (WO), and Wrong Task (WA).
Metrics:
- Evidence Axis: Exact-video R@1, MRR, and span-IoU.
- Plan Axis: Per-step Mean Accuracy (MAcc) and Exact-Sequence Success (PLAN-SR).
- Full Metric: FULL-SR, the probability of correctly retrieving the video, localizing the window, and predicting the exact action sequence simultaneously.

4. Key Results

The paper evaluates OSEF against nine adapted planner families (including latent, graph, generative, and discrete-sequence planners) across 14 source–horizon cells.

Ranking Performance: OSEF ranks first on all six native cells (COIN, CrossTask, NIV) where method ranking is valid.
Performance Gains: On four matched same-task cells (COIN-T3/T4 and CrossTask-T3/T4), OSEF improves FULL-SR by 2.9 to 10.7 percentage points over an enhanced hard-selection State-of-the-Art (SOTA) baseline.
Component Analysis: An ablation study reveals that the token-global interface (retaining all cells rather than pooling) accounts for the largest single increment in performance. Soft pooling alone is insufficient; the ability to read the whole lattice via learned queries is critical.
Failure Modes:
- Retrieval is nearly saturated on distinct-task candidates for specific sources (e.g., NIV-T3 reaches 99.20% R@1), but planning remains the bottleneck.
- Localization is weak; a correct plan does not guarantee a correct window (FULL-SR remains high even when span-IoU is low).
- On converted sources with coarse labels (e.g., CEgo, TACoS), performance often sits near the "majority-sequence floor," indicating these cells are label-degenerate stress tests rather than ranking evidence.

5. Significance and Claims

The paper claims to be the first benchmark to score retrieval, localization, and planning together from an answer-redacted query. Its significance lies in:

Formalizing the Coupled Problem: It explicitly separates "same-task evidence ambiguity" from "retrieval-to-plan error propagation," showing that hard selection cascades are brittle.
Introducing Soft Fusion: OSEF demonstrates that keeping evidence "soft" over the entire candidate lattice and feeding it to the planner via a token-global adapter significantly outperforms hard-selection baselines.
Benchmarking Transfer: It provides a rigorous testbed for how existing VSPP planners transfer when the evidence is not supplied, revealing that most inherited planners fail without the specific OSEF interface.

The authors modestly note limitations: the current setup uses frozen features and deterministic queries, and the lattice cost grows with the number of candidates ( $K$ ). They identify faithful span decoding and stronger backbones as open challenges. The supplementary package includes model constructors, evaluation code, and per-cell audits.