💻 computer science

Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception

This paper proposes a novel, lightweight scheduling framework for multimodal streaming perception in human-robot collaboration that leverages scene context and previous frame outputs to dynamically allocate computational resources, significantly reducing latency while improving activation recall and keyframe accuracy compared to conventional parallel pipelines.

Original authors: Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi

Published 2026-03-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the brain of a helpful robot assistant working in a busy office. Your job is to watch the room, listen to conversations, and figure out what the humans need so you can help them.

To do this, you have a team of specialized "detectives" inside your head:

The Eye (Object Detection): Scans the room to see what objects are there (chairs, cups, laptops).
The Body Scanner (Pose Estimation): Tracks exactly how people are moving, where their hands are, and if they are sitting or standing.

The Problem: The "Always-On" Exhaustion

In the old way of doing things, both detectives worked every single second, 24/7, even if nothing was happening.

If a person is just sitting still reading a book for ten minutes, the Body Scanner is still frantically calculating their pose every millisecond.
If the room is empty, the Eye is still scanning for objects that aren't there.

This is like having a security guard who runs a full marathon every time you walk to the mailbox. It wastes a huge amount of energy (computing power) and slows you down because your brain is too busy processing useless data to react quickly when something actually important happens. This causes latency—the robot gets slow and clumsy.

The Solution: The "Smart Manager"

This paper introduces a Smart Manager (the Scheduling Framework) that sits between your brain and your detectives. Instead of letting them work blindly, the Manager decides who works, when, and for how long, based on what's actually happening in the room.

Here is how the Manager thinks, using simple analogies:

1. The "Relevance" Filter (The Coffee Shop Analogy)

Imagine you are in a coffee shop.

Scenario A: You are sitting alone, reading a book. The world is static. The Manager says, "Hey, nothing is changing. The Eye doesn't need to scan every second; it can just guess the room is the same as it was a second ago. And the Body Scanner? The person isn't moving, so let's take a break."
Scenario B: Suddenly, a person walks in and drops a tray. The Manager's sensors (based on the previous second's data) scream, "Wait! Motion detected! A new human entered! The scene changed!"
The Action: The Manager instantly wakes up the Eye and the Body Scanner to get the full picture.

2. The "Cost vs. Reward" Calculator

The Manager doesn't just guess; it does a quick math problem for every second:

The Reward: "If I turn on the Body Scanner now, how much new information will I get?" (e.g., "Will I learn something new about where the person's hand is?")
The Cost: "How much battery and time will it take to run the scanner?"
The Decision: If the Reward is low (the person is still) and the Cost is high, the Manager says, "Skip it. I'll just estimate where they are based on where they were last second." If the Reward is high (someone is running), the Manager says, "Turn it on immediately!"

3. The "Prediction" Trick

When the Manager decides not to turn on a heavy scanner, it doesn't leave the robot blind. It uses a lightweight prediction (like a simple physics guess).

Analogy: If you know a car was moving at 30mph, and you don't look at it for one second, you can safely guess it's now 30 feet further down the road. You don't need to take a photo to know that. The robot does the same thing for people and objects.

The Results: Faster and Smarter

The researchers tested this "Smart Manager" on videos of people reading, eating, and walking. Here is what happened:

Speed Boost: The robot became 27% faster because it stopped wasting time on boring, repetitive frames.
Better Timing: It was 72% better at catching the exact moments when a human needed help (like when they stood up or dropped something).
Accuracy: Even though it skipped some scans, it didn't miss the important stuff. It was like a sniper who only fires when the target is in the crosshairs, rather than a machine gunner spraying bullets everywhere.

Why This Matters

In the future, robots working alongside humans (in hospitals, factories, or homes) need to be fast and efficient. They can't afford to be slow and tired.

This paper gives robots a "Reticular Activating System" (similar to how our human brains filter out background noise so we can focus on a conversation). It teaches the robot to ignore the boring stuff and focus intensely on what matters, making human-robot collaboration smoother, faster, and more natural.

1. Problem Statement

In modern Human-Robot Collaboration (HRC), robots rely on multiple perception modules (e.g., object detection, pose estimation) to understand dynamic environments. Current systems typically execute these modules on a frame-by-frame basis in parallel to ensure high accuracy. However, this approach creates significant challenges in streaming perception scenarios:

Latency Accumulation: Running heavy modules (like full-body pose estimation) every frame introduces substantial computational latency, degrading real-time responsiveness.
Resource Redundancy: Many frames contain redundant information (e.g., static backgrounds, stationary humans), yet modules are still invoked, wasting computational resources.
Limitations of Existing Solutions:
- Keyframe-based methods often require future frame access (unsuitable for real-time) or rely on task-agnostic criteria (visual saliency) that do not align with specific robotic tasks.
- Efficient perception methods usually optimize within a single frame (e.g., resizing inputs) but fail to address the higher-level decision of whether to activate a module at all.

The core problem is how to dynamically schedule perception modules in real-time to minimize computational load and latency without significantly compromising the accuracy of critical scene understanding.

2. Methodology

The authors propose a Relevance-Driven Perception Scheduling Framework that treats perception modules as a shared "Perception Toolkit." The system decides which modules to activate for each incoming frame based on an estimated Information Gain vs. Computational Cost trade-off.

A. Core Architecture

Perception Region Segmentation:
- The scene is segmented into regions (background, objects, humans) using the previous frame's RGB image and relevance predictions.
- Motion Status: Determined via frame differencing (grayscale change ratio) to identify moving vs. stationary regions.
- Relevance State: Updated based on prior relevance predictions, estimating the probability that a region contributes to the current human objective.
Reward Estimation (The Scheduling Policy):
- For each module $m_j$ $m_{j}$ , a reward $\rho_j$ $ρ_{j}$ is calculated:
  $\rho_j = \Phi_R(S_k, m_j) - C_j$
  - $\Phi_R$ : Estimated Information Gain (based on entropy reduction and relevance).
  - $C_j$ : Computational Cost (inference time penalty).
- Decision Logic: A module is activated if its reward is positive ( $\rho_j > 0$ ) or if a hard constraint requires it. This avoids combinatorial search, allowing independent per-module decisions.
Specific Reward Models:
- Object Detection (YOLO):
  - Gain: Detects scene composition changes (entry/exit of objects) via histogram analysis and frame differencing. Updates bounding box state estimates using Kalman Filter entropy reduction.
- Human Pose Estimation (MMPose):
  - Gain: Quantifies the reduction in uncertainty when moving from a coarse "human bounding box" to precise "keypoint locations."
  - Uncertainty Modeling: Uses a uniform distribution entropy model for the bounding box and a Gaussian distribution model (derived from confidence scores) for keypoints. The gain is the difference between pre-execution and post-execution entropy.
Fallback Mechanism:
- When a module is not activated, the system uses lightweight estimation (e.g., motion models, Kalman filtering) to approximate its output, maintaining stream continuity without the full computational cost.

3. Key Contributions

Novel Scheduling Concept: Introduces the first perception scheduling framework specifically for HRC that optimizes module activation at the frame level to minimize overall system load.
Information-Theoretic Utility Estimation: Proposes a versatile methodology for estimating module utility based on information theory (entropy reduction) and relevance, applicable to various perception modules.
New Evaluation Metric: Develops Activation Recall and Keyframe Accuracy to evaluate real-time scheduling performance, distinguishing between the ability to identify critical frames and the ability to process them within latency constraints.
Scalable Framework: Demonstrates a modular approach ("Perception Toolkit") that can theoretically extend to complex foundation models (e.g., Vision-Language Models) by adapting reward formulations.

4. Experimental Results

The framework was evaluated on three video domains: Indoor Reading (static), Eating (moderate interaction), and Walking (dynamic). It was compared against a Parallel Pipeline (baseline) and an Oracle Scheduler (ideal ground truth).

Latency Reduction: The proposed framework reduced computational latency by up to 27.52% compared to the parallel pipeline. The reduction was most significant in static scenes (Indoor Reading) where redundant computation was highest.
Activation Recall:
- MMPose (Pose Estimation): Achieved a 72.73% improvement in activation recall in the Walking domain compared to the parallel baseline. This indicates the scheduler successfully identified critical dynamic frames that the baseline missed due to latency.
- YOLO (Object Detection): Maintained high recall (0.93–0.98), with only a minor drop compared to the baseline, proving that efficiency gains did not severely compromise detection quality.
Keyframe Accuracy: The framework achieved up to 98% accuracy in identifying frames that should be processed, demonstrating high precision in relevance detection independent of inference latency.
Trade-offs: While MMPose recall remained lower than ideal due to inherent inference delays (missing frames during processing), the framework significantly reduced the number of missed critical events compared to standard pipelines.

5. Significance

This paper addresses a critical bottleneck in deploying intelligent robots: the conflict between high-fidelity perception and real-time constraints.

Efficiency: It shifts the paradigm from "process everything every frame" to "process only what matters," enabling robots to operate with lower latency and on constrained hardware.
Context-Awareness: By integrating human intent and scene dynamics into the scheduling loop, the system mimics the human Reticular Activating System (RAS), prioritizing relevant sensory inputs.
Scalability: The framework provides a systematic solution for managing increasing numbers of perception modules (e.g., adding VLMs) without linearly increasing computational costs, making it a foundational step for next-generation collaborative robotics.