🤖 AI

Coordinated Networking for On-Device Agent-Augmented Real-Time Communication

This paper presents HFS, a framework that utilizes an app-guided multi-flow transport approach to effectively manage network contention between live video and agent context traffic in on-device agent-augmented real-time communication, thereby significantly improving video quality and reducing agent response latency.

Original authors: Goodsol Lee, Juheon Yi, Jinglu Wang, Haowen Xu, Saewoong Bahk, Yan Lu

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Goodsol Lee, Juheon Yi, Jinglu Wang, Haowen Xu, Saewoong Bahk, Yan Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are at a bustling digital party where everyone is trying to do two things at once: have a lively, face-to-face conversation and work together on a massive, complex project with the help of a super-smart robot assistant. In the world of computer science, this is called "Real-Time Communication" (RTC), the technology that powers video calls like Zoom or Teams. Usually, these calls are just about humans talking. But a new wave of technology is introducing "AI agents"—smart programs that can read documents, find facts, and write drafts for you while you talk. The big question scientists are asking is: What happens when your video call is trying to stream your face, and your robot assistant is trying to stream a giant pile of data to do its job? If they fight over the same internet connection, one of them will likely get stuck, making the video freeze or the robot take forever to answer.

This paper introduces a clever new system called HAFS (Human-Agent Flow Scheduling) to solve this traffic jam. The researchers found that when you try to run these AI agents directly on your own device (like your laptop or phone) instead of a giant cloud server, you get better privacy and lower costs, but you create a new kind of internet traffic chaos. They discovered that without a traffic cop, the robot's data floods the road, causing your video to stutter and look blurry, or the robot to sit idle while waiting for its data to arrive. HAFS acts like a super-organized traffic controller that knows exactly how much space the video needs to stay smooth and how much space the robot needs to work fast. By carefully balancing the two, the system ensures your video stays crystal clear while your AI assistant gets its job done in record time.

The Problem: A Digital Two-Lane Highway

Picture your internet connection as a two-lane highway. On one lane, you have "Human Video," which is like a stream of high-speed race cars. These cars need to move smoothly and constantly; if they hit a traffic jam, the video freezes, and the conversation breaks. On the other lane, you have "Agent Context," which is like a fleet of heavy delivery trucks carrying giant boxes of information. These trucks are trying to deliver data so the AI can answer questions, like "Summarize this 1,000-page legal contract."

The problem is that in current video call systems, these two lanes don't talk to each other. When the delivery trucks (the AI) get a big order, they just start driving as fast as they can, filling up the highway. This causes the race cars (your video) to get stuck in a traffic jam behind them. The video system sees the delay and panics, thinking the road is broken, so it slows the cars down to a crawl, making your video look pixelated and low-quality. Meanwhile, the AI trucks are still stuck in the jam, waiting for the road to clear.

The researchers tested this on real video call platforms like Zoom and Microsoft Teams. They found that when they tried to send AI data alongside a video call, the video quality dropped by more than half. Even worse, once the AI finished its job, the video took over 30 seconds to get back to normal speed. It was a messy, inefficient mess where neither the humans nor the robots could do their jobs well.

The Solution: HAFS, the Smart Traffic Cop

To fix this, the team built HAFS, a system that acts like a smart traffic cop sitting right inside your video call app. Instead of letting the video and the AI fight for space, HAFS looks at the whole highway and decides exactly how fast each lane should go.

1. The "Frame-Level" Watchdog
The first job of HAFS is to watch the video lane very closely. It doesn't just look at the average speed; it looks at the tiny gaps between every single video frame (like the space between individual frames in a flipbook). The system knows that the video can tolerate a tiny bit of waiting, but only up to a specific limit (about 150 milliseconds). If the AI trucks start to push the video cars too close to that limit, HAFS immediately tells the trucks to slow down. But if there is extra space on the road, HAFS tells the trucks to speed up and fill the empty lanes. This way, the video stays smooth, and the AI uses every bit of extra space available without causing a crash.

2. The "Predictive" Scheduler
The second job is even smarter. Sometimes, the AI needs to send many trucks at once to answer one question. In old systems, these trucks would line up one by one, like cars at a toll booth, waiting for their turn. This is slow. HAFS, however, looks at the size of the boxes each truck is carrying and guesses how long it will take to unpack them once they arrive. It then arranges the trucks in the perfect order so that as soon as one truck arrives and starts being unpacked, the next one is already pulling up to the dock. This is called "pipelining." It's like a chef who starts chopping vegetables for the second dish while the first dish is still frying, so nothing sits idle.

The Results: A Faster, Clearer Future

The researchers tested HAFS on real devices, including a MacBook Pro, a Samsung Galaxy phone, and a small computer called a Jetson. They ran simulations with different types of AI models and network speeds, from Wi-Fi 6 to 5G.

The results were impressive. Compared to the old way of doing things, HAFS made the video quality 1.5 times better. At the same time, it cut the time it took for the AI to respond by 31%. In some cases, the AI was able to finish its job in just 16 seconds instead of the usual 105 seconds, simply because it wasn't stuck waiting in traffic.

The paper suggests that this approach is a promising way to make AI assistants a normal part of our video calls without ruining the experience. By keeping the humans and the robots working together in harmony, we can have high-quality video calls where the AI is always ready to help, right there on your own device, keeping your secrets safe and your internet running smoothly.

Technical Summary: Coordinated Networking for On-Device Agent-Augmented Real-Time Communication

1. Problem Statement

The emergence of AI agents is driving a new paradigm in Real-Time Communication (RTC): Agent-Augmented RTC. In this model, AI agents run locally on user devices to autonomously retrieve, analyze, and generate information to support human collaboration (e.g., co-authoring legal documents or summarizing meeting logs).

While on-device agents offer significant advantages in privacy (keeping sensitive data encrypted end-to-end) and cost (offloading GPU inference from the cloud), they introduce a critical networking challenge: contention between heterogeneous traffic flows.

Human Video Flows: Require low latency (<150 ms) and high quality, typically managed by delay-based congestion control (e.g., GCC).
Agent Context Flows: Involve streaming large Key-Value (KV) caches (up to 80 MB after compression) to enable fast LLM inference. These flows use loss-based congestion control (e.g., SCTP) to maximize throughput, aggressively filling network buffers.

The Core Conflict: When these flows share a network, the aggressive nature of agent context flows causes queuing delays for human video. This triggers delay-based video controllers to drastically reduce bitrates (e.g., a 53% drop in VMAF scores) and recover slowly (>30 seconds). Furthermore, when multiple agents stream contexts simultaneously for a single query, standard round-robin scheduling leads to inflated queuing delays and underutilized compute resources, increasing agent response time by up to 56%.

2. Methodology: HAFS Framework

The authors propose HAFS (Human-Agent Flow Scheduling), a coordinated networking framework designed to ensure high video quality and low agent response latency simultaneously. HAFS operates as an app-guided multi-flow transport layer atop standard WebRTC and llama.cpp, requiring no modifications to the underlying network infrastructure.

2.1 System Architecture

HAFS utilizes a Selective Forwarding Unit (SFU) architecture common in modern RTC. It introduces two coordinators operating at the app layer on both the sender and the SFU:

Human-Agent Flow Coordinator: Jointly controls the sending rates of video (RTP) and agent context (SCTP) flows.
Multi-Agent Flow Coordinator: Schedules the transmission order of concurrent agent flows to minimize total response time.

2.2 Key Design Components

A. Human-Agent Flow Coordination

To resolve contention between video and agent traffic, HAFS employs a frame-level queuing-aware rate control mechanism:

Dual Feedback Estimator: It estimates queuing delay by combining RTP feedback (packet delay) with SCTP ACKs. SCTP ACKs densely populate the idle intervals between video frames, allowing the system to measure queuing caused specifically by agent traffic without interfering with native RTP pacing.
Joint Rate Control (MI/AI-MD): The system ramps up the aggregate sending rate until the frame-level queuing delay approaches, but does not exceed, the app-specified deadline (e.g., 150 ms).
- Multiplicative Increase (MI): Rapidly increases rate when the queue is empty.
- Additive Increase (AI): Gently increases rate when the queue is building but below the deadline.
- Multiplicative Decrease (MD): Reduces rate if the deadline is approached.
Bandwidth Allocation: Once the total rate is determined, the system strictly provisions the required bitrate for live video and allocates the remaining "slack" capacity to agent flows.
Agent-Probed Bandwidth: Since agent flows often probe at rates higher than video, the system uses the observed agent throughput to estimate available bandwidth, allowing the video controller to recover quickly from congestion without waiting for slow delay-based feedback loops.

B. Multi-Agent Flow Coordination

To handle multiple concurrent context streams, HAFS treats the scheduling problem as a two-machine flow shop problem (network transmission followed by LLM decoding).

Pipelining-Aware Scheduling: The system predicts the transmission time ( $T_{net}$ $T_{n e t}$ ) and computation time ( $T_{comp}$ $T_{co m p}$ ) for each flow. It applies Johnson's rule to determine the optimal transmission order:
- Flows where $T_{net} \le T_{comp}$ are scheduled first (sorted by $T_{net}$ ascending).
- Flows where $T_{net} > T_{comp}$ are scheduled last (sorted by $T_{comp}$ descending).
Prediction Strategy: Since LLM output length is unknown before decoding, HAFS performs a lightweight single-step LLM decoding using the prebuilt KV cache to predict the response length. This prediction is used solely for relative ordering, making the system robust to absolute prediction errors.
Robustness Mechanisms:
- Dynamic Priority Update: Re-evaluates priorities if network bandwidth fluctuates significantly (>20%).
- Overlapped Prefill: If network transfer is slower than local prefill, the system opportunistically streams the raw text context ahead of the KV cache to keep the receiver's compute resources busy.

3. Key Contributions

Identification of a New Networking Paradigm: The paper highlights the specific contention issues arising from the coexistence of latency-sensitive video and throughput-heavy agent context flows in on-device RTC, a scenario not addressed by traditional RTC or cloud-based agent architectures.
App-Guided Multi-Flow Transport: HAFS introduces a unified app-layer orchestrator that achieves in-network-like precision in flow scheduling by leveraging the short RTTs of SFU-based RTC architectures, avoiding the need for router-level modifications.
Novel Rate Control Mechanism: The design of a frame-level queuing estimator that utilizes SCTP ACKs to isolate agent-induced queuing from video traffic, enabling simultaneous optimization of video quality and agent throughput.
Compute-Network Pipelining: The formulation of multi-agent flow scheduling as a flow shop problem, utilizing lightweight LLM prediction to minimize end-to-end response latency.

4. Experimental Results

The authors prototyped HAFS using WebRTC and llama.cpp, evaluating it across diverse environments (MacBook Pro, Samsung Galaxy S25, NVIDIA Jetson Orin Nano) and networks (Wi-Fi 6, 5G).

Video Quality: HAFS achieves 1.5× higher video quality (measured by VMAF) compared to baselines (Zoom, Teams) when agent flows are active, effectively maintaining video quality comparable to video-only streaming.
Agent Latency: HAFS reduces agent response time by 31% compared to state-of-the-art baselines.
Scalability: The system maintains performance gains across varying numbers of concurrent agent flows, different LLM models (Qwen, Llama families), and network conditions.
Overhead: The framework incurs negligible overhead in power consumption, CPU, and memory usage.

5. Significance

The paper claims that HAFS provides a practical, deployable solution for the next generation of collaborative AI applications. By enabling on-device agent-augmented RTC, it addresses the dual challenges of privacy (avoiding cloud data exposure) and cost (reducing cloud GPU dependency) while solving the inherent networking contention that previously made such systems unviable. The work demonstrates that with coordinated, app-guided transport strategies, it is possible to support high-bandwidth, latency-sensitive AI workloads alongside real-time human communication without compromising the user experience.