SCOPE: Real-Time Natural Language Camera Agent at the Edge

Technical Summary: SCOPE: Real-Time Natural Language Camera Agent at the Edge

Problem Statement
Deploying language-driven agents in robotics requires evaluations that reflect real-world task demands, specifically natural-language instructions with reproducible outcomes. Existing benchmarks for language models often fail to capture the closed-loop behavior, sim-to-real transferability, and deployment-critical constraints (such as latency, cost, and error modes on edge hardware) necessary for embodied control. Specifically, controlling a pan–tilt–zoom (PTZ) camera via natural language involves a non-trivial feedback loop coupling perception, reasoning, and motion under strict memory and latency budgets. Current evaluations remain indirect for such embodied tasks, making it difficult to predict agent behavior and failure modes in real robotic control loops.

Methodology and System Architecture
The authors present SCOPE (Simulation and Camera Operations for Perception and Evaluation), a modular framework and benchmark for executable, closed-loop evaluation of language-driven PTZ camera agents.

• Architecture: SCOPE employs a decoupled design where a compact Small Language Model (SLM) serves as a high-level planner, interleaving reasoning with camera control and perception queries. Visual understanding is delegated to lightweight Vision-Language Models (VLMs) exposed as callable tools. This separation avoids streaming image tokens into the dialogue, which would compound latency, and instead allows the planner to reason over concise, task-aligned outputs (e.g., JSON-formatted counts or OCR results).
• Tool Interface: The SLM interacts with the environment via a fixed set of OpenAI-compatible JSON schema tools. These include direct camera control commands (ADJUST_PTZ, GO_TO_PRESET, TAKE_IMAGE) and perception tools backed by VLMs (COUNT_OBJECTS, QUERY_ANSWER, ZOOM_TO_OBJECT, TRACK_OBJECT).
• Simulation Environment: To enable reproducible evaluation, the authors implemented a Blender-based simulation environment that mirrors the PTZ action space and observation loop of a physical AXIS M5075-G PTZ camera. The simulation supports deterministic replay and includes a "panorama" tool that stitches frames from a deterministic sweep to handle queries requiring awareness beyond a single frame.
• Benchmark Construction: The authors released a 536-task benchmark spanning QA, single- and multi-step commands, counting, spatial reasoning, descriptions, and optical character recognition (OCR). Tasks were generated using ground-truth annotations of four diverse urban Blender scenes and refined through an iterative validation process involving an LLM-as-Judge (gpt-oss-120B) and manual inspection.

Key Contributions

1. SCOPE Framework: A modular, language-driven PTZ camera agent operating in both simulation and on physical hardware, enabling closed-loop control under real deployment constraints.
2. Sim-to-Real Benchmark: A Blender-based simulation framework with a 536-task benchmark that exposes realistic PTZ control affordances and enables faithful sim-to-real evaluation.
3. Systematic Evaluation: An analysis of 19 planner–perception model combinations, measuring accuracy, latency, and error modes to identify architectural choices suitable for edge deployment.

Results
The study evaluated 19 combinations pairing Qwen3 SLMs (ranging from 4B to 80B parameters, including Mixture-of-Experts and quantized variants) with Moondream and Qwen VLMs.

• Error Decomposition: Errors are structured by model capacity. Weaker SLM planners frequently suffer from hallucinations and improper tool usage (e.g., incorrect arguments or routing). As planner capacity increases, these failures diminish, and visual perception (VLM query and counting errors) becomes the dominant bottleneck. In an oracle setting where VLM errors are treated as correct, average accuracy increases from 66.2% to 82.0%, indicating that perception failures account for 46.7% of errors on average.
• Task Difficulty: Accuracy on visually demanding categories (Counting: 52.6–76.8%; OCR: 38.9–83.3%) is substantially lower than on tool-centric categories (Single Call: 97.2–100.0%; Multi-step Command: 86.0–98.2%).
• Latency: End-to-end latency is dominated by perception rather than planning. VLM backbones vary widely in inference cost, and expanding observation scope to a full panorama increases latency by 22.4%. Planner latency is comparatively stable; MoE planners achieve inference times close to small dense models despite larger total capacity.
• Architecture Selection: Mixture-of-Experts (MoE) models on both the planning and perception sides consistently match or exceed dense alternatives while operating at latencies and memory footprints comparable to much smaller networks. Quantization provides additional efficiency gains with minimal accuracy degradation. The best-performing configuration was Moondream3 paired with Qwen3-30B-A3B, achieving 73.8% task success.

Significance and Claims
The paper claims to identify a practical, sim-to-real–validated design point for real-time, edge-feasible language-driven PTZ control. The authors argue that:

1. Modularity is Essential: Decoupling planning from perception enables predictable latency and modularity, making real-time edge deployment practical.
2. Perception is the Ceiling: Once a sufficiently capable SLM is used, perception becomes the dominant performance bottleneck. Future accuracy gains will hinge on stronger VLMs.
3. MoE and Quantization are Viable: Compact MoE and quantized architectures provide the best accuracy–latency balance. Scaling planners beyond 30B-A3B yields no clear accuracy gains at substantially higher memory costs, making 30B-A3B the most favorable deployment point.
4. Sim-to-Real Parity: By exposing an identical PTZ action space and observation loop in both simulation and on physical hardware, SCOPE enables faithful sim-to-real transfer and evaluation, addressing a gap in existing benchmarks that struggle to make such claims.