Technical Summary: EventVLA

Problem Statement

Standard Vision-Language-Action (VLA) policies for robotic manipulation typically operate under a strict Markovian assumption, mapping current observations and language instructions directly to actions. This approach fails in long-horizon, non-Markovian tasks where critical task-relevant information becomes transiently occluded or unobservable over time. For instance, a robot may need to recall the original location of a displaced item or the color of an object briefly revealed under a cover.

Existing memory-augmented solutions suffer from three primary limitations:

1. Dual-system architectures: Decouple cognition from control, leading to high latency and severe error propagation.
2. Recurrent architectures: Compress history into hidden states, creating an information bottleneck that discards fine-grained visual details.
3. Memory Buffers: Blindly accumulate redundant frames without selective mechanisms, incurring high computational overhead and drowning out sparse, critical evidence.

The core challenge addressed is determining when and what visual evidence a VLA should preserve to maximize execution success without overwhelming computational limits.

Methodology: EventVLA Framework

EventVLA is an end-to-end framework designed around the concept of sparse visual evidence memory. It avoids massive redundancy by combining static heuristics with a dynamic, predictive memory module.

1. Foundational Visual Anchors ($A_t$)

To handle structurally simple memory requirements, EventVLA employs a deterministic, rule-based baseline comprising:

• Initial Frame ($o_0$): Captures the invariant global layout of the workspace.
• Short-term History Window: A sliding window of recent frames ($o_{t-K} \dots o_{t-1}$) providing local motion cues.
  These anchors serve as a permanent spatial reference and immediate temporal context but fail to capture unpredictable, transient events occurring mid-execution.

2. Keyframe Evidence Memory (KEM) Module

To capture transient, interaction-driven events (e.g., an object briefly revealed under a cover), EventVLA introduces the KEM module.

• Foresight-Driven Prediction: Unlike rule-based heuristics, KEM operates as a lightweight, parallel prediction head alongside the main action head. It ingests the hidden states ($h_t$) from the VLA's autoregressive transformer.
• Chunk-wise Probability Forecasting: Instead of step-wise classification, KEM predicts a vector of probabilities ($\hat{p}_t$) spanning a future execution horizon ($H$). This allows the policy to proactively identify "memory schedules" for critical intermediate states before they become unobservable.
• Sparse Memory Writes: When a predicted probability exceeds a threshold ($\hat{p}_i^t \ge \tau_{commit}$), the raw image at that future timestep is committed to an event buffer ($E_t$).
• Buffer Management: The event buffer is bounded by a maximum capacity ($N_{max}$) and managed via a First-In-First-Out (FIFO) eviction policy to ensure real-time efficiency.

3. Training and Inference

• Automatic Labeling: To avoid prohibitive manual annotation, the authors utilize an offline pipeline based on Qwen3-VL to extract ground-truth timestamps of task-critical events from demonstrations.
• Loss Function: The model is trained end-to-end using a standard action generation loss ($L_{action}$) combined with a sequence-averaged Binary Cross-Entropy loss ($L_{kem}$) for keyframe prediction, utilizing temporally smoothed soft labels to mitigate temporal ambiguity.
• Curriculum Learning: A scheduled teacher-to-student curriculum transitions memory construction from ground-truth labels to autonomous predictions to bridge train-test distribution shifts.
• Inference: During inference, dense probability predictions are distilled into discrete write events using 1D Non-Maximum Suppression (NMS) and a temporal cooldown pipeline to enforce strict sparsity.

Key Contributions

1. The EventVLA Framework

The paper introduces a novel architecture that seamlessly unites foundational visual anchors with a dynamic, foresight-driven KEM module. This approach allows the policy to autonomously capture and store sparse, task-critical visual events, bridging the temporal gap between an event's brief appearance and its eventual utility in downstream execution.

2. RoboTwin-MeM Benchmark

The authors propose RoboTwin-MeM, a diagnostic simulation benchmark specifically designed to evaluate non-Markovian manipulation tasks with interactive visual evidence.

• Distinctiveness: Unlike existing benchmarks (e.g., RMBench) that can often be solved by static anchors, RoboTwin-MeM features tasks where critical evidence emerges transiently and disappears.
• Structure: It comprises 8 challenging tasks with execution horizons averaging 430–1544 steps. Task complexity is parameterized by $n$ (1 to 5), representing the exact number of intermediate keyframes that must be memorized.
• Capabilities Tested: The benchmark evaluates transient memory, sequence tracking/counting, and in-context learning.

Experimental Results

Simulation Benchmarks

• RMBench (Standard Memory Tasks): EventVLA utilizing only foundational visual anchors achieved a 67.8% average success rate, setting a new state-of-the-art and demonstrating that rule-based anchors suffice for tasks with persistent spatial layouts.
• RoboTwin-MeM (Non-Markovian Tasks):
  • Baseline non-memory VLAs (e.g., $\pi_{0.5}$) achieved only 7.8%.
  • Existing memory-augmented VLAs (e.g., MemoryVLA) achieved 10.8%.
  • EventVLA with only visual anchors (VA only) achieved 18.0%, highlighting the insufficiency of static anchors for transient events.
  • Full EventVLA (VA + KEM) achieved a 75.2% average success rate, vastly outperforming all baselines.
  • Ablation studies confirmed that removing KEM, using implicit latent memory, or restricting buffer capacity significantly degrades performance.

Real-World Evaluation

Evaluated on the ARX ACONE bimanual robot across four non-Markovian tasks (Find Block, Pick-X-Times, Pick in Order):

• $\pi_{0.5}$ (Reactive): Failed almost entirely (0–10% success).
• $\pi_{MEM}$ (Memory-Augmented Baseline): Showed partial success but degraded on complex multi-event tasks (e.g., 30–40%).
• EventVLA: Achieved success rates of 60% to 90% across tasks, demonstrating robust situational awareness and the ability to retain critical transient visual cues in physical environments.

Significance and Claims

The paper claims that EventVLA addresses the critical bottleneck of memory in long-horizon robotic manipulation by shifting from "blind accumulation" to "sparse, event-driven preservation." By proactively predicting and storing only task-critical visual evidence, the framework achieves:

1. Efficiency: Avoids the computational overhead and redundancy of dense memory buffers.
2. Robustness: Enables agents to handle non-Markovian dynamics where information is transiently occluded.
3. Scalability: Demonstrates superior performance across both simulation and real-world bimanual tasks, significantly outperforming state-of-the-art memory-augmented VLAs.

The authors conclude that while the bounded event buffer may limit scalability in exceptionally long-horizon tasks with high event densities, the proposed framework establishes a new paradigm for memory-efficient, long-horizon robotic control.