Technical Summary: LiteVLA-H

1. Problem Statement

Vision-Language-Action (VLA) models have demonstrated strong semantic grounding and task generalization in robotic manipulation. However, deploying these models on aerial robots (drones) remains challenging due to strict onboard compute and communication constraints. Aerial robots require low-latency closed-loop guidance to react to visual changes, yet they also benefit from higher-level semantic understanding for obstacle description, runway awareness, and operator supervision.

Existing literature often emphasizes navigation success, benchmark creation, or server-class inference. A critical gap remains in addressing the practical scheduling problem for compact onboard models: how can a single edge VLA support both reactive, high-frequency guidance and slower, sentence-level semantic reasoning without compromising either? Current approaches often treat inference as a single end-to-end metric, failing to account for the specific latency bottlenecks of edge deployment.

2. Methodology: LiteVLA-H

The authors propose LiteVLA-H, a compact 256M-parameter VLA system designed for dual-rate operation on an NVIDIA Jetson AGX Orin. The system is built around three core technical pillars:

A. Dual-Rate Scheduling Architecture

The system explicitly separates the VLA's output into two distinct modes managed by a scheduler:

1. Fast Outer-Loop Guidance: Produces short-horizon action tokens (1–2 tokens) at approximately 19.74 Hz (50.65 ms latency). These tokens serve as high-level commands (e.g., velocity, heading, waypoints) for the outer loop, while a conventional flight controller handles inner-loop attitude stabilization.
2. Slow Semantic Mode: Produces sentence-level descriptions for scene understanding, hazard description, and operator narration at 6.08–6.67 Hz (149.90–164.57 ms latency).

The scheduler maintains a relationship where the semantic query period ($\Delta_s$) is an integer multiple ($K > 1$) of the action query period ($\Delta_a$), allowing semantic updates to occur opportunistically without blocking critical action deadlines.

B. Latency Decomposition and Pre-fill Dominance

A central empirical observation of this work is that in the compact edge regime, end-to-end latency is dominated by multimodal pre-fill rather than token decoding.

• Pre-fill Cost ($P$): Approximately 47.8 ms (shared vision-language prompt pass).
• Marginal Decoding Cost ($D_i$): Approximately 1.4 ms per token.
• Implication: For short outputs (e.g., a single action token), the pre-fill fraction ($\rho$) is ~94.4%. This means optimizing the number of decoded tokens yields diminishing returns; the system bottleneck is the initial image-language fusion. Consequently, the system can afford to run semantic decoding (which requires more tokens) at a lower rate without significantly impacting the action loop, provided the pre-fill path is not re-computed unnecessarily.

C. Knowledge-Preserving Fine-Tuning

To specialize the model for aerial tasks without "catastrophic forgetting" of general visual-language capabilities, the authors employ a mixed-loss training objective:
$$\mathcal{L} = \lambda_a\mathcal{L}_{act} + \lambda_s\mathcal{L}_{sem} + \lambda_g\mathcal{L}_{gen} + \lambda_{kp}\mathcal{L}_{kp}$$

• $\mathcal{L}_{act}$: Reactive flight data (action tokens).
• $\mathcal{L}_{sem}$: Aerial semantic data (scene descriptions).
• $\mathcal{L}_{gen}$: Generic caption/VQA supervision to maintain broad competence.
• $\mathcal{L}_{kp}$: An optional knowledge-preserving regularizer (distillation against a pre-specialized backbone) to prevent the model from collapsing into a narrow action classifier.

3. Key Contributions

1. Identification of Pre-fill Dominance: The paper establishes that for compact edge-deployed VLAs, the "time-to-first-action" is the primary systems bottleneck, driven by multimodal pre-fill rather than decoding length.
2. Dual-Rate Scheduler: Introduction of a scheduling strategy that achieves 19.74 Hz outer-loop action emission while maintaining 6.08–6.67 Hz sentence-level semantic perception on a single Jetson AGX Orin.
3. Knowledge-Preserving Training: A fine-tuning recipe that successfully balances reactive flight performance with retained general visual-language competence (CIDEr and F1 scores).
4. Systems-Level Positioning: Comparative analysis against recent state-of-the-art frameworks (AnywhereVLA, FutureVLA, ReMem-VLA), demonstrating that LiteVLA-H achieves higher edge inference rates for action while retaining periodic semantic awareness, without the latency penalties associated with heavy memory or prediction modules in this specific embedded context.

4. Results

Experiments were conducted on an NVIDIA Jetson AGX Orin with FP16 precision and a context window of 2048.

• Latency Performance:

  • Reactive Guidance: 50.65 ms (19.74 Hz).
  • Scene Captioning: 149.90 ms (6.67 Hz).
  • Contextual Awareness (3 sentences): 164.57 ms (6.08 Hz).
  • The semantic branch is approximately 3x slower than the action branch, validating the feasibility of the dual-rate approach.

• Ablation Studies:

  • Data Mixture: Action-only fine-tuning resulted in severe catastrophic forgetting (CIDEr dropped to 0.31). Adding generic caption/VQA rehearsal recovered general capability (CIDEr 0.82) while keeping action success within 1.1% of the action-only variant.
  • Runtime Efficiency: Compared to single-rate semantic execution (6.67 Hz) or memory-heavy variants (ReMem-VLA, FutureVLA), LiteVLA-H maintained the high action rate (19.74 Hz) with only a 0.1 GB memory increase and 3.6 W power increase over the action-only baseline.
  • Closed-Loop Performance: In simulated runway navigation and hardware obstacle courses, LiteVLA-H achieved an 84.1% and 81.3% task success rate respectively, outperforming baselines like ReMem-VLA (78.2%) and FutureVLA (70.0%) in the hardware setting, with significantly lower latency (50.65 ms vs. 205–250 ms).

5. Significance and Claims

The paper positions LiteVLA-H not as a universal dominance over all robotic settings, but as a specialized solution for the compact, onboard aerial regime.

• Design Philosophy: The authors argue that onboard VLA design should optimize time-to-first-action separately from sentence-level response latency. By recognizing pre-fill dominance, they demonstrate that a single model can serve two timescales effectively.
• Safety and Control: The system is designed to operate at the outer-loop level, providing guidance commands rather than direct motor control. This separation allows the VLA to be filtered by downstream safety logic (e.g., clamping commands, emergency stops), reducing the safety burden on the model itself.
• Modesty: The authors acknowledge limitations, noting that the strongest evidence is onboard inference timing. Broader closed-loop flight evaluation across varying wind, lighting, and payload conditions is still required. Comparisons to prior work are not fully controlled due to differences in hardware and embodiment, but the paper claims that under the reported Jetson AGX Orin deployment, their dual-rate approach offers a superior trade-off between reactivity and semantic awareness.

In summary, LiteVLA-H demonstrates that by exploiting the pre-fill-dominant nature of edge inference and employing a dual-rate scheduler, it is possible to deploy a compact VLA that reacts at near-real-time speeds (20 Hz) while still providing meaningful semantic context (6 Hz) for aerial autonomy.