Technical Summary: Functional Cache Grafting (FCGRAFT)

Problem Statement

Embodied agents operating in open-domain environments face significant challenges when utilizing Code-writing Large Language Models (CodeLLMs) for policy synthesis. The prevailing "Code-as-Policies" (CaP) paradigm, which translates natural language instructions into executable control code, suffers from two fundamental limitations in dynamic settings:

1. Latency: Fully generative decoding requires repetitive prefill computation over long prompts for every new instruction, causing delayed responses that undermine responsiveness in time-critical scenarios.
2. Robustness: Generating full code from scratch often leads to API mismatches, missing safety guards, and unstable control logic, resulting in task failures.

Existing memory-based approaches primarily operate at the text level, offering limited computational efficiency gains. Similarly, prior Key-Value (KV) caching methods generally support only prefix-based reuse or document-level caching, failing to accommodate the function-level adaptation required for the dynamic and evolving nature of open-domain embodied tasks.

Methodology

The paper proposes FCGRAFT (Functional Cache Grafting), a framework designed to enhance CodeLLM-based robotic programming through function-level KV caching and a unified cache-grafting synthesis pipeline.

1. Two-Tier Function-Level KV Cache

FCGRAFT restructures the native KV caching mechanism into a hierarchical two-tier code cache ($H$), indexed by function identifiers ($f_k$):

• Function-Interface Tier ($I$): Stores callable function signatures (text) and their associated KV states ($i^{KV}_k$). These lightweight entries allow the model to reuse function interfaces without recomputing cross-attention, serving as the basis for cache-stitching.
• Function-Code Tier ($C$): Stores validated full implementations ($c^{text}_k$) and their corresponding KV states ($c^{KV}_k$). These are used during execution and for cache-patching, providing the context needed to repair specific code spans without regenerating the entire policy.

Cache Management: To manage GPU memory constraints, the framework assigns a locality score ($\ell(f_k)$) to each function based on:

• Recency ($\ell_{curr}$): Immediate retention of recently used functions.
• Frequency ($\ell_{freq}$): Usage frequency over recent tasks.
• Association ($\ell_{asso}$): Conditional co-occurrence with other functions in execution traces.
• Semantic Diversity ($\ell_{sema}$): Perplexity-based distinctiveness to retain rare but useful functions.
  Low-scoring entries are offloaded to DRAM, while high-scoring ones remain on the GPU.

2. Cache-Grafting Synthesis Pipeline

New policies are synthesized through two interdependent stages:

• Cache-Stitching: When a new task arrives, the system retrieves relevant function interfaces from tier $I$. The CodeLLM composes a new policy by injecting the cached KV states of these interfaces. This eliminates redundant prefill computation for validated function structures, ensuring that internal control logic and API sequences are robust and pre-verified.
• Cache-Patching: If execution errors occur (e.g., API mismatches, runtime exceptions), the system triggers cache-patching. The policy is split into a prefix, an error span, and a suffix. The system reuses the KV states of the prefix and generates only the corrected middle span, guided by a Chain-of-Thought (CoT) trace identifying the root cause. This avoids full regeneration, drastically reducing decoding time.

Key Contributions

1. FCGRAFT Framework: A novel approach that improves CodeLLM-based robotic programming by implementing function-level KV caching, reducing redundant prefill while ensuring behavioral consistency in open-domain environments.
2. Interdependent Mechanisms: The introduction of cache-stitching (to eliminate structural errors and reduce prefill) and cache-patching (to correct localized errors with minimal decoding) as a unified pipeline.
3. Empirical Validation: Demonstration that FCGRAFT achieves a superior trade-off between robustness and latency across diverse benchmarks (ALFRED, TEACh, RLBench) and real-world robotic manipulation tasks.

Experimental Results

The authors evaluated FCGRAFT against nine competitive baselines, including general CodeLLM methods (CaP, SCoT), memory-based approaches (HELPER, LRLL, PromptBook), and KV caching methods (CAG, RAGCache, PromptCache, EPIC).

• Performance Gains: Across simulation benchmarks, FCGRAFT achieved an average 18.31% higher task success rate (SR) and 2.3× faster policy synthesis latency compared to the strong baseline RAGCache.
• Specific Benchmarks:
  • ALFRED: Outperformed the second-best baseline (LRLL) by 4.04% in SR and reduced latency by 2.67×.
  • RLBench: Achieved a 16.21% higher SR and 2.25× latency reduction compared to CAG, particularly excelling in tasks requiring fine-grained control and loop structures.
  • Real-World Robotics: In physical robot experiments (Office Desk Rearrangement and Cooking Workstation Preparation), FCGRAFT achieved a 37.04% higher SR than CaP and reduced latency by 2.88× compared to RAGCache.
• Consistency: The framework maintained high code similarity (CSIM) and positive forward transfer (FWT) without catastrophic forgetting (non-negative backward transfer), indicating stable accumulation of reusable skills.

Significance and Claims

The paper claims that FCGRAFT represents a significant step toward scalable and general-purpose embodied intelligence by enabling agents to generate and adapt code policies rapidly and reliably.

• Robustness via Reuse: By reusing validated control structures and API call patterns from the cache, the framework biases policy synthesis toward proven program structures, avoiding common failure modes like incompatible primitive compositions.
• Efficiency via Grafting: The separation of invariant regions (handled by stitching) from adaptation points (handled by patching) allows the system to bypass redundant computation while maintaining the flexibility to correct errors locally.
• Practicality: The successful transfer of the framework from simulation to real-world robotic manipulation demonstrates its viability for time-critical, safety-sensitive applications where rapid response to unexpected state changes (e.g., gas leaks, object misplacements) is crucial.

The authors acknowledge limitations, noting that the current implementation maintains an isolated code cache per agent, which may limit scalability in multi-agent cooperative settings. Future work is proposed to explore code cache sharing and learning-based eviction policies.