Technical Summary: Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference

1. Problem Statement

Long-running Large Language Model (LLM) agents maintain substantial state on GPUs, including KV caches, request schedulers, communication states, and online adapters. Unlike traditional stateless inference, these workloads accumulate minutes to hours of computation. A failure in a GPU rank, communicator, or application kernel can destroy this state, rendering the session unrecoverable without restarting the entire serving stack, reloading model weights, and replaying tool calls—a process that is often semantically impossible once external interactions have occurred.

Existing recovery mechanisms face a fundamental trade-off:

• Application-specific logging: Requires modifying every attention kernel, allocator, and library to log KV updates explicitly. This is brittle in modern stacks involving fused libraries, vendor kernels, and dynamic graph compilers.
• Host-mediated checkpointing: Relies on the CPU to launch checkpoint kernels or scan GPU memory. This introduces high latency due to PCIe transfers and CPU-side page comparisons, and places the host on the critical path during recovery, preventing the GPU from executing while the host orchestrates recovery.

The paper argues that fault tolerance for these workloads requires a GPU-resident execution context that can observe binary kernels, perform dirty-page detection at HBM bandwidth, and execute recovery logic without host intervention.

2. Methodology: The Concordia Runtime

Concordia is a runtime system designed to interpose on GPU module loading and provide a persistent, device-resident substrate for fault tolerance. Its architecture consists of four core mechanisms:

2.1 Persistent Kernel Executor

The foundation of Concordia is a persistent kernel that runs for the lifetime of an LLM session.

• Operation: It continuously polls a lock-free ring buffer in host-mapped memory.
• Tasks: It executes compute, checkpoint, communication, and recovery tasks without requiring new host-launched CUDA kernels.
• Resource Footprint: It reserves a minimal number of Streaming Multiprocessors (SMs) (e.g., 1 block on RTX PRO 6000, ~0.53% SM footprint), leaving the rest of the GPU available for application kernels.
• Control Path: It acts as a trusted control loop. If the application kernels or communicators fail, the persistent worker remains alive to orchestrate recovery until the device is declared lost.

2.2 PTX/SASS Instrumentation and JIT Compilation

Concordia instruments GPU binaries to expose safe checkpoint points and generate specialized handlers.

• Instrumentation: It intercepts cuModuleLoad calls. For PTX modules, it rewrites the code to insert cooperative pause and checkpoint hooks at entry, exit, and barrier sites. For binary-only (SASS) kernels, it applies conservative patching to redirect execution through trampoline stubs.
• JIT-Compiled Handlers: For every registered memory region (e.g., PagedAttention KV blocks, LoRA adapters, opaque buffers), Concordia JIT-compiles a specialized delta-checkpoint handler.
  • KV Regions: Handlers scan physical block tables and dirty bitmaps provided by the serving runtime.
  • Adapter/Op Regions: Handlers perform dense page scanning or shadow-compare against GPU-resident copies.
• Hot-Swapping: These handlers are compiled via NVRTC and inserted into the persistent kernel's operator table, allowing dynamic updates without service interruption.

2.3 GPU-Side Delta Checkpointing

Instead of full snapshots, Concordia performs delta checkpointing directly on the GPU.

• Process: At kernel or NCCL collective boundaries, the persistent kernel executes the JIT-compiled handler to identify dirty pages at HBM bandwidth (~1.8 TB/s).
• Logging: Only dirty pages and metadata are appended to an Append-Only File (AOF) log stored in CPU-visible CXL memory or host DRAM.
• Structure Exploitation: The system exploits LLM state structure: base weights are static (immutable), while KV caches and adapters change sparsely. This allows the system to avoid logging every update and instead detect changes efficiently.

2.4 Recovery and Fault Tolerance

• Fail-Stop Model: Concordia assumes application kernels or communicators may fail, but the persistent worker is trusted. If the worker heartbeat stops, the GPU is treated as lost.
• Recovery Flow: Upon failure, a replacement GPU (or the same device after reset) loads the last committed base snapshot and replays the AOF log from CXL/DRAM.
• NCCL Integration: An enhanced NCCL wrapper intercepts collective operations to identify safe boundaries. In the event of a failure, it switches to a pre-computed fallback ring or activates a replacement device, reconstructing the communicator DAG without a full NCCL re-initialization.
• Cross-Architecture (Optional): For recovery across different GPU architectures, Concordia can normalize instrumented code into a portable intermediate representation (CTX) and lower it to the target ISA, though this is an optional extension.

3. Key Contributions

1. Persistent-Kernel Substrate: The paper identifies the need for a live device-side executor to enable transparent fault tolerance, avoiding the latency and critical-path dependencies of host-mediated checkpointing.
2. Unified Recovery Contract: Concordia combines PTX/SASS instrumentation, a lock-free persistent executor, JIT-compiled GPU-side delta handlers, and append-only CXL/DRAM logging into a single system.
3. Structure-Aware Delta Checkpointing: The system exploits LLM inference structure (static weights, sparse KV updates) to perform dirty-page detection entirely on the GPU at HBM bandwidth, eliminating the need for application-specific KV logging.
4. Performance Evaluation: The system is evaluated on Blackwell GPUs and heterogeneous targets, demonstrating significant speedups in checkpointing and sub-second recovery times.

4. Experimental Results

The evaluation was conducted on NVIDIA RTX PRO 6000 (Blackwell) and RTX 5090 GPUs, using PyTorch 2.10.0 and NCCL 2.27.5.

• Checkpointing Speed: GPU-side delta checkpointing is 85× to 219× faster than CPU-side transparent page scanning.
  • For a 256 MB region with one dirty page, CPU-side diffing took ~111 ms (dominated by CPU scanning), while GPU-side took ~0.56 ms (dominated by HBM bandwidth).
  • The speedup scales with region size because CPU diffing scales with total region size, whereas GPU diffing scales with dirty data size.
• Trigger Latency: Submitting a checkpoint task via the persistent ring buffer takes <0.1 µs, compared to 7.6 µs for a synchronous cudaLaunchKernel. This eliminates the host launch overhead at every checkpoint boundary.
• Resource Overhead: The persistent kernel occupies only 0.53% of SMs (1 block out of 188), significantly lower than existing persistent thread approaches like CLC (5–10%) or Green Contexts (3–8%).
• Inference Throughput: In Qwen3-0.6B inference, the checkpoint overhead was 18.9 ms per boundary (less than 4% of generation time), with the system sustaining 108.2 tokens/sec.
• Fault Recovery: A two-GPU failure recovery prototype restored service in ~1.5 seconds (10 ms detection + 300 ms isolation + 800 ms state restore + 400 ms reintegration). This contrasts with 47+ seconds required for a standard NCCL restart and model reload.
• LoRA Fine-Tuning: The system effectively handled online adaptation, detecting only the mutable adapter pages (1.75% of total pages) and achieving a 299× speedup over CPU-side delta checkpointing for the full model.

5. Significance and Claims

The paper claims that Concordia provides a necessary substrate for fault-tolerant long-running LLM inference by shifting the recovery control loop from the host to the device.

• Modest Claims on Static Performance: The authors explicitly state that for static, single-model decode benchmarks, carefully engineered CUDA Graphs may still outperform eager execution. Concordia is not presented as a replacement for CUDA Graphs but as a complementary layer that operates below the framework boundaries to handle dynamic workloads and fault tolerance.
• Transparency: The system targets unmodified or partially modified GPU binaries, covering framework-generated kernels and vendor libraries where application-level logging is infeasible.
• Recovery Model: The significance lies in enabling a Redis-style AOF replay model for LLMs, separating durability from scarce HBM capacity and allowing recovery without rebuilding communicators or reloading weights.
• Limitations: The system assumes the persistent worker does not fail independently (Byzantine faults are not handled) and requires cooperative checkpointing at barrier points for cross-architecture migration. It also notes that opaque mutable regions still require GPU-resident shadow copies, consuming HBM.

In summary, Concordia demonstrates that by maintaining a persistent, instrumented execution context on the GPU, it is possible to achieve high-performance, transparent fault tolerance for LLM inference, reducing recovery times from minutes to seconds and eliminating the host CPU from the critical path of checkpointing.