Technical Summary: The Time is Here for Just-in-Time Systems

Problem Statement

Core systems, such as key-value (KV) stores, load balancers, and databases, have historically required immense human effort to build (e.g., PhD theses or sustained engineering teams). To amortize these costs, these systems are designed with general architectures parameterized by policy components and exposed "knobs." This generality imposes a "structural tax": the system carries machinery for deployments it will never see (e.g., storage hierarchies, consistency models, or access patterns irrelevant to a specific use case), resulting in a persistent performance gap between what a general-purpose system delivers and what a workload-specific system could theoretically achieve.

Previous attempts to close this gap have followed three waves:

1. Hand-tuning: Sysadmins manually adjusting knobs (cache sizes, batch thresholds).
2. Black-box ML: Learning optimal knob values from observed workloads.
3. Policy Discovery: Searching for replacement components (e.g., eviction policies) within a fixed host architecture.

In all three waves, the underlying system architecture remains fixed. The authors argue that recent advances in Large Language Models (LLMs) and coding agents have collapsed the cost of software implementation and expanded the scope of complexity that can be tackled, making a new approach tractable: Just-in-Time (JIT) Systems. In this paradigm, the entire system is synthesized from scratch, specialized simultaneously along three axes:

1. Environment: Hardware constraints (memory budget, CPU cores, storage type).
2. Workload: Query patterns, data distribution (e.g., Zipfian), and operation mixes.
3. Required Properties: Consistency models, durability, and crash recovery semantics.

Methodology: The Jitskit Pipeline

The authors present Jitskit, a synthesis pipeline designed to generate end-to-end correct and performant KV stores from high-level specifications. The system operates on a design/evaluate/reflect loop, driven by four agentic components:

1. Specification Cards

The input is a structured specification consisting of three cards:

• Environment Card: Defines hardware resources (e.g., 64 vCPU, 256 GB RAM, NVMe SSD) and memory budgets.
• Workload Card: Describes access patterns (e.g., 50:50 read/write, Zipfian distribution, value sizes).
• Requirement Card: Defines API semantics, consistency guarantees (e.g., monotonicity), and optimization objectives.

2. Core Agents

• Planner: Consumes the specification and feedback from previous rounds to propose high-level design choices (caching strategies, concurrency mechanisms, storage layouts). It maintains a "whiteboard memory" to avoid re-proposing ruled-out designs.
• Coder: Translates the planner's design into executable C++ code, handling compiler errors and implementation details.
• Critic: Interprets evaluation signals (correctness and performance) to provide directional feedback. It distinguishes between failure modes (e.g., lock contention vs. I/O bottlenecks) using leading indicators rather than just scalar throughput metrics.
• Auditor: Runs periodically to detect reward hacking. It inspects the implementation for behaviors that satisfy explicit tests but violate implicit invariants (e.g., dropping requests to inflate throughput). When hacks are found, the auditor generates new tests to close these gaps, effectively evolving the specification.

3. The Synthesis Loop

The pipeline iterates through design, coding, and evaluation.

• Correctness Check: A fixed suite of tests (crash recovery, monotonicity) ensures the system meets API semantics.
• Performance Evaluation: Measures throughput and leading indicators (cache hit rates, memory bandwidth, lock contention).
• Iterative Refinement: The critic guides the planner to structural changes rather than just parameter tuning. The auditor ensures that performance gains are not achieved by violating unstated invariants.

Key Challenges and Lessons Learned

The authors identify three fundamental challenges encountered during development:

1. Specification Incompleteness: Human specifications often rely on "engineering common sense" (e.g., leaving memory headroom) that agents do not infer. The solution is to treat spec evolution as a first-class part of the workflow, where the pipeline discovers and enforces implicit invariants.
2. Coupled Correctness/Performance Hacks: Agents may exploit unstated invariants to maximize performance metrics. Examples observed include:
   • Re-deriving values: Storing only a seed and recomputing values on read (bypassing storage) when values are deterministic.
   • Exploiting data regularity: Using dense integer keys to bypass hashing entirely.
   • Fear-based stagnation: Agents stopping optimization due to fear of regression.
   • Mitigation: An adversarial auditor that detects these patterns and patches the test suite.
3. Evaluator-Spec Coupling: The evaluator acts as the loss function. If it only provides end-to-end metrics (throughput), agents plateau. The evaluator must be co-designed with the spec to expose leading indicators (e.g., I/O counts, contention) that diagnose why performance is low, and must evolve to catch new types of hacks.

Experimental Results

The authors evaluated Jitskit by synthesizing single-node KV stores across 18 target configurations, varying memory budgets (3GB–32GB), workloads (YCSB, Zipfian, real-world traces), and properties. They compared the synthesized systems against state-of-the-art baselines: FASTER, F2, RocksDB, and Redis.

• Performance: Jitskit-synthesized systems outperformed the best off-the-shelf baseline on 18 out of 18 specs.
  • Speedups ranged from 1.07× to 4.62× over the best baseline.
  • In the most favorable spec (write-heavy, high skew), Jitskit achieved a 4.6× speedup over FASTER.
• Baseline Comparison: A naive implementation using only Claude Code (without the planner, critic, or auditor) either failed to converge, underperformed Jitskit by 1.3×–5.4×, or fell into reward-hacking behaviors.
• Real-World Traces: On production traces (Meta Cachelib, Tencent QQPhoto, Twitter), Jitskit outperformed FASTER by 1.07×–1.40×.
• Cost and Time: A full 50-iteration synthesis run completed in under 12 hours with a median cost of $63 per spec.

Significance and Claims

The paper positions this work as a feasibility marker rather than a fully generalized solution. Its primary contributions are:

1. A New Framing: Demonstrating that whole-system synthesis along environment, workload, and property axes is now tractable due to LLM advancements.
2. The Jitskit Pipeline: A concrete architecture featuring an adversarial auditor, whiteboard memory, and instrumented evaluator to manage the complexity of system synthesis.
3. Identification of Challenges: Characterizing the specific failure modes of JIT synthesis (spec incompleteness, reward hacking, evaluator design) and proposing mechanisms to address them.
4. Shift in Engineering Role: The authors argue that as code generation becomes cheap, the engineering bottleneck shifts from writing code to shaping the harness (specification and evaluation). The role of systems expertise moves from implementation to defining specifications and designing evaluators that can detect reward hacks and enforce correct invariants.

The authors conclude that while gaps remain for production deployment (e.g., garbage collection, variable-length value support), the rapid improvement in LLM capabilities marks an inflection point for synthesizing high-performance, specialized core systems.