Shift schema drift left: policy-aware compile-time contracts for typed JVM and Spark pipelines

This paper introduces a Scala 3 framework that bridges the gap between compile-time type safety and runtime schema enforcement by proving structural compatibility under explicit policies, deriving Spark schemas from contracts, and performing policy-aware runtime checks at the sink to detect schema drift before data is written.

The Gist

Imagine you are running a massive, automated factory that processes raw materials (data) into finished products. In this factory, you have a Blueprint (the code you write) and a Delivery Truck (the actual data arriving from the outside world).

The problem this paper solves is called "Schema Drift."

The Problem: The "Surprise" Breakage

Imagine your factory blueprint says: "We need a box containing 3 red apples."
But one day, the delivery truck arrives with a box containing 4 red apples, or maybe 3 red apples and 1 green one, or the apples are arranged in a different order.

In many data systems, the factory doesn't realize anything is wrong until the machine tries to pack the apples and explodes. This happens late, often after the data has already been processed, causing costly errors and downtime.

• Old Way 1 (Typed Wrappers): Some factories try to replace every single machine with a new, super-smart version that checks the apples. This is expensive and hard to do (like rebuilding the whole factory).
• Old Way 2 (Table Enforcement): Others wait until the truck parks at the loading dock to check the apples. This is better, but if the truck driver made a mistake, the damage is already done by the time you check.

The Solution: "Shift Left" with a Smart Inspector

This paper introduces a two-step safety system that catches mistakes before they can cause a disaster. It's like having a Smart Inspector who works in two shifts:

1. The Morning Shift: The "Blueprint Check" (Compile-Time)

Before the factory even opens for the day, a super-smart robot (the Scala 3 Macro) looks at your Blueprint (your code).

• It asks: "Does the blueprint for the 'Apple Box' match the 'Delivery Contract' we agreed on?"
• If you wrote code that expects 3 apples but the contract says 4, the robot refuses to let the factory open. It stops you immediately with a clear note: "Hey, you forgot to update the blueprint!"
• The Magic: It does this using a set of Rules (Policies).
  • Strict Rule: "Must be exactly 3 red apples, no more, no less."
  • Flexible Rule: "We can handle 3 or 4 apples, as long as they are red."
  • Backward Rule: "If the truck brings extra apples, we can handle it. But if it's missing a required apple, we stop."

If the blueprint doesn't match the rules, the job never starts. This saves you from wasting time building a broken machine.

2. The Evening Shift: The "Truck Check" (Runtime)

Even if your blueprint is perfect, the actual truck might still be lying. Maybe the driver swapped the apples for oranges at the last minute.

• Just before the machine loads the apples, the Smart Inspector looks at the actual truck (the real data).
• It compares the real apples against the Contract (derived from your blueprint).
• The Special Trick: Most inspectors only check the top-level box. This one checks a specific detail deep inside: whether each slot in a nested list or map can be empty or not. For example, if the contract says 'a box of apples where every slot must be filled' but the truck sends 'a box of apples where some slots are empty,' this inspector catches it. Standard tools often miss these deep optionality mismatches inside lists and maps.

The "Policy" Concept: Setting Your Rules

The paper introduces a flexible way to set the rules of engagement, called Policies. Think of these as different types of contracts you sign with your suppliers:

• Exact Match: "I want exactly what I ordered. No substitutions."
• Backward Compatible: "I can handle extra stuff you send me, but don't take away what I need." (Great for when you upgrade your system).
• Forward Compatible: "I can handle missing stuff, as long as I don't get anything weird." (Great for when your supplier upgrades).

Why This Matters (The Analogy)

Think of it like building a house:

• Compile-Time Check: The architect checks the blueprints against the building code before you pour a single drop of concrete. If the blueprints say "2 bedrooms" but the code requires "3," the city stops you immediately.
• Runtime Check: Before you move in, the inspector checks the actual house. Even if the blueprints were perfect, maybe the contractor accidentally swapped a window for a door. The inspector catches this right before you sign the keys.

The Bottom Line

This paper presents a small, honest tool that:

1. Catches code errors early (before you even run the job).
2. Catches data errors late (right before you save the data).
3. Doesn't require rebuilding your whole factory (unlike other complex solutions).

It doesn't promise to solve every business problem (like "are these apples fresh?"), but it solves the structural problem of "does the box fit the machine?" It shifts the safety net from the end of the line to the very beginning, saving time, money, and headaches.

Technical Summary

1. Problem Statement

Schema drift in data pipelines is frequently detected too late—often only when a job processes real data at runtime. Existing solutions suffer from specific gaps:

• Typed-Dataset Libraries (e.g., frameless, iskra): Require wholesale adoption of typed wrappers across the entire job, creating high adoption costs.
• Table-Level Enforcement (e.g., Delta Lake, Iceberg): Operate at write time against stored schemas, meaning checks occur after the job is submitted and code is built.
• Runtime Comparators: Spark's built-in schema comparators run after execution begins and often miss specific drift types, such as nested collection optionality (e.g., changes in nullability within arrays or maps).

The core problem is the lack of a mechanism that provides compile-time proof of structural compatibility between a producer and a contract, while simultaneously enforcing a runtime check at the data boundary to catch external schema drift.

2. Methodology

The paper presents a Scala 3 framework that fuses compile-time verification with policy-aware runtime enforcement. The methodology relies on three main pillars:

A. Structural Contract Model

The framework defines a "structural data contract" as a typed description of record shape (field names, nesting, collection types, optionality, defaults) paired with a comparison policy. It explicitly excludes business semantics (e.g., valid ranges, temporal rules) to maintain a direct mapping between Scala types and Spark StructType.

B. Policy Family

The system supports a flexible family of policies to define what constitutes "compatibility":

• Exact Policies: Exact, ExactUnorderedCI, ExactOrdered, ExactOrderedCI, ExactByPosition. These enforce strict matching based on name, order, and case sensitivity.
• Subset Policies:
  • Backward: Producer can have extra fields; missing contract fields are allowed only if optional or defaulted.
  • Forward: Producer fields must exist in the contract; the contract can have extra fields.
• Full: Accepts all structural combinations (used for disabling enforcement).

C. The "Seam" Architecture

The framework operates at the "seam" between code and data:

1. Compile-Time Path: Uses Scala 3 Macros and Quoted Reflection to derive a normalized structural model (TypeShape) from Scala case classes. It generates a compile-time witness (SchemaConforms[Out, Contract, P]). If the producer type does not conform to the contract under the specified policy, compilation fails with rich diagnostic paths.
2. Runtime Path: Derives a Spark StructType from the same contract type used at compile time. Before writing to a sink, it performs a policy-aware runtime comparison.
   • Key Innovation: The runtime comparator explicitly checks nested collection optionality (a gap in Spark's native comparators) and enforces the subset semantics (Backward/Forward) defined in the policy.
3. Typed Builder Fuse: A builder API (addSink[R, P]) requires the compile-time witness to be present. If the code compiles, the pipeline is built, but the runtime check is still executed immediately before the write operation to validate the actual incoming DataFrame schema.

3. Key Contributions

1. Scala 3 Macro Derivation: A mechanism to generate compile-time evidence of structural conformance under explicit policies, providing path-rich drift diagnostics (e.g., identifying exactly which nested field is missing or has the wrong optionality).
2. Policy-Aware Runtime Comparator: A custom runtime layer that extends Spark's comparators to support:
   • Subset semantics (Backward/Forward compatibility).
   • Explicit checks for nested collection optionality (e.g., Array[Option[T]] vs Array[T]).
3. Unified Artifact: A reproducible framework that ties the compile-time witness to the runtime sink boundary, ensuring that the declared contract type drives both the proof and the enforcement.
4. Honest Evaluation: The paper explicitly frames itself as a "mechanism artifact" rather than a production-effectiveness study, focusing on proving the structural properties and measuring overhead rather than claiming broad industrial productivity gains.

4. Results and Evaluation

The evaluation covers compile-time proofs, runtime policy tests, end-to-end builder tests, and benchmarks on two environments (local macOS and GitHub-hosted Ubuntu).

• Correctness: The framework successfully rejects incompatible producer-contract pairs at compile time (e.g., reordered fields, missing required fields, nested optionality drift) and accepts valid ones.
• Runtime Drift Detection: The runtime layer successfully catches drift that Scala types cannot rule out, such as external CSV/JSON inputs violating nested optionality rules or subset policy violations.
• Performance Overhead:
  • Compile-Time: Adding schema pairs increases compilation time by 0.27s to 1.88s (depending on schema complexity and environment). This is considered acceptable for the safety gained.
  • Runtime: The custom comparator is roughly 17–25× slower than Spark's built-in equalsIgnoreCaseAndNullability baseline. However, the absolute cost remains in the microsecond range (e.g., ~4.7μs for unordered exact matching on local, ~8.1μs on Ubuntu) because the check runs once per sink write, not per row.

5. Significance

This work is significant because it addresses a specific "seam" in data engineering that is left open between typed-Dataset layers and table-level schema enforcement:

• Shift Left: It moves structural drift detection from runtime failure to compile-time proof, catching errors before code is even submitted to a cluster.
• Potentially Lower Adoption Cost: By focusing the contract enforcement at the sink boundary (where the declared Scala producer type meets the contract type), the author argues this design may lower adoption cost compared to a full typed-Dataset rewrite. This is a design argument; the paper does not measure adoption cost empirically.
• Complementary Safety: It demonstrates that compile-time proofs and runtime checks are not redundant but complementary. The compile-time check validates the code structure, while the runtime check validates the actual data boundary against external inputs.
• Honest Scope: By limiting the scope to structural contracts and explicitly acknowledging limitations (no business logic, no schema registry integration), the paper provides a credible, reproducible foundation for future work in more complex contract systems.

In summary, the paper proves that a focused combination of Scala 3 macros and custom runtime comparators can effectively reduce schema drift risks in Spark pipelines with manageable performance overhead, bridging the gap between static type safety and dynamic data reality.