💻 computer science

A Trace-based Approach for Code Safety Analysis

This paper presents a systematic framework for analyzing unsafe code and undefined behavior in Rust by reviewing its safety design and real-world projects, ultimately establishing soundness criteria and actionable guidance for achieving sound encapsulation.

Original authors: Hui Xu

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hui Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, high-speed train system. In most programming languages, the engineers (programmers) have to be incredibly careful because if they make a tiny mistake, the train could derail, the tracks could melt, or the passengers could vanish into thin air. This is called "undefined behavior" in the computer world—it's chaos.

Rust is a special language designed so that, by default, the train tracks are reinforced, the signals are automated, and derailments are mathematically impossible. It's incredibly safe.

However, sometimes you need to do something the automated system can't handle. Maybe you need to manually override a signal or weld a track in a way the computer doesn't understand. In Rust, this is called "Unsafe Code." It's like handing a human engineer a welding torch and saying, "You can do this, but you are now personally responsible for not blowing up the station."

The paper you shared is essentially a rulebook and a safety manual for those engineers holding the welding torch. Here is the breakdown using simple analogies:

1. The Core Idea: The "Taint" Analogy

The authors propose a way to think about safety called a "Trace-based Approach."

Imagine "Undefined Behavior" (the explosion) is a toxic gas.

Safe Code is a clean room. No gas exists here.
Unsafe Code is a factory that produces the gas.

The main rule of the paper is: The gas can only come from the factory. If you see gas in the clean room, it means someone from the factory leaked it in.

The paper says: "Don't panic about the whole system. Just look at the factory (the unsafe code). If the factory follows its own safety rules (contracts), no gas will leak out. If gas does leak, it's because the factory broke its own rules."

2. The "Safety Contract" (The Rulebook)

Every time an engineer uses the welding torch (Unsafe Code), they must write a Safety Contract.

Example: "I promise not to weld while the train is moving," or "I promise to wear a helmet."
The Rule: If the engineer follows their contract, the train stays safe. If they ignore the contract, the train might crash.

The paper argues that we don't need to check every single bolt in the entire train system. We just need to check:

Did the engineer write a contract?
Did they follow it?

If yes, the rest of the system (the "Safe Code") is automatically safe.

3. The "Clean Room" vs. The "Factory" (Encapsulation)

The paper introduces a concept called Encapsulation. Think of a Rust function as a sealed box.

Inside the box: You might have the welding torch (unsafe code).
Outside the box: People see a clean, safe machine.

The Golden Rule: As long as the person inside the box follows their contract, the box remains sealed. The toxic gas never escapes to the outside world.

If you are a "Safe Function" (a clean box), you can use "Unsafe Functions" (other factories) inside you, but only if you make sure they follow their contracts.
If you are an "Unsafe Function" (a factory), you must ensure that your internal factories also follow their contracts.

It's like a chain of command: The CEO (Safe Function) tells the Manager (Unsafe Function), "You can do the dangerous work, but you must make sure your workers follow the rules." If the Manager does that, the CEO is safe.

4. The "Lego Castle" (Structs)

So far, we talked about single functions (boxes). But what about Structs? A Struct is like a Lego Castle made of many bricks (data) and many workers (functions) who build or fix it.

The problem with a castle is that the workers are interdependent. If Worker A breaks a wall, Worker B might fall off the roof later. You can't check them one by one; you have to check the whole castle.

The Solution: The "Safety Invariant"
The paper suggests giving the castle a Blueprint (called a Safety Invariant).

The Blueprint says: "At all times, the castle must have a roof, and the walls must be standing."
The Rule: Every worker (function) who touches the castle must promise: "If I start with a castle that has a roof, I will finish with a castle that still has a roof."

If a worker cannot guarantee this (e.g., they need to remove the roof to fix the foundation), they must wear the "Unsafe Vest" and declare it. But as long as they wear the vest and follow the rules, the castle remains valid.

5. The Big Picture: Why This Matters

Currently, finding bugs in Rust's "unsafe" code is like looking for a needle in a haystack. It's hard and scary.

This paper provides a map. It tells developers:

Identify the source: Only look at the "Unsafe" parts.
Check the contract: Did they promise to be safe?
Check the chain: Did they make sure the people they hired also promised to be safe?

If everyone follows this "Trace-based" map, we can build complex, high-performance software (like the Linux kernel or web browsers) with the confidence that the "toxic gas" of crashes and security holes will never leak out.

Summary in One Sentence

This paper gives us a simple way to verify that even when programmers use dangerous tools, they are strictly following a set of written rules that guarantee the rest of the software remains perfectly safe.

1. Problem Statement

While Rust is renowned for its memory safety guarantees, the language relies on unsafe blocks to perform low-level operations (e.g., raw pointer manipulation) that the compiler cannot verify. The core challenge addressed in this paper is:

The "Unsafe" Gap: Although safe code cannot cause undefined behavior (UB) by definition, the presence of unsafe code introduces a risk where UB can occur if the programmer fails to uphold specific safety contracts.
Lack of Systematic Framework: There is a need for a rigorous, systematic framework to analyze how UB propagates from unsafe code through safe abstractions, particularly in complex structures like structs and modules where internal state is shared and interdependent.
Encapsulation Verification: Developers need clear criteria to determine when a function or struct is "sound" (i.e., guarantees no UB) and how to encapsulate unsafe operations so that the rest of the system remains safe.

2. Methodology

The paper proposes a Trace-based Approach grounded in formal logic and trace analysis. The methodology involves:

Formal Definitions: Establishing precise definitions for "Safe Code," "Unsafe Code," "Safety Contracts," and "Undefined Behavior" within the Rust type system.
Trace Analysis Analogy: The authors draw an analogy to taint analysis. In this model:
- Unsafe code is the taint source.
- Function entrances/exits are the sinks.
- The goal is to ensure that the "risk" (potential for UB) introduced by unsafe code does not cross encapsulation boundaries unless the safety contract of the boundary is satisfied.
Safety Invariants for Structs: To handle the complexity of structs (where methods share internal state), the paper introduces Safety Invariants ($SI$). These are properties that all valid instances of a struct must maintain. If a method can produce an instance violating this invariant, it must be marked unsafe.
Modular Reasoning: The approach decouples the soundness of individual components (functions, structs, modules) by assuming that if a component satisfies its contract/invariant, it does not introduce UB.

3. Key Contributions

A. The Main Theorem

The paper establishes a fundamental theorem characterizing the relationship between UB and unsafe code:

Theorem 1: In a well-typed Rust program $P$ , undefined behavior occurs if and only if $P$ contains unsafe code ($UC$) and $P$ violates the safety contract ( $SC_{UC}$ ) associated with that code.
Formally: $P \Downarrow UB \implies P \supseteq UC \land P \not\models SC_{UC}$ .

This theorem asserts that UB is exclusively determined by the safety contracts of the unsafe code involved.

B. Soundness Criteria for Functions

Based on the Main Theorem, the paper derives encapsulation rules for functions:

Safe Functions: A safe function is sound if it contains no unsafe code, OR if it upholds the safety contracts of all unsafe functions it calls.
Unsafe Functions: An unsafe function is sound if it upholds the safety contracts of all unsafe functions it calls internally.
Unified Rule: A function (safe or unsafe) is sound if the safety constraints of all its unsafe callees are satisfied.

C. Soundness Criteria for Structs

The paper extends the analysis to structs, introducing a more complex set of criteria due to shared state:

Safety Invariant ($SI$): A struct must define an invariant that valid instances must satisfy.
Constructor Soundness: Constructors must ensure that any instance they create satisfies the struct's $SI$.
Method Soundness: Methods must ensure that if the input instance satisfies $SI$ and the method's contract holds, the output instance also satisfies $SI$.
Encapsulation Rules:
- If a struct has a non-empty $SI$, its literal constructor must be hidden (private) to prevent external creation of invalid instances.
- Field projections must preserve the $SI$.
- Methods must preserve the $SI$ and uphold contracts of internal unsafe calls.

D. Module and Crate Soundness

The paper defines module soundness as the conjunction of the soundness of all public free-standing functions and public structs within the module. It argues that static mutable variables can be treated as part of the safety contracts of the functions accessing them, allowing modular verification without analyzing global interleavings.

4. Results and Findings

Formal Verification Framework: The paper successfully formalizes the conditions under which Rust code is sound, moving beyond empirical observation to logical derivation.
Decoupling Complexity: By introducing Safety Invariants for structs, the paper provides a mechanism to reason about complex, stateful types without needing to analyze every possible execution path of every method in isolation.
Actionable Guidance: The derived corollaries provide a checklist for developers:
1. Identify all unsafe callees.
2. Define and document safety contracts for unsafe functions.
3. Define safety invariants for structs.
4. Verify that public APIs (constructors/methods) preserve these invariants and contracts.

5. Significance

Bridging Theory and Practice: The work bridges the gap between theoretical soundness proofs and practical Rust development. It translates the abstract concept of "soundness" into concrete engineering practices (e.g., "If your struct has an invariant, hide the literal constructor").
Standardization for "Unsafe" Rust: As Rust adoption grows in critical systems (e.g., Linux kernel, embedded), the ability to rigorously audit unsafe code is vital. This framework provides a standardized language for discussing and verifying safety contracts.
Tooling Potential: The trace-based approach and formal definitions serve as a foundation for developing automated static analysis tools that can verify soundness and detect violations of safety contracts before code is deployed.
Root Cause Analysis: The framework aids in debugging UB by pinpointing exactly which component (function or struct) failed to uphold its contract, rather than treating UB as a monolithic failure of the program.

In conclusion, Hui Xu's paper provides a rigorous, trace-based theoretical framework that defines exactly how undefined behavior arises in Rust and offers a systematic method for developers to encapsulate unsafe code safely, ensuring the integrity of the entire software stack.