Pitfalls in VM Implementation on CHERI: Lessons from Porting CRuby

This paper identifies and categorizes the specific pitfalls encountered when porting virtual machines to the CHERI architecture, highlighting how common C language assumptions and VM implementation idioms conflict with CHERI's strict memory safety model, and proposes verified workarounds through a case study of porting CRuby.

The Gist

Here is an explanation of the paper "Pitfalls in VM Implementation on CHERI: Lessons from Porting CRuby," translated into simple language with creative analogies.

The Big Picture: Building a Fortified House

Imagine you have a very old, complex house (a Virtual Machine, or VM) that runs your favorite apps. This house was built decades ago using standard blueprints. It works great, but it has some weak spots where thieves (hackers) can sneak in and steal things or break the walls.

Now, imagine a new, super-secure building code called CHERI.

• Old Way: In the old house, a "key" (a pointer) was just a number telling you which room to go to. If you knew the number, you could go anywhere.
• CHERI Way: In the new house, a "key" is a smart card (a capability). This card doesn't just say "Go to Room 10." It also says, "You can only go to Room 10," "You can only open the door, not break the window," and "This card expires in 5 minutes." If you try to use the card to open a door it wasn't meant for, the lock jams instantly.

The Problem: The authors tried to move their old house (CRuby, the engine behind the Ruby programming language) into this new, super-secure building. They expected it to be a simple renovation. Instead, they found that the old house's construction habits clashed with the new security rules, causing the house to collapse in weird ways.

This paper is their "Lesson Learned" guide for anyone else trying to move their software into this new, secure world.

---

The Six Traps (Pitfalls)

The authors found six specific ways the old software broke the new rules. Here is how they happened, using analogies:

1. The "Too-Small Key" Trap (Invalid Derived Pointer)

• The Old Habit: Imagine a construction worker holding a master key to the whole building. To open a specific drawer, they just make a copy of the key and say, "This copy is for the drawer." In the old world, this copy still worked for the whole building because keys were just numbers.
• The CHERI Rule: The new smart card system is strict. If you make a copy of the key for the drawer, the system automatically shrinks the card so it only opens that drawer.
• The Crash: The worker tries to use that "drawer-only" card to open the front door. The system slams the door shut.
• The Fix: Keep the "Master Key" (a super-capability) handy. When you need to open a new area, derive the new key from the Master Key, not from a tiny, restricted copy.

2. The "Fake Key" Trap (Dereferencing Ambiguous Pointers)

• The Old Habit: The security guard (Garbage Collector) looks at a pile of random numbers on a table. If a number looks like a room number (e.g., 1004), the guard assumes it's a real key and tries to open that door to check if it's empty.
• The CHERI Rule: In the new world, a number is just a number. It doesn't have a "validity stamp." If you try to use a random number as a key, the system screams, "That's not a real card!" and stops you.
• The Crash: The guard tries to open a door with a random number, gets rejected, and the whole system panics.
• The Fix: Before trying to open a door, check the "validity stamp" on the card. If it doesn't have a stamp, it's just a number, not a key. Ignore it.

3. The "Moving Wall" Trap (In-Place Reallocation)

• The Old Habit: You have a storage box. You need more space, so you ask the landlord to expand the box right where it is. The landlord says, "Okay, I expanded it, but I kept your old key." You assume the old key still fits the new, bigger box.
• The CHERI Rule: When the landlord expands the box, they give you a brand new key that fits the new size. The old key is now too small (it only covers the original size).
• The Crash: You try to put a large item into the expanded part of the box using your old, small key. The system blocks you because the key says, "You can't go past this point."
• The Fix: Never assume the key stays the same after an expansion. Always grab the new key the landlord gives you and throw away the old one.

4. The "Hidden Bits" Trap (Using Padding Bits)

• The Old Habit: Imagine a 64-bit integer is a 64-slot parking lot. You use all 64 slots to park cars (store data).
• The CHERI Rule: In the new world, the top 32 slots of that parking lot are actually reserved for security guards (metadata). You can't park cars there; those slots are invisible to you.
• The Crash: The software tries to park a car in the "guard" slots or shift cars around in a way that assumes the whole lot is empty. The system gets confused because it sees data where it expects security guards.
• The Fix: Use a smaller, exact-sized parking lot (like a 64-bit integer type that is guaranteed to have no hidden guards) for your data, so you don't accidentally try to use the security slots.

5. The "Sealed Envelope" Trap (Modifying Temporary Capabilities)

• The Old Habit: You have a sealed envelope (a sealed capability) containing a return address. You want to do some math on the address inside to find a symbol. You rip the envelope open, do the math, and put it back.
• The CHERI Rule: The envelope is sealed with a special wax stamp. If you try to rip it open or change the address inside, the wax breaks, and the system declares the envelope void.
• The Crash: The software tries to do math on a sealed address, breaks the seal, and the system throws an error.
• The Fix: Don't try to do math on the sealed envelope. Copy the address out onto a piece of paper (convert it to a normal integer), do your math on the paper, and leave the envelope alone.

6. The "Wrong Map" Trap (Pointer Arithmetic on Non-Capability Types)

• The Old Habit: You have a map (a pointer). You want to find a location 10 steps away. You convert the map to a coordinate number, add 10, and convert it back to a map.
• The CHERI Rule: If you convert the map to a generic number (like size_t), you lose the "security rules" attached to the map. When you convert it back, you get a map with no rules, which the system rejects.
• The Crash: You try to walk 10 steps using a map that has no boundaries, and the system stops you.
• The Fix: Always use the "Smart Map" type (uintptr_t) for your calculations. It keeps the security rules attached to the number so they survive the math.

---

The Result: Is it Worth It?

The authors managed to fix these issues and get CRuby running on the new CHERI system.

• Success Rate: They passed about 87% of the tests (29,609 out of 34,046). The failures were mostly due to missing libraries or a few remaining bugs, not the core security issues.
• Speed: The new, secure version runs at 98.2% of the speed of the old, insecure version. That's a tiny slowdown for a massive security upgrade.

The Takeaway

Moving software to a secure, capability-based system like CHERI is like moving a family from a regular house to a high-tech fortress. You can't just carry the furniture over; you have to re-pack everything because the locks, keys, and rules are different.

The main lesson? Don't trust the old habits.

• Don't assume a number is a key.
• Don't assume a key works for the whole building.
• Don't assume you can shrink or expand things without getting a new key.

By following these new rules, we can build software that is much harder for hackers to break into, without slowing it down too much.

Technical Summary

Here is a detailed technical summary of the paper "Pitfalls in VM Implementation on CHERI: Lessons from Porting CRuby".

1. Problem Statement

CHERI (Capability Hardware Enhanced RISC Instructions) is a hardware architecture designed to enforce memory safety by replacing traditional pointers with hardware-enforced capabilities (fat pointers containing address, bounds, permissions, and a validity tag). While CHERI offers robust protection against memory vulnerabilities, porting complex software systems like Virtual Machines (VMs) to CHERI is non-trivial.

The core problem identified is that VMs, often written in C, rely heavily on undefined behaviors (UB) and implementation-specific assumptions of traditional architectures. These assumptions conflict with CHERI's strict memory safety model. Specifically:

• Undefined Behaviors: C standards allow certain behaviors (e.g., casting integers to pointers, pointer arithmetic on non-capability types) that are implementation-defined or undefined. VMs exploit these for performance and efficiency.
• Capability Constraints: CHERI enforces strict bounds and validity tags. Operations that are safe on conventional hardware (e.g., deriving a pointer from a local variable for stack scanning, or using bit-packing in integers) cause hardware faults (bounds faults, tag faults, or sealed faults) on CHERI.
• Lack of Specific Guidance: While general CHERI programming guides exist, they do not address the specific, subtle pitfalls inherent to VM runtime systems (e.g., conservative garbage collection, interpreter loops, and object layout).

2. Methodology

The authors employed a case study approach combined with a survey of existing literature to identify and categorize these pitfalls.

• Case Study (CRuby Port): The authors ported CRuby (the reference implementation of the Ruby interpreter, version 3.4.1) to the CHERI-RISC-V architecture (purecap mode).
  • Scope: They focused on the runtime system and interpreter, excluding the Just-In-Time (JIT) compiler and Foreign Function Interface (FFI).
  • Process: They compiled the source code, fixed compilation errors, and resolved test failures. They modified approximately 464 lines of code across 40 files (0.077% of the codebase).
  • Outcome: The port passed 29,609 out of 34,046 test cases. Failures were attributed to missing libraries (libyaml) or a single crashing test program, with 688 cases remaining unexplained.
• Comparative Survey: The authors analyzed three prior porting efforts to CHERI:
  1. MicroPython: A lightweight Python interpreter.
  2. JavaScriptCore: A production-level JavaScript VM.
  3. Rust: The Rust compiler and runtime.
• Categorization: Issues found in CRuby and the surveyed works were categorized based on their root causes (e.g., invalid derived pointers, ambiguous pointer dereferencing).

3. Key Contributions: Categorized Pitfalls and Workarounds

The paper identifies six specific categories of pitfalls unique to VM implementation on CHERI, providing concrete examples from CRuby and proposed workarounds.

A. Invalid Derived Pointers

• Issue: VMs often derive a new pointer from an existing one for a different purpose (e.g., taking the address of a local stack variable to determine the stack top for GC). On CHERI, the & operator returns a capability with bounds limited only to that specific variable. Deriving a pointer for a larger region (like the whole stack) from this narrow capability causes a bounds fault.
• Workaround: Maintain a "super capability" with bounds covering the entire region (e.g., the whole stack) and derive new pointers from that, rather than from narrow local variables.
• Impact: Requires careful management of wide-bounds capabilities, potentially increasing the attack surface if not compartmentalized.

B. Dereferencing Ambiguous Pointers

• Issue: Conservative Garbage Collectors (GC) scan the stack for values that look like pointers (bit-pattern matching) and attempt to dereference them. On CHERI, converting an integer to a pointer does not set the validity tag. If the GC dereferences a "pointer-like" integer, it triggers a tag fault.
• Workaround: Instead of bit-pattern matching, check the validity tag of the capability before dereferencing.
• Impact: Improves performance by avoiding the marking of dead objects that happen to look like pointers. It also enables the implementation of copying GCs without object pinning.

C. In-Place Reallocation

• Issue: Custom allocators in VMs often assume that if realloc succeeds in-place, the original pointer remains valid. In CHERI, realloc may return a new capability with different (wider) bounds. Using the old pointer (which has narrow bounds) to access the newly expanded memory region causes a bounds fault.
• Workaround: Always update all references to the pointer when reallocation occurs. Do not rely on in-place reallocation guarantees.
• Impact: Requires more complex tracking of pointer locations or avoiding in-place reallocation entirely, which may incur performance overhead.

D. Using Padding Bits of Integer Types

• Issue: VMs often pack data into integers (e.g., bitmaps for GC, embedding object IDs in headers). In CHERI, (u)intptr_t is a capability where the upper 64 bits are metadata (padding for the integer value). Shifting or bitwise operations on these types can yield random values or undefined behavior if they touch the metadata bits.
• Workaround: Use exact-width integer types (e.g., uint64_t) for bit manipulation and packing, reserving (u)intptr_t only for actual pointers.
• Impact: No performance overhead; simply requires stricter type usage.

E. Modifying Temporary Capabilities Introduced by the Compiler

• Issue: CHERI compilers may generate code that creates temporary capabilities during binary operations (e.g., arithmetic on (u)intptr_t). If the operand is a sealed capability (e.g., a return address), attempting to modify its address part (even temporarily) causes a sealed fault.
• Workaround: Cast sealed capabilities to non-capability integer types (e.g., uint64_t) before performing arithmetic or bitwise operations, then cast back if necessary (though usually, the result is just an integer).
• Impact: Eliminates unnecessary temporary capability creation, potentially improving performance.

F. Pointer Arithmetic on Non-Capability Types

• Issue: VMs frequently cast pointers to generic integer types (like size_t) for arithmetic (e.g., calculating offsets for downcasting in inheritance hierarchies) and cast back. On CHERI, casting to a non-capability integer type loses the capability metadata. Casting back results in an invalid capability.
• Workaround: Use (u)intptr_t for all pointer arithmetic to preserve metadata.
• Impact: Requires code refactoring; introduces instructions to manipulate capability metadata.

4. Results and Performance Evaluation

• Functionality: The ported CRuby successfully passed the majority of its test suite (approx. 87% of total cases), demonstrating that VMs can be ported to CHERI with manageable code changes.
• Performance: The authors benchmarked the ported CRuby against the original x86 version.
  • Overall Throughput: The ported VM achieved 0.982x the throughput of the original, with a standard deviation of 0.038x.
  • Outliers: Fiber-related benchmarks showed significantly lower performance (up to 10x slower). This was attributed to the port using a general implementation relying on standard library functions, whereas the original used highly optimized x86 assembly.
  • Conclusion: The workarounds for the identified pitfalls introduce negligible performance overhead for the core VM operations.

5. Significance

• Practical Guidance: This paper provides the first comprehensive catalog of VM-specific pitfalls when porting to CHERI, filling a gap left by general CHERI guides.
• Security vs. Portability: It highlights the tension between C's undefined behaviors (which enable high-performance VMs) and CHERI's strict safety. It proves that VMs can be made safe without sacrificing significant performance, provided developers adhere to strict capability rules.
• Future Directions: The work suggests that while CHERI requires changes to runtime systems, it also enables new optimizations (e.g., precise GC without pinning) and offers a path toward constructing fundamentally safer language runtimes.
• Broader Impact: The lessons learned are applicable to any complex C-based system porting to capability-based architectures, aiding the broader adoption of memory-safe hardware.