Experimental Analysis of FreeRTOS Dependability through Targeted Fault Injection Campaigns

This paper presents KRONOS, a non-intrusive software-based fault injection framework used to evaluate FreeRTOS dependability under ionizing radiation, revealing that corruption of pointer and scheduler variables frequently causes system crashes while many Task Control Block fields have limited impact on availability.

The Gist

Imagine a busy, high-stakes airport control tower. This tower is run by a sophisticated computer system (the RTOS, or Real-Time Operating System) that manages hundreds of flights (tasks), ensuring they take off and land on time, in the right order, and without crashing into each other.

In the real world, this control tower isn't just sitting in a quiet office; it's often located in space or high-altitude environments where cosmic rays (like invisible, tiny bullets) can hit the computer chips. These hits can flip a single "bit" of data—a 0 becomes a 1, or vice versa. This is called a Single Event Upset (SEU).

The paper you shared is about a team of researchers who wanted to answer a scary question: "If a cosmic ray hits our control tower's brain, how badly does the airport crash?"

Here is a breakdown of their work using simple analogies:

1. The Problem: Invisible Rain

Usually, when we test how strong a building is, we might throw a ball at it. But you can't easily throw a "cosmic ray" at a computer chip in a lab without expensive, specialized equipment.

The researchers realized that by the time a cosmic ray hits a chip and causes a problem, the damage has already traveled up to the computer's "brain" (the software). So, instead of trying to shoot the chip with radiation, they decided to pretend the damage had already happened. They built a tool to manually flip the bits in the software's memory to see what happens next.

2. The Tool: KRONOS (The "Digital Saboteur")

The researchers built a software tool called KRONOS. Think of KRONOS as a ghostly saboteur that can sneak into the control tower's computer while it's running.

• Non-Intrusive: It doesn't break the tower down to fix it; it just whispers lies into the computer's ear.
• Post-Propagation: It assumes the "cosmic ray" has already done its damage and is now sitting in the computer's memory.
• The Mission: KRONOS picks a specific piece of data (like a list of flight times or a pointer to a pilot's name), flips a bit, and then watches to see if the airport keeps running smoothly, slows down, or explodes into chaos.

3. The Experiment: Testing the Weak Links

They ran thousands of tests (a "campaign") on FreeRTOS, a very popular operating system used in everything from pacemakers to satellites. They targeted four main areas of the control tower:

• Global Variables (The Signage): Things like "How many flights are active?" or "What is the current time?"
• Pointers (The Finger Posts): These are like signs pointing to specific locations (e.g., "The list of delayed flights is here"). If you break the sign, the controller looks in the wrong place.
• Lists (The Queues): The actual lines of flights waiting to take off or land.
• TCBs (The Pilot's ID Badge): A file containing all the details of the currently flying plane (its priority, its stack, its name).

4. The Results: What Happened?

When KRONOS started flipping bits, the results were a mix of "oops" and "disaster."

• The "Crash" Zone (70% of the time):
  When KRONOS messed with Pointers (the finger posts) or Critical Scheduler Variables (the main decision-maker), the system almost always crashed.

  • Analogy: Imagine the controller looks at a sign that says "Gate A," but the sign was flipped to say "Gate A" is actually "The Ocean." The plane tries to take off into the ocean. Boom. System crash.
  • Key Finding: The most dangerous thing to break was the Current TCB (the active pilot's ID) or the Scheduler (the person deciding who flies next). If these get corrupted, the whole system stops.

• The "Delay" Zone (3% of the time):
  Sometimes the system didn't crash, but it got confused and took too long to finish its job.

  • Analogy: The controller is still working, but they are stuck in traffic because they are looking for a flight manifest that got lost. The plane lands, but 10 minutes late. This is bad for a satellite, but maybe okay for a toaster.

• The "Silent" Zone (Rare):
  Very rarely, the system finished its job on time but gave the wrong answer (e.g., the plane landed at the wrong airport, but the controller didn't notice). This is called Silent Data Corruption. It's the most dangerous because no one knows anything is wrong until it's too late. The researchers found this happened very rarely in FreeRTOS.

• The "Resilient" Zone:
  Surprisingly, messing with some less important lists (like a list of planes that were already cancelled) didn't cause a crash. The system was robust enough to ignore those specific errors.

5. The Big Takeaway

The researchers discovered that FreeRTOS is like a house of cards.

• If you poke the foundation (the scheduler and pointers), the whole house collapses immediately.
• If you poke the decorations (some minor lists), the house might wobble but stay standing.

They also found that it didn't matter if the damage was a "one-time glitch" (transient) or a "permanent broken sign" (permanent). If the critical data was corrupted, the result was usually the same: Game Over.

Why Does This Matter?

This study is crucial for engineers building safety-critical systems (like self-driving cars or Mars rovers). It tells them:

1. Don't trust everything: You can't just assume the software will handle a cosmic ray hit gracefully.
2. Protect the VIPs: You need to put extra armor (hardware protection or software checks) specifically around the Scheduler and Pointers, because those are the weak points.
3. Better Design: Now that we know exactly where the system breaks, we can design better safety nets to catch those errors before they cause a crash.

In short, the paper is a "stress test" for the brain of our digital world, showing us exactly which neurons need to be protected so our satellites and planes don't fall out of the sky.

Technical Summary

1. Problem Statement

Real-Time Operating Systems (RTOSes) like FreeRTOS are critical for safety-critical domains (aerospace, automotive, industrial control). However, these systems are increasingly vulnerable to ionizing radiation, which causes Single Event Upsets (SEUs) and permanent hardware faults.

• The Gap: While Fault Injection (FI) is a standard method for assessing dependability, existing approaches often focus on the application layer or rely on heavy hardware simulations (e.g., gem5) that are time-consuming and complex.
• The Challenge: There is a lack of systematic, software-based analysis that specifically targets RTOS kernel data structures (schedulers, Task Control Blocks, queues) to understand how faults propagate to the OS-visible state and compromise system dependability. Traditional high-level simulations often miss microarchitectural effects, while hardware-based methods are difficult to scale for kernel-level analysis.

2. Methodology: The KRONOS Framework

The authors introduce KRONOS (Kernel-level FreeRTOS Object-oriented Non-intrusive Open fault injection System), a novel, software-based, non-intrusive FI framework.

• Architecture:
  • KRONOS leverages the FreeRTOS port for Linux/Windows, running the RTOS as a set of user-space processes and threads on a host machine.
  • It operates in a post-propagation mode: it assumes faults have already reached the kernel-accessible memory state, allowing for controlled analysis of OS behavior without modeling low-level radiation physics.
  • Components: It consists of a command module, target module (to locate kernel objects), OS-abstraction layer, injector module, and a logging module that hooks into FreeRTOS events (e.g., task switches).
• Fault Models:
  • Transient Faults: Simulates a one-time bit flip in the kernel memory at a specific runtime instant.
  • Permanent Faults: Simulates a "stuck-at" condition. This is achieved via a compile-time patching mechanism (using PCRE2) that inserts bitwise masks into the FreeRTOS kernel code to force specific bits to remain corrupted across subsequent writes.
• Workflow:
  1. Target Specification: Identifies kernel objects (variables, pointers, lists, TCBs) by name and address.
  2. Golden Run: Establishes a baseline for timing and functional correctness.
  3. Injection: The injector thread (running in the same process address space) wakes at a configured offset, applies the fault, and monitors the scheduler.
  4. Classification: Outcomes are categorized as:
     • Benign: Correct execution within time.
     • Delay: Correct function but missed deadline.
     • Silent Data Corruption (SDC): Incorrect result without error signals.
     • Hang: Deadlock or livelock.
     • Crash: Abrupt termination (exception/abort).
     • Invalid: Target was not valid at the injection time.

3. Experimental Setup

• Targets: The study targeted four categories of FreeRTOS kernel structures:
  1. Global Variables: (e.g., uxCurrentNumberOfTasks, xTickCount, xTimerQueue).
  2. Pointers: (e.g., pxCurrentTCB, pxDelayedTaskList).
  3. Lists: (e.g., Ready task lists, delayed task lists, timer lists).
  4. Current TCB: Fields of the Task Control Block for the running task (e.g., uxPriority, pxStack).
• Workload: Five benchmarks from the TACLeBench suite (SHA, FFT, CUBIC, HUFF_DEC, ADPCM_ENC) were executed with varying priorities to induce preemption and complex scheduling behaviors.
• Scale: The campaign performed 83,916 injections (666 injections per target location) to achieve 99% confidence with a 5% margin of error. Both transient and permanent faults were tested.

4. Key Results

The experimental campaign revealed that FreeRTOS has moderate fault tolerance in its default configuration, with ~70% of runs completing successfully (Benign), but a significant portion resulting in crashes or delays.

• Overall Impact:

  • Crashes: ~20% of injections caused system crashes.
  • Delays: ~3% caused deadline violations.
  • SDC: Very rare (<2%), usually associated with delays.
  • Transient vs. Permanent: The impact was nearly identical for both fault types. Once a critical data structure is corrupted, the system usually fails immediately, making the persistence of the fault less relevant to the immediate outcome.

• Vulnerability by Category:

  • Pointers (Most Critical): Pointer corruption is the most dangerous. Corruption of pxCurrentTCB (pointer to the running task's control block) caused 100% crashes for both fault types. Other pointers (e.g., pxCurrentTimerList, xIdleTaskHandle) caused >50% crashes.
  • Global Variables:
    • Variables like uxDeletedTasksWaitingCleanup, uxTopReadyPriority, and xTimerQueue caused near-total failure rates (crashes).
    • xPendedTicks uniquely produced a high rate of SDC with delays, indicating a specific vulnerability in tick handling.
  • Lists: Corruption of task lists resulted in 20–50% crash rates. Lists related to active operations (e.g., xTasksWaitingTermination) were highly sensitive, while suspended task lists were less so.
  • TCB Fields: Corruption of the current TCB fields generally had lower impact than pointers, but uxPriority was a major exception, causing crashes in >80% of runs. ucDelayAborted also caused >40% crashes.
  • No SDC in Pointers/Lists/TCBs: Notably, corruption in these categories rarely resulted in Silent Data Corruption; the system either crashed, delayed, or ran correctly.

5. Key Contributions

1. KRONOS Framework: Development of a non-intrusive, software-based FI tool that allows repeatable, large-scale kernel-level fault injection without specialized hardware or debug interfaces.
2. Comprehensive Campaign: The first extensive analysis of FreeRTOS covering both transient and permanent fault models across a wide spectrum of kernel data structures (variables, pointers, lists, TCBs).
3. Vulnerability Mapping: Identification of specific "critical" kernel components (e.g., pxCurrentTCB, uxPriority, scheduler queues) that require stronger protection in safety-critical systems.
4. Outcome Characterization: Detailed classification of failure modes, highlighting that pointer corruption leads to immediate crashes, while specific timing variables can lead to Silent Data Corruption.

6. Significance

This work provides RTOS designers and safety engineers with a conditional vulnerability assessment of FreeRTOS. By isolating the OS layer from hardware propagation models, the study clarifies exactly which kernel structures are "single points of failure."

• Practical Implication: It suggests that in safety-critical deployments, specific kernel structures (particularly pointers and priority fields) should be protected via hardware redundancy, software checks, or memory protection units (MPUs), as the default FreeRTOS kernel lacks inherent robustness against these specific corruptions.
• Future Direction: The framework is designed to be portable to other RTOSes and can be extended to correlate OS-visible faults with specific device-level radiation phenomena.