EmbC-Test: How to Speed Up Embedded Software Testing Using LLMs and RAG

This paper introduces EmbC-Test, a Retrieval-Augmented Generation (RAG) pipeline that leverages project-specific artifacts to ground large language models, enabling the automated generation of syntactically correct and runtime-valid embedded C tests that reduce testing time by up to 66% while producing 270 tests per hour.

The Gist

Imagine you are a master chef running a busy restaurant. Your job is to create delicious, safe, and perfectly consistent dishes (software) for your customers. But before any dish hits the table, you have to run a rigorous "taste test" to make sure it's not burnt, under-seasoned, or missing ingredients.

In the world of embedded software (the code that runs inside cars, medical devices, and robots), writing these "taste tests" manually is a nightmare. It's slow, boring, and if you get tired, you might miss a tiny detail that causes a disaster later.

This paper, EmbC-Test, introduces a new way to solve this problem using a "Smart Sous-Chef" powered by Artificial Intelligence. Here is how it works, explained simply:

The Problem: The "Hallucinating" Intern

Imagine you hire a very smart but inexperienced intern (a standard AI) to write your taste tests. You tell them, "Make sure the soup tastes like salt."

• The Issue: Because the intern has never worked in your specific kitchen, they might invent a fake ingredient that doesn't exist, use a spoon when you use a ladle, or forget that your kitchen only has gas stoves. In tech terms, the AI "hallucinates" (makes things up) or uses the wrong tools, creating tests that look good on paper but fail in reality.

The Solution: The "Smart Sous-Chef" (RAG)

The authors built a system called RAG (Retrieval-Augmented Generation). Think of this as giving your AI intern a giant, organized library of your kitchen's rulebook, past recipes, and safety manuals right before they start working.

Instead of guessing, the AI first looks up exactly how your kitchen works:

1. The Library (Knowledge Base): It scans all your old code, your safety rules, and your previous successful tests.
2. The Search (Retrieval): When asked to test a new feature (like a new brake system in a car), the AI doesn't just guess. It quickly finds the specific pages in the library that talk about brakes, how they are named, and how they were tested before.
3. The Generation: With this specific context in hand, the AI writes the test. It knows exactly which tools to use and what the rules are.

The "Smart Search" Technique

The paper describes a clever way of organizing this library. Imagine trying to find a specific sentence in a book.

• Bad Way: Cutting the book into random 10-page chunks. You might cut a sentence in half, making it unreadable.
• Good Way (AST-based): The AI acts like a librarian who understands the structure of the book. It cuts the text at the end of paragraphs or chapters, ensuring that every "recipe" (function) stays whole and makes sense.

The Results: From "Slow Cook" to "Fast Food"

The team tested this system at a real company (Hydac Software) that builds software for heavy machinery. Here is what happened:

• Accuracy: The AI wrote tests that were 100% grammatically correct (no syntax errors) and 85% actually worked when run on the real machine.
• Speed:
  • Human Chef: Takes about 1 hour to write one perfect test.
  • Smart Sous-Chef: Generates 270 tests per hour.
• Time Saved: For a project with 57 requirements, the team went from needing 57 hours of work down to just 19 hours. That is a 66% time saving.

The Best Part: Changing the Job

The most important takeaway isn't just speed; it's about what the humans do.

• Before: Engineers spent all day writing repetitive tests, like a factory worker on an assembly line.
• After: The AI writes the draft tests instantly. The human engineers just act as editors. They review the AI's work, fix the 15% that needs tweaking, and then move on to the hard stuff: figuring out the tricky edge cases and making sure the product is truly safe.

The Bottom Line

This paper proves that you don't need to replace your expert engineers with AI. Instead, you give them a super-powered assistant that knows your company's specific rules inside and out. It turns a slow, manual bottleneck into a fast, automated pipeline, letting humans focus on being creative and critical rather than repetitive.

In short: It's like giving your team a magic wand that instantly writes the boring paperwork, so they can focus on saving the world.

Technical Summary

Here is a detailed technical summary of the paper "EmbC-Test: How to Speed Up Embedded Software Testing Using LLMs and RAG".

1. Problem Statement

The paper addresses the critical bottleneck in embedded software development: the manual creation of automatic unit tests for C-based microcontroller software.

• The Challenge: As software release cycles accelerate, verification efforts struggle to scale. Manual test writing is time-consuming, resource-intensive, and prone to human error.
• Limitations of Naïve LLMs: While Large Language Models (LLMs) offer a potential shortcut, "zero-shot" generation (asking an LLM to write tests without context) is unreliable. It frequently leads to:
  • Hallucinations: Inventing non-existent APIs, functions, or data types.
  • Context Mismatch: Ignoring project-specific coding conventions, internal APIs, and established test frameworks.
  • False Confidence: Generating tests that compile but fail at runtime due to incorrect expectations or implicit dependencies, which is dangerous in safety-critical embedded domains.

2. Methodology: The RAG Pipeline

The authors propose a Retrieval-Augmented Generation (RAG) pipeline tailored for embedded C. Instead of relying solely on an LLM's general training, the system grounds the generation process in the organization's specific engineering artifacts.

System Architecture

The system operates in two phases:

1. Offline Phase (Knowledge Base Construction):

   • Artifacts: The knowledge base includes C header files, source code, legacy Python tests, and project documentation.
   • Chunking Strategies: To ensure relevant context is retrieved, the authors implemented four chunking strategies:
     • Fixed-size: Basic segmentation (risk of splitting functions).
     • Brace-aware: Splits at zero-depth bracket points to keep functions intact.
     • AST-based: Parses code into an Abstract Syntax Tree to group semantically meaningful structures (functions, enums, types). This was found to produce the tightest semantic clusters.
     • Test unit: Splits legacy tests by individual test cases.
   • Embedding: Chunks are vectorized using a locally hosted embedding model and stored in a vector database.

2. Online Phase (Generation):

   • Hybrid Retrieval: When a software requirement is input, the system retrieves relevant context using a Hybrid Retrieval approach:
     • Dense Search: Uses cosine similarity on vector embeddings for semantic matching.
     • Sparse Search (BM25): Uses lexical matching with code-aware tokenization for exact keyword matches.
     • Fusion: Results are merged using Reciprocal Rank Fusion (RRF) to balance semantic and lexical relevance.
   • Prompt Engineering: The retrieved chunks (headers, source snippets, legacy tests) are combined with a system prompt and the specific software requirement to guide the LLM.
   • LLM Integration: The prompt is sent to a cloud-hosted LLM, which generates the test code.

Evaluation Framework

The system was evaluated across five dimensions:

1. Code Coverage: Line and branch coverage.
2. Test Correctness: Three-step validation (Syntax, Compilation/Import, Runtime).
3. Retrieval Quality: Precision@k and Recall@k against ground truth.
4. System Performance: Latency and throughput.
5. Human Evaluation: Domain experts rated tests on relevance, assertion correctness, edge-case completeness, and readability using a 5-point Likert scale.

3. Key Contributions

1. Pragmatic Reference Architecture: A complete RAG pipeline design for embedded C, detailing knowledge base construction, AST-based chunking, hybrid retrieval, and prompt integration.
2. Industrial Validation: Unlike theoretical research, this solution is deployed and actively used at Hydac Software GmbH.
3. Context-Aware Generation: Demonstrates that grounding LLMs in project-specific artifacts (requirements, legacy tests, headers) significantly reduces hallucinations and improves adherence to internal coding standards.
4. Human-in-the-Loop (HITL) Workflow: Proposes a workflow where the AI handles the repetitive drafting of tests, shifting human experts to high-value activities like risk analysis, edge-case refinement, and oracle improvement.

4. Results

The system was tested against baselines (Random Retrieval and No Retrieval) across 44 configurations.

• Correctness:
  • Syntactic Validation: 100% for RAG (vs. 96.8% for No Retrieval).
  • Runtime Validation: 84.5% for RAG (vs. 62.4% for Random and 50.5% for No Retrieval).
• Coverage:
  • Achieved 43% branch coverage and 67% line coverage in a single run without refinement.
  • Note: While lower than manually refined suites (76% branch, 93% line), the manual suites took months to achieve, whereas the RAG results were immediate.
• Human Evaluation:
  • The best configuration scored 4.33/5 for relevance, 4.61/5 for correctness, and 4.83/5 for readability.
  • 94.4% Test Usability Rate: 38.9% were accepted directly, 55.6% required minor modifications, and only 5.6% required rewriting.
• Efficiency & Throughput:
  • Generation Speed: ~270 tests per hour.
  • Manual Baseline: ~1 test per hour.
  • Time Savings: For a library with 57 requirements, testing time was reduced from 57 hours to 19.2 hours, representing a 66% reduction in effort.

5. Significance and Conclusion

The paper demonstrates that scaling test generation does not equate to scaling confidence unless the generation is grounded in specific project context.

• Industrial Impact: The RAG approach transforms the testing workflow from a manual bottleneck to an AI-assisted pipeline. It allows engineers to focus on "Jobs to Be Done" (verifying behavior and ensuring quality) rather than repetitive coding.
• Disruptive Potential: The solution meets the criteria for disruptive innovation by being "good enough" (94.4% usable) while offering massive gains in speed and scalability.
• Future Outlook: The authors suggest that with further refinements (e.g., feedback loops, optimized configurations), this tool can become a standard engine for AI-assisted testing ecosystems, drastically accelerating the embedded development lifecycle.

In summary, EmbC-Test successfully bridges the gap between the theoretical potential of LLMs and the rigorous demands of safety-critical embedded software, proving that RAG is a viable, high-impact solution for automated test generation in industrial settings.