Technical Summary: SmartHomeSecure

Problem Statement

Smart home automation platforms, particularly open-source solutions like Home Assistant, rely heavily on user-authored YAML configuration files to define complex device behaviors. While flexible, these files are highly susceptible to syntax, formatting, and semantic logic errors. Such defects can lead to automation failures, abnormal device behavior, and safety risks. Existing solutions are insufficient for end-to-end diagnosis and repair:

• Static Analysis & Validators: Traditional YAML validators and static analysis tools lack domain-specific understanding of smart home semantics, failing to detect logical errors or conflicts between rules.
• General-Purpose LLMs: While Large Language Models (LLMs) show promise in code generation, they often lack precise context understanding, cannot guarantee correctness, and are prone to "hallucinating" plausible but incorrect code.
• Current Gap: There is a lack of tools that combine domain knowledge with generative AI to provide a trustworthy, automated repair workflow for end-user programmable smart home configurations.

Methodology and System Design

The paper employs Design Science Research (DSR) to develop and evaluate SmartHomeSecure, a prototype system designed to automatically detect and repair Home Assistant configuration errors. The methodology integrates lightweight program analysis with constraint-guided LLM generation.

System Architecture

SmartHomeSecure is implemented as a modular web application with four distinct layers:

1. UI Shell Layer: Handles user interaction, routing, and visual feedback.
2. Feature Orchestrator Layer: Manages the workflow state machine, including model selection, file upload, and the progression from validation to repair.
3. Domain Engine Layer (Core): This layer encapsulates domain-specific knowledge and program analysis techniques:
   • Custom Parser & Validation: Extends standard YAML libraries to handle Home Assistant-specific schemas, detecting syntax errors and domain-specific correctness (e.g., trigger-action structures).
   • Error Normalization: Translates parser exceptions into a standardized format capturing error type, location, and context.
   • Context Extraction: Constructs simplified control flow graphs and generates structural hints (e.g., required fields, valid value ranges) to guide the LLM.
   • Deterministic Auto-Fix: Applies rule-based transformations for routine errors (e.g., indentation, key misspellings) before invoking the LLM.
4. Integration Layer: Interfaces with external LLM providers. It utilizes constraint-guided prompt engineering, embedding the structured error context and structural hints directly into the prompt to constrain the LLM's generative space to plausible repairs.

Workflow

The repair pipeline follows a systematic progression:

1. Ingestion & Validation: The system parses the uploaded YAML file.
2. Deterministic Repair: If errors are found, the system first attempts rule-based auto-fixes.
3. LLM-Guided Repair: If auto-fixes are insufficient, the system constructs a constrained prompt containing the original snippet, normalized error descriptions, and structural hints.
4. Iterative Refinement: The LLM generates a repair, which is re-validated. If validation fails, the system iterates with the previous output and remaining errors until success or exhaustion of attempts.

Experimental Evaluation

The prototype was evaluated on a dataset of 100 real-world Home Assistant YAML files (sourced from the HA_Blueprints repository) with manually injected single errors across five categories: syntax/parsing, indentation, mapping, sequence, and scalar quoting.

Models Tested

Four LLMs were evaluated within the same constraint-guided pipeline:

• gpt-oss-20b (20B parameters)
• gpt-oss-120b (120B parameters)
• llama-3.1-8b (8B parameters)
• llama-3.3-70b (70B parameters)

Key Results

• Error Detection: Three models (gpt-oss-120b, llama-3.1-8b, llama-3.3-70b) achieved 100% error detection accuracy, relying on the system's robust pre-detection module. The gpt-oss-20b model failed to detect errors in 12 of the 100 files (88% accuracy), primarily struggling with longer files (>2,000 lines) where it either returned empty responses or failed to parse the context.
• Repair Success Rates: Among the models with 100% detection:
  • gpt-oss-120b: 93% success rate.
  • llama-3.3-70b: 91% success rate.
  • llama-3.1-8b: 87% success rate.
• Hallucination & Correctness: Manual verification of all successful repairs found zero instances of hallucinated or incorrect fixes. The system produced no repairs that introduced new errors or altered the semantic logic of the original automation.
• Model Efficiency: The framework ran successfully on a consumer-grade MacBook Pro (M4 chip, 16GB RAM), demonstrating practical deployability.

Key Contributions and Significance

The paper claims the following contributions:

1. A Novel Method: It proposes a hybrid approach combining lightweight program analysis with constraint-guided LLM generation. By injecting precise structural and semantic constraints into the prompt engineering process, the system significantly reduces LLM hallucination and improves repair accuracy.
2. A Functional Prototype: SmartHomeSecure realizes a complete, end-to-end workflow for configuration repair, demonstrating the engineering feasibility of integrating deterministic analysis with generative AI.
3. Empirical Evidence: The study provides a controlled evaluation showing that combining domain-aware analysis with generative AI is a feasible approach for improving the reliability of smart home configurations. It highlights that even smaller models (8B parameters) can perform competently when provided with rich structural constraints.
4. Design Knowledge: The research contributes to the understanding of AI-assisted configuration repair, suggesting that exposing structural constraints to LLMs is critical for success in domain-specific tasks.

Limitations and Future Directions

The authors maintain a modest stance regarding the current scope:

• Error Scope: The evaluation was limited to five common error types and single-file configurations. It did not cover multi-file dependencies, temporal logic flaws, or complex cross-automation interference.
• Validation Depth: The current validation is static (syntactic and structural). It does not include dynamic runtime simulation to catch race conditions or device unavailability issues.
• Deployment: The current client-side prototype lacks backend isolation and security features required for production use.

Future work is proposed to extend error coverage to multi-file and temporal logic flaws, integrate simulation-based runtime validation, introduce human-in-the-loop repair modes for ambiguous cases, and explore cross-domain generalization to other configuration languages.