Technical Summary: PLCBench

Problem Statement

Industrial Control Systems (ICS) rely on Programmable Logic Controllers (PLCs) to bridge networked computation and physical control. While tool-using Large Language Model (LLM) agents have demonstrated increasing capability in digital cybersecurity tasks (e.g., penetration testing, vulnerability exploitation), their ability to translate network reachability into sustained physical impact remains unquantified.

Existing evaluations often stop at intermediate digital milestones, such as finding an open service, achieving a valid software write, or gaining tool access. In the context of ICS, these milestones are insufficient indicators of physical risk. A valid PLC write may be irrelevant to the control loop, overwritten by existing logic, or fail to sustain a hazardous physical condition. There is a lack of a comprehensive, end-to-end evaluation framework that assesses whether an autonomous agent can:

1. Interact with heterogeneous, real PLCs via vendor-native interfaces.
2. Adapt behavior based on closed-loop process feedback.
3. Achieve a physically realized objective that is independently verified.

Methodology: PLCBench Framework

The authors present PLCBench, the first real-PLC hardware-in-the-loop (HIL) framework designed to characterize cyber-to-physical capabilities and their boundaries. The framework is modular, allowing the recombination of LLM backends, commercial PLCs, and process workloads without altering the core evaluation loop.

Core Components

1. Agent Framework: A long-horizon interaction loop based on the ReAct (Reasoning and Acting) paradigm.
   • Prompt Contract: Uses fixed system prompts and task descriptions that hide target-specific details (e.g., native object maps, active protocols).
   • Audited Tools: Provides shell, Python, and public protocol libraries. The agent must configure clients and issue native requests without pre-configured wrappers.
   • Context Management: Implements deterministic compaction of interaction history to handle long episodes without losing temporal order or critical evidence, preserving a full raw transcript for evaluation.
2. HIL Platform:
   • Real PLCs: Four commercial PLCs (Siemens S7-300, Schneider M241, Beckhoff CX2030, Mitsubishi R08CPU) running vendor-native protocols (S7comm, Modbus/TCP, ADS, MC/SLMP).
   • Closed-Loop Workloads: Four distinct process simulations (e.g., quadruple-tank, thermal mixing) that provide sensor feedback and actuator control.
   • Isolation: Agents operate in isolated sandboxes; the HIL bridge manages the state exchange between the process server and the PLC.
3. Deterministic Evaluator:
   • Operates outside the agent's context.
   • Analyzes independent evidence sources (runner logs, packet captures, object audits, process traces).
   • Assigns six hidden diagnostic flags to categorize progress:
     • PLC-Interface Acquisition: discover (service found), read (valid data returned), write (accepted write).
     • Physical-Control Progression: manipulate (write to process-linked object), disrupt (warning condition sustained), impact (full task objective sustained).

Experimental Setup

• Models: Five LLM families (GPT 5.5, Sonnet 5, Gemini 3.5 Flash, DeepSeek V4 Pro, Kimi K2.7).
• Configuration: A crossed design of 4 PLCs × 4 Workloads × 5 Models × 3 Repetitions = 240 episodes.
• Constraints: 100-action budget, 3600-second time limit, no controller management actions (e.g., rebooting), and no modification of the PLC program.

Key Results

Overall Impact

• Success Rate: Across 240 episodes, 75 (31.3%) achieved sustained physical impact.
• Model Performance: GPT 5.5 was the most capable, reaching impact in 38 of 48 episodes (79.2%) and succeeding in all 16 PLC-workload configurations. Other models showed significantly lower success rates and coverage.
• Repeatability: While GPT 5.5 succeeded in all 16 cells, it only achieved success in all three repetitions for 9 of those cells, indicating that success is not uniformly repeatable even for the strongest model.

Barrier Analysis

The evaluation identified two distinct barriers where agents frequently fail:

1. Barrier I: Native-Interface Acquisition (98 episodes stopped here)

   • Agents struggled to transition from network reachability to a usable vendor-native interface.
   • Protocol Complexity: Significant drop-offs occurred with less common protocols. For example, on the Beckhoff (ADS) and Mitsubishi (MC/SLMP) PLCs, many agents could discover the service but failed to obtain a valid read.
   • Finding: Interface acquisition is highly dependent on protocol familiarity and client configuration, acting as a friction point rather than a hard security boundary.

2. Barrier II: Physical Conversion (62 episodes stopped here)

   • Agents successfully wrote to process-linked objects (manipulate) but failed to sustain the hazardous state (impact).
   • Process Dynamics: Failures were often due to the complexity of closed-loop control and protection logic. For instance, in the quadruple-tank scenario, agents could not maintain the specific tank level constraints against coupled dynamics.
   • Observability: Providing richer process observations (intermediate variables, control-loop state) increased the conditional attainment of impact after a successful write from 44.2% to 64.0%, suggesting that limited observability is a significant bottleneck.

Ablation Studies

• Shared Protocol: When all workloads were exposed via a single shared Modbus/TCP path (removing protocol heterogeneity), manipulation attainment rose to 100% (vs. 57.1% on heterogeneous native paths), and raw impact rose to 50%.
• Observation Depth: Richer observation conditions did not improve interface acquisition but significantly improved the conversion of writes into sustained physical impact.

Significance and Claims

The paper claims to provide the first systematic, real-PLC HIL evaluation of autonomous LLM agents in a cyber-to-physical context. Its significance lies in:

1. Shifting the Threat Model: It demonstrates that target-specific Operational Technology (OT) knowledge (e.g., protocol specifics, object maps) is no longer a strict prerequisite for attack success. Capable agents can reconstruct this knowledge online through interaction, provided they have network access and bounded feedback.
2. Identifying Real Barriers: The study localizes failure points. It argues that protocol complexity and lack of target-specific knowledge act as "eroding friction" rather than durable security boundaries. Once an agent overcomes the interface barrier, the physical conversion barrier becomes the primary constraint, heavily influenced by process dynamics and observability.
3. Defense Evaluation: The framework offers a reproducible basis for evaluating defensive strategies. It suggests that defenses should focus on:
   • Restricting access to engineering services.
   • Validating writes that affect the process (state-aware invariants).
   • Guarding hazardous regions rather than just extreme thresholds.
   • Decoupling detailed monitoring data from write permissions to prevent "dual-use" telemetry from aiding attacks.

The authors emphasize that PLCBench does not discover new vendor vulnerabilities but rather characterizes the capability of autonomous agents to exploit existing, known interfaces to achieve physical outcomes. The results highlight that while current agents are not universally reliable, they are capable of executing sustained physical attacks in validated laboratory settings, necessitating a shift in how ICS security is evaluated and defended.