Beyond the EPICS: comprehensive Python IOC development with QueueIOC

This paper introduces QueueIOC, a comprehensive Python framework that addresses EPICS architectural deficiencies by leveraging the caproto protocol and event loops to create flexible, maintainable IOCs capable of handling complex applications like StreamDevice workalikes, sequencers, and detector integration.

The Gist

The Big Picture: Fixing a Clunky Control System

Imagine you are running a massive, high-tech orchestra (a particle accelerator like HEPS). You need to control hundreds of instruments (motors, lasers, detectors) perfectly in sync.

For decades, scientists have used a system called EPICS to conduct this orchestra. It's reliable, but it's like trying to conduct a symphony using a 1980s typewriter. It works, but it's clunky, hard to learn, and writing new music (software) for it is a nightmare. You have to follow strict, confusing rules just to make two instruments play together.

The authors of this paper say: "We can do better." They built a new conductor's podium using Python (a modern, flexible programming language) called QueueIOC. It speaks the same language as the old system (so the orchestra doesn't have to stop playing), but it lets the conductor write music that is shorter, clearer, and much easier to understand.

---

The Problem: The "Rulebook" is Too Complicated

In the old EPICS system, connecting two devices is like trying to solve a puzzle where the pieces change shape depending on the time of day.

• The Analogy: Imagine you want to tell a robot arm to move. In the old system, you have to fill out a 10-page form with checkboxes for "Input Link," "Output Link," "Passive Process," and "Forward Link." If you get one tiny rule wrong, the robot arm might spin out of control or do nothing at all.
• The Result: Developers spend 90% of their time fighting the system's rules and only 10% actually solving the physics problems. It's inefficient and frustrating.

The Solution: QueueIOC (The "Smart Queue")

The authors created QueueIOC. Think of this as replacing the 10-page form with a simple text message.

• How it works: Instead of complex forms, the system uses a simple "Request and Reply" model.
  • The Old Way: "I want to set the temperature, but first I must check if the link is active, then verify the subtype, then..."
  • The New Way (QueueIOC): "Hey, set the temperature to 50." (Request). "Okay, done." (Reply). "Oh, by the way, the temperature just changed." (Notification).

This is called the Submit/Notify Pattern.

• Analogy: Imagine a busy restaurant kitchen.
  • Old EPICS: The waiter has to run to the chef, check a clipboard, ask if the stove is on, check the ingredient list, and then whisper the order.
  • QueueIOC: The waiter drops a ticket in a queue (a line) at the front. The chef (the main loop) picks up the ticket, cooks the meal, and puts the finished dish in a notification bin. The waiter just picks it up. No complex rules, no running around.

The "Sequencer" Problem: Avoiding the "Spaghetti Code"

In the old system, if you wanted to move three motors so they don't crash into each other (anti-bumping), you had to write a "sequencer."

• The Analogy: Writing a sequencer in the old system is like writing a story using only the word "Go" and "Stop." If you want the story to be complex, you end up with a tangled mess of instructions that looks like spaghetti code. If you change one "Go," the whole story might break, and you can't find where the error is.
• The QueueIOC Fix: They built a "Smart Traffic Light" system. You just tell the system: "Motor A and Motor B must stay 1 meter apart." The system automatically handles the complex logic of who moves when. The developer writes a simple rule, and the system handles the messy details.

The "SDK" Analogy: Building with LEGO vs. Clay

The paper also talks about how they handle different hardware (detectors, lasers).

• Old Way: Every time you get a new laser, you have to sculpt a new clay model from scratch. It takes forever, and if you make a mistake, the whole thing collapses.
• QueueIOC Way: They built a set of LEGO bricks (called "Mini-SDKs"). Whether you have a laser, a camera, or a motor, you just snap the right bricks together.
  • If a new laser comes out, you don't rebuild the whole system; you just snap on a new brick.
  • This makes the code modular. If a part breaks, you can pull it out and test it alone without breaking the whole orchestra.

The Result: Faster, Smarter, Simpler

By switching to this new Python-based system:

1. Code is shorter: What used to take 500 lines of confusing code now takes 50 lines of clear English-like code.
2. Fewer bugs: Because the logic is simple, there are fewer places for errors to hide.
3. Easier to learn: A new engineer can understand the system in a day, not a month.
4. Future-proof: The system is so simple that it could eventually be used to control any device, not just particle accelerators. It could become a universal language for controlling robots, smart homes, or factory machines.

Summary in One Sentence

The authors took a complex, rigid, and confusing control system (EPICS) and rebuilt it using modern, flexible tools (Python/QueueIOC) to make it as simple as sending a text message, saving scientists hours of headaches and allowing them to focus on science rather than software headaches.

Technical Summary

1. Problem Statement

The authors identify significant architectural deficiencies in the Experimental Physics and Industrial Control System (EPICS) that lead to inefficiencies in development and application, particularly at the High Energy Photon Source (HEPS). The core issues include:

• Inexpressiveness and Complexity: The EPICS architecture relies on a complex, non-decomposable rule set for "record links" (input, output, forward links) and attributes. This makes learning difficult and limits the ability to naturally express complex logic (e.g., motor anti-bumping or PID controllers).
• Lack of Modularity: EPICS databases (collections of records) cannot be nested at runtime, making it hard to abstract reusable functionalities as pluggable sub-databases.
• Inconvenient Scripting: The st.cmd language lacks looping or conditional constructs, forcing developers to rely on external code generators or error-prone "sequencers" (based on the seq module).
• High Maintenance Costs: Customizing records often requires deep knowledge of EPICS internals and results in significant code duplication.
• GUI Limitations: While EPICS Operator Interfaces (OPIs) are easy to compose, the underlying logic often suffers from mixing business logic with widget operations, leading to race conditions and complex state management.

2. Methodology

The authors propose a systematic replacement of traditional EPICS IOCs with Python-based IOCs using the caproto library (a pure-Python implementation of the Channel Access protocol). Their methodology is built on three pillars:

A. The Submit/Notify Pattern (GUI Architecture)

Inspired by EPICS OPIs but generalized for software engineering, the authors introduce the Submit/Notify pattern to decouple frontends (widgets) from backends:

• Widgets only "submit" update requests (e.g., user inputs) to a main event loop.
• The Main Event Loop processes these and "notifies" widgets of actual state changes.
• This eliminates direct widget-to-widget interaction, shared implicit states, and race conditions, effectively applying the Actor Model and CSP (Communicating Sequential Processes) principles to GUI programming.

B. The QueueIOC Framework

The core contribution is QueueIOC, a framework that treats EPICS operations (caput, caget, camonitor) as specialized combinations of Requests/Replies and Notifications.

• Architecture: Instead of the complex EPICS database/record support stack, QueueIOC uses a regular Python thread running a main event loop.
• Async Handling: It uses a thin layer of message queues to isolate the non-async Python logic from the async/await requirements of the underlying caproto library.
• Abstraction: It abstracts away the "database" concept. Device drivers, interrupt routines, and monitors are handled implicitly in the underlying control layer or via Python classes.

C. Systematic Simplification

The framework aims to approach the complexity/cost lower-bound for control software. By leveraging Python's expressiveness, the authors replace:

• StreamDevice/Asyn: With native Python classes handling serial/TCP/UDP communication and data parsing.
• Seq-based Sequencers: With Python state machines and decorators, avoiding the "goto hell" of traditional sequencers.
• Hardware Abstraction: Using runtime class creation to adapt to variable device interfaces without code generators.

3. Key Contributions

1. QueueIOC Framework: A general-purpose Python IOC framework that systematically reduces development and maintenance costs by replacing the EPICS record/link architecture with a clean, event-loop-based Python architecture.
2. Submit/Notify Pattern: A robust GUI architecture that decouples logic from presentation, reducing concurrency bugs and improving maintainability.
3. Specialized IOC Implementations:
   • QScanIOC: A workalike for StreamDevice/Asyn that supports native Python data processing (text/binary) and dynamic PV creation.
   • QMotorSeqIOC: A simplified state machine for motor control (e.g., anti-bumping, multiplexing) that replaces complex seq logic with clear Python logic.
   • QMonochromatorIOCBase: A modular framework for monochromators that encapsulates complex coordinate transformations (Bragg angle, wavelength, energy) into reusable functions, separating calibration from control logic.
   • QDetectorIOC: A framework for detector integration (detailed in an accompanying paper) that overcomes architectural limitations of areaDetector.
4. s6-epics & qioc_s6: A management system using s6 (a service manager) to handle IOC lifecycle (start/stop/exit gracefully) and logging, replacing procServ with a more robust solution compatible with both Python and C++ IOCs.
5. Modular SDKs: The creation of "mini-SDKs" (e.g., PyHfda) that encapsulate vendor-specific protocols, allowing for isolated testing and reuse across different IOCs.

4. Results and Examples

The paper provides concrete examples demonstrating the efficacy of QueueIOC:

• Performance: Benchmarks show QueueIOC achieving refresh rates of 100 Hz and monitor rates of 500 Hz, comparable to traditional EPICS IOCs.
• Code Reduction:
  • ADXspress3: Merged and simplified from two upstream versions, significantly easier to maintain.
  • Motor Anti-Bumping: Complex logic abstracted into a library where developers only specify mathematical constraints and motor lists.
  • Monochromators: Complex 3-way conversions (Energy $\leftrightarrow$ Wavelength $\leftrightarrow$ Angle) are encapsulated in single functions, reducing the code size of new IOCs by orders of magnitude compared to optics or seq implementations.
• Flexibility: The qscan_hfda IOC demonstrates handling of three different laser controller models with varying protocols and state-readback capabilities within a single, clean codebase, something difficult to achieve with StreamDevice or seq.
• Usability: Device parameters (IP addresses, ports) can be passed via command line, separating configuration from source code, unlike the rigid st.cmd approach.

5. Significance

• Efficiency Boost: The authors report efficiency improvements of 1–2 orders of magnitude in development and maintenance, correlating with a significant reduction in code volume.
• Future-Proofing: By moving away from hardware-limited design paradigms (VME-era constraints) to modern Python practices, the framework is better suited for the current landscape of PC-based control systems.
• AI Readiness: The pursuit of minimal complexity and modularity aligns with the goal of facilitating collaboration between human and artificial intelligence, making code more understandable and automatable.
• Unified Ecosystem Potential: The authors suggest that QueueIOC's agnostic approach to device protocols (focusing on requests/replies and notifications) could serve as a foundation for a unified device-control ecosystem, potentially bridging EPICS, TANGO, and Karabo.
• Backward Compatibility: The solution is designed to be a smooth transition path, reusing the Channel Access (CA) protocol to ensure compatibility with existing EPICS clients and tools.

In conclusion, QueueIOC represents a paradigm shift from the rigid, record-based architecture of EPICS to a flexible, Python-native, event-driven architecture. It successfully addresses the expressiveness and maintainability issues of EPICS while maintaining high performance and protocol compatibility.