Doberman: a modular and distributed slow control system for small- to medium-scale experiments

This paper introduces Doberman, a lightweight, modular, and open-source slow control system with a web-based interface designed to bridge the gap between industrial SCADA frameworks and ad hoc solutions for small- to medium-scale physics experiments, as validated through diverse deployments ranging from underground spectrometers to liquid xenon facilities.

The Gist

Imagine you are running a high-tech, multi-story house. You have a fancy security system, a complex heating and cooling setup, and a bunch of smart appliances.

If you use a giant industrial SCADA system (like the ones used to control nuclear power plants), it's like hiring a team of 50 engineers just to change a lightbulb. It's powerful, but overkill, expensive, and hard to manage for a small house.

If you use ad-hoc scripts (random bits of code written by students), it's like trying to control your house by shouting instructions to your dog. It might work for a day, but if the dog gets sick or forgets a command, the whole house goes dark, and you have no idea why.

Doberman is the "Goldilocks" solution. It's a smart, lightweight, open-source system designed specifically for the "medium-sized" experiments in physics labs. It's not too heavy, not too flimsy, and it's built to be flexible.

Here is how Doberman works, broken down into simple concepts:

1. The "Hypervisor": The Conductor of the Orchestra

Think of the Doberman system as a symphony orchestra.

• The Hypervisor is the conductor. It doesn't play the instruments; it just makes sure everyone is awake, in tune, and playing the right tempo.
• If a violinist (a sensor) stops playing, the conductor notices immediately. If a musician is on a different stage (a different computer in the lab), the conductor can send a remote signal to wake them up.
• The "Heartbeat": To make sure the conductor isn't just sitting there staring at the ceiling, Doberman has a safety net. If the main server crashes, a tiny "watchdog" in a different building checks if the conductor is still alive. If not, it screams for help so no one misses a critical alarm.

2. The "Monitors": The Specialized Workers

Instead of one giant brain trying to do everything, Doberman uses a team of specialized workers called Monitors.

• Device Monitors: These are the workers who actually go out and talk to the machines (like checking the temperature of a liquid nitrogen tank). They are like the "eyes and ears" of the system.
• Pipeline Monitors: These are the "thinkers." They take the data from the eyes and ears and decide what to do.
  • Example: "The liquid nitrogen level is low? Pipeline: Open the valve to refill it."
  • Example: "The pressure is too high? Pipeline: Send an alarm to the human on duty."
• Alarm Monitors: These are the "messengers." If something goes wrong, they don't just beep; they call, text, or email the right person based on how serious the problem is.

3. The "Pipelines": The Decision Trees

Imagine a flowchart drawn on a whiteboard.

• Input: "Temperature is 50°C."
• Check: "Is it above 45°C?" -> Yes.
• Action: "Turn on the fan."
• Output: "Fan is ON."

In Doberman, these flowcharts are called Pipelines. They are built like Lego blocks. You can snap a "Temperature Check" block onto a "Valve Control" block. If you need to change the rules, you just swap the blocks. You don't have to rebuild the whole wall. This makes it incredibly easy to adapt when the experiment changes.

4. The "Doberview": The Dashboard

You don't need to be a computer programmer to use Doberman. The Doberview is a website (like a dashboard in a car) that shows you everything happening in real-time.

• The Map: It shows a picture of your experiment (like a blueprint of the house). If a pipe is leaking, the picture turns red. If a pump is running, it animates.
• The Details: Click on a box, and you see the history of that sensor. Did the temperature spike 10 minutes ago? You can see it on a graph.
• The Controls: You can turn valves on or off or change settings right from your phone or laptop, even if you are at home.

5. Real-World Examples

The paper shows Doberman working in three very different "houses":

• GeMSE (The Remote Cabin): A gamma-ray spectrometer deep underground in the Swiss mountains. No humans live there. Doberman acts as the caretaker, watching the temperature and refilling liquid nitrogen automatically for weeks at a time.
• XeBRA (The Workshop): A small lab where they test new ideas. Because they change equipment often, Doberman's "plug-and-play" nature lets them add new sensors in minutes, not days.
• PANCAKE (The Mansion): A massive liquid xenon experiment with 300 sensors. It's huge and complex. Doberman runs it like a distributed army, with small computers (Revolution Pi) sitting right next to the equipment to handle the heavy lifting, while the main brain stays safe on a central server.

Why Does This Matter?

Before Doberman, small physics labs were stuck between "too expensive/complex" and "too messy/unreliable."

• Doberman says: "You don't need a nuclear power plant to control your experiment. You just need a smart, flexible, open-source system that can talk to anything, wake itself up if it crashes, and tell you exactly what's wrong."

It turns a chaotic lab into a well-oiled machine, allowing scientists to focus on discovering new physics rather than fixing broken scripts.

Technical Summary

1. Problem Statement

Physics experiments, particularly those in the small-to-medium scale range, face a significant gap in slow control infrastructure:

• Industrial/High-End Solutions: Systems like SCADA or EPICS are robust and powerful but are often too complex, resource-intensive, and costly for smaller laboratories or R&D platforms.
• Ad-hoc Solutions: Many smaller setups rely on custom scripts or lack a unified infrastructure entirely. This leads to issues with long-term stability, data integrity, operational safety, and the inability to handle automated alerts or rapid responses to anomalies.
• Operational Needs: There is a need for a system that can monitor auxiliary parameters (voltages, temperatures, pressures, flows) at low sampling rates (~1 Hz), ensure detector stability, protect equipment, and provide an intuitive interface for operators who may not be experts in the underlying control system.

2. Methodology and Architecture

The authors present Doberman (Detector OBsERving and Monitoring ApplicatioN), a lightweight, modular, and open-source system designed to bridge the aforementioned gap. Its architecture is built on the following pillars:

A. Deployment Model

• Scalability: The system scales from a single-host installation to a distributed multi-host deployment.
• Distributed Readout: Device readout is handled by DeviceMonitors which can run on secondary hosts located physically close to the experimental hardware. This reduces cable lengths and improves resilience; if one host fails, only its subset of sensors is affected, while the rest of the system remains operational.
• Central Coordination: A Hypervisor runs on a central server to coordinate communication, supervise component health (via heartbeats), and route commands.

B. Database Backend

Doberman utilizes a dual-database architecture to optimize performance:

• Configuration Database (MongoDB): Stores system settings, device definitions, sensor metadata, and log messages. Its document-based (JSON) structure allows for flexible schema evolution as new device types are added.
• Time-Series Database (InfluxDB): Stores high-frequency measurement data and system health metrics (e.g., CPU load) optimized for time-series queries.

C. Core Components

• Monitor Class: The fundamental building block. It is a multi-threaded class where each thread executes a task (e.g., reading a sensor) and sleeps. Specialized subclasses include:
  • DeviceMonitors: Interface with physical instruments.
  • PipelineMonitors: Execute automation logic.
  • AlarmMonitor: Evaluates conditions and distributes notifications.
• Plugins: Device integration is achieved via Python plugins. These handle communication protocols (Ethernet, Serial, etc.) and parse device-specific messages, exposing a unified interface to the core system.
• Pipelines: A node-based framework for continuous processing and automation. Pipelines are directed graphs consisting of:
  • Source Nodes: Ingest data from the live broker or database.
  • Intermediate Nodes: Perform filtering, threshold checks, derivatives, or mathematical expressions.
  • Sink Nodes: Write derived data to the database or trigger control actions (e.g., opening a valve).
  • Types: Pipelines are categorized as Alarm (detect anomalies), Convert (compute derived quantities), or Control (actuate hardware).

D. Alarm Distribution

• Leveled Alarms: Alarms have a base level and an escalation level if not cleared.
• Notification: The system supports multiple channels including Email, SMS (via Nextp GmbH), and automated phone calls (via Twilio).
• Recipients: Grouped based on severity (e.g., shifters for routine issues, experts for critical failures).

E. Graphical User Interface (Doberview)

• Technology: A web-based interface built with Node.js and Bootstrap, accessible on desktop and mobile.
• Features:
  • Interactive Overview: An SVG-based P&ID diagram where elements link to sensors, showing real-time values and status (colors/animations for binary states).
  • Sensor Inspection: Allows viewing history, adjusting thresholds, and issuing direct control commands.
  • Pipeline Management: Visualizes pipeline status (active/silent/error) and allows editing of node graphs.
  • Visualization: Includes a built-in plotter for ad-hoc analysis and supports embedding Grafana dashboards.

3. Key Contributions

1. Bridging the Gap: Doberman successfully fills the niche between heavy industrial SCADA systems and fragile ad-hoc scripts, offering enterprise-grade reliability with laboratory-friendly simplicity.
2. Modular Plugin Architecture: The ability to integrate heterogeneous instrumentation (commercial and custom) via Python plugins without modifying the core system.
3. Robust Distributed Design: The separation of DeviceMonitors from the central server ensures that local hardware failures do not crash the entire monitoring infrastructure.
4. Automated Control Logic: The pipeline framework allows for complex, automated responses (e.g., refilling liquid nitrogen based on consumption rates) without human intervention.
5. Open Source Availability: The software, documentation, and a growing library of device plugins are publicly available under permissive licenses.

4. Results and Validation

The system has been validated in three distinct experimental environments at the University of Freiburg:

• GeMSE (Gamma-ray Spectrometer): A remote underground facility. Doberman enabled fully autonomous operation for years, managing cryogenic cooling and bias voltage without on-site staff, requiring manual intervention only every few weeks.
• XeBRA (Liquid Xenon R&D): A small-scale platform (~10 kg) used for testing novel detector concepts. The system successfully handled frequent hardware changes and rapid integration of new instruments, demonstrating flexibility for iterative R&D workflows.
• PANCAKE (Large Liquid Xenon Test Facility): A complex, medium-scale experiment (~400 kg inventory, ~300 sensors). This is the most complex deployment, featuring:
  • Distributed readout using industrial Revolution Pi modules (Raspberry Pi Compute Module based).
  • Integration of serial devices via Ethernet converters.
  • Continuous monitoring and automated control over multi-month campaigns.
  • Successful support of a 130 kg active mass TPC operation for ~60 days with high stability in cryogenic and high-voltage parameters.

5. Significance

• Operational Efficiency: Doberman significantly reduces the need for continuous human supervision through robust automation and alarm handling.
• Data Integrity: By providing a unified infrastructure, it ensures consistent data logging and traceability, which is crucial for physics data correction and calibration.
• Scalability: The system has proven capable of scaling from a single host monitoring a few sensors to a distributed network managing hundreds of quantities across multiple subsystems.
• Future Outlook: The system is considered functionally mature and is selected as the slow control system for the upcoming DELight (light dark matter) experiment. Its success suggests it is a viable standard for laboratory-scale and medium-scale experimental infrastructures globally.