Design and Implementation of a UDP-Based Command Interface for the INO ICAL Experiment

This paper presents a reliable, UDP-based command interface for the INO ICAL experiment that mitigates packet loss through a custom handshaking and checksum mechanism, with its performance and robustness validated on the Mini ICAL prototype.

The Gist

Imagine you are the conductor of a massive orchestra, but instead of 100 musicians, you have 28,800 instruments. Each instrument is a tiny, independent robot (called an RPC-DAQ) sitting in a giant underground cavern, waiting to detect invisible particles zipping through the air.

Your job is to tell all 28,800 robots exactly what to do at the exact same moment: "Start listening!" "Stop!" "Check your batteries!" or "Send me your data!"

This is the challenge faced by the INO ICAL experiment in India. The paper you shared describes how the scientists built a special "remote control" system to manage this army of robots without causing chaos.

Here is the story of their solution, broken down into simple concepts:

1. The Problem: The "Too Many Phones" Dilemma

In a normal office, if you want to talk to everyone, you might use a phone system (like TCP). It's reliable; it makes sure every message is heard and in the right order. But imagine trying to call 28,800 people individually on a phone line. The lines would get clogged, the calls would take too long to connect, and the system would crash.

The scientists needed a way to shout a message to everyone at once, instantly. They chose UDP (User Datagram Protocol).

• The Analogy: Think of UDP like a megaphone or a radio broadcast. You shout a message, and everyone within earshot hears it immediately. It's super fast and doesn't require a connection.
• The Catch: With a megaphone, you don't know if everyone heard you. Maybe the wind blew the sound away, or someone was wearing noise-canceling headphones. In computer terms, packets can get lost. If a robot misses the "Start" command, the whole experiment could fail.

2. The Solution: The "Smart Megaphone" (HPCI)

The scientists couldn't use the slow phone system (TCP), but they couldn't trust the unreliable megaphone (UDP) either. So, they invented a hybrid system called HPCI (Hybrid Protocol based Command Interface).

Think of HPCI as a smart megaphone with a "Did you hear me?" feature.

Here is how it works:

1. The Shout (Multicast): The central server (the conductor) shouts a command to a specific group address. All 28,800 robots listen to this group channel.
2. The Checksum (The ID Badge): Every message comes with a special "mathematical seal" (called a CRC Checksum). It's like a wax seal on a letter. If the letter gets torn or the ink smudges during the journey, the seal breaks, and the robot knows, "This message is damaged; I can't trust it."
3. The Handshake (The Nod):
   • If a robot hears the command and the seal is intact, it sends a quick "Nod" (an Acknowledgment) back to the server.
   • If the server doesn't get a "Nod" within a split second, it assumes the robot missed the message.
   • The server then switches to Unicast mode (like a direct phone call) and shouts the command only to that specific robot until it gets a "Nod."

3. The "Mini-ICAL" Test Drive

Before building the full system for the giant experiment, they tested it on a smaller version called mini-ICAL.

• The Analogy: This is like testing a new airplane engine on a small drone before putting it on a jumbo jet.
• They put 20 robots in a room and ran the system. They simulated heavy traffic (like a busy highway) to see if the "Smart Megaphone" could still get through.
• The Result: It worked perfectly. The system handled the traffic, fixed lost messages automatically, and kept everything synchronized.

4. Why Does This Matter?

In particle physics, timing is everything. If one robot starts listening 1 second later than the others, the data is useless.

• The Goal: The HPCI system ensures that all 28,800 robots start and stop within less than 1 millisecond of each other.
• The Safety Net: Because the system checks for errors and re-sends lost messages, it is 99.99% reliable. It's like having a backup generator that kicks in the exact millisecond the main power flickers.

Summary

The paper describes a clever engineering trick: taking a fast but unreliable communication method (UDP) and adding just enough "safety features" (handshakes and error checks) to make it reliable enough for a massive scientific experiment.

Instead of building a slow, heavy system that tries to be perfect, they built a fast, lightweight system that checks its work as it goes. This allows the INO ICAL experiment to control a massive network of detectors efficiently, ensuring that when the universe sends a neutrino, the entire team of 28,800 robots is ready to catch it together.

Technical Summary

1. Problem Statement

The India-based Neutrino Observatory (INO) Iron Calorimeter (ICAL) experiment requires the control and data acquisition of a massive array of 28,800 Resistive Plate Chamber (RPC) detectors. Each detector is interfaced with a Front-End Data Acquisition module (RPC-DAQ) equipped with an Ethernet interface.

• Scalability Challenge: Managing 28,800 network nodes simultaneously requires a protocol capable of efficient group addressing (multicast) and low latency.
• Protocol Limitations:
  • TCP: While reliable, it is connection-oriented, resource-heavy, lacks native multicast support, and introduces latency due to connection handshakes and congestion control. This makes it unsuitable for large-scale, real-time command distribution in embedded systems.
  • Standard UDP: Lightweight and supports multicast, but lacks reliability mechanisms (no guaranteed delivery, no ordering, no retransmission), leading to potential packet loss and command inconsistency.
• Resource Constraints: The RPC-DAQ modules utilize low-end FPGAs (Intel Cyclone IV) with limited memory and processing power, precluding the implementation of a full TCP/IP stack or complex reliability algorithms.

2. Methodology: The Hybrid Protocol based Command Interface (HPCI)

The authors propose HPCI, a custom command interface that hybridizes the efficiency of UDP with the reliability features of TCP, specifically optimized for embedded FPGA environments.

Core Architecture

• Dual-Socket Mechanism: Each RPC-DAQ maintains two UDP sockets:
  1. Multicast Socket: Receives global commands (e.g., "Run Start") sent to a specific group IP (e.g., 239.0.0.1).
  2. Unicast Socket: Used for individual status queries and, crucially, for sending acknowledgments back to the server.
• Handshaking & Retransmission:
  • The server sends a command with a unique Sequence Number.
  • The DAQ processes the command and sends an acknowledgment (ACK) containing the same sequence number via Unicast.
  • If the server does not receive an ACK within a defined timeout, it retransmits the command in Unicast mode to the specific non-responsive DAQ.
  • The server maintains a local database of active DAQs, updating it based on ACKs and filtering out failed nodes.

Packet Structure & Error Detection

• Application-Level CRC: To mitigate the risk of data corruption in the FPGA's internal data paths (NIOS soft-core, DMA transfers) which standard Ethernet hardware CRC might miss, HPCI implements a 16-bit CRC (CRC-16, polynomial 0x8005) over the entire application payload.
• Packet Fields: Includes Start Markers, DAQ ID, Command Word, Sequence Number, Payload Size, Data, and Checksum.
• Sequence Tracking: The sequence number allows the server to match ACKs to specific command instances, detect duplicates, and ignore stale packets, ensuring application-level ordering despite UDP's unordered nature.

Implementation Details

• Front-end (DAQ): Uses an interrupt-driven approach. The Ethernet controller (Wiznet W5300) triggers an interrupt to the NIOS soft-core processor. The Interrupt Service Routine (ISR) performs a quick CRC check and sets a flag, allowing the main loop to handle the heavy lifting of command decoding and processing without blocking other critical tasks.
• Back-end (Server): Runs on a Linux server (Intel Xeon). It manages the DAQ database, handles multicast transmission, processes incoming ACKs, and manages the retransmission logic for failed nodes.

3. Key Contributions

1. Optimized Reliability for Embedded Systems: HPCI provides TCP-like reliability (acknowledgments, retransmission, ordering) without the overhead of a full TCP stack, making it feasible for low-resource FPGAs.
2. Hybrid Multicast/Unicast Strategy: Efficiently combines the speed of multicast for global commands with the precision of unicast for status verification and error recovery.
3. Application-Layer Integrity: The introduction of a custom CRC-16 check specifically targets vulnerabilities in the FPGA's internal data path, adding a layer of security beyond standard Ethernet frame checks.
4. Scalable Configuration Management: A mechanism to remotely update network parameters (IP, MAC, Gateway) stored in flash memory, allowing for dynamic reconfiguration of the 28,000+ node network.

4. Results and Performance

The system was validated using the Mini-ICAL prototype (20 RPC-DAQ units) before full-scale deployment.

• Cycle Times:
  • Non-Busy Mode: Average command cycle times ranged from 174 µs to 615 µs depending on payload size (18 to 100 bytes).
  • Busy Mode (High Data Load): Even under a synthetic load of 45 Mbps (simulating 10 kHz event rates), cycle times remained stable, increasing only slightly to 864 µs for 100-byte commands.
• Reliability: The protocol achieved a delivery success rate exceeding 99.99% for critical control commands through the combination of ACK tracking and conditional retransmission.
• Latency: The system meets the strict requirement of <1 ms response latency per DAQ, essential for synchronized operation.
• Operational Status: The HPCI-based Run Control software has been successfully used for daily data taking in Mini-ICAL since 2018, demonstrating robustness in a real-world environment.

5. Significance

• Enabling Mega-Science Projects: HPCI provides a proven, scalable solution for experiments requiring the simultaneous control of tens of thousands of distributed sensors, a common challenge in high-energy physics.
• Robustness in Harsh Environments: By addressing both network-level packet loss and internal FPGA data corruption, the protocol ensures the integrity of critical commands (e.g., High Voltage configuration, Run Start/Stop) which, if corrupted, could damage equipment or ruin data.
• Cost-Effectiveness: The ability to run on low-cost, low-power FPGAs without requiring complex operating systems or heavy software stacks makes the solution economically viable for large-scale deployments.
• Future Readiness: While currently tested on 20 units, the architecture is designed to scale to the full 28,800 DAQs of the INO ICAL experiment, ensuring the experiment's physics goals (tracking charged particles with high precision) can be met.