A monitoring system for collecting and aggregating metrics from distributed clouds

This paper presents the design and implementation of a monitoring system for distributed clouds that utilizes node-based agents to collect, aggregate, and persist machine, container, and application metrics, making them accessible via multiple APIs to ensure the necessary observability for real-world viability.

The Gist

Imagine you are the manager of a massive, traveling circus. But instead of tents in one big field, your circus sets up temporary camps in different towns every day. Some camps are in busy cities, others in quiet villages. You have thousands of performers (your data and applications) and hundreds of wagons (your servers) moving around constantly.

This is what a Distributed Cloud (DC) is: a flexible, temporary network of computers scattered across different locations, created on the fly to do specific jobs.

The problem? When your circus is this scattered and moving so fast, it's hard to know:

• Is the elephant (a server) overheating?
• Is the clown (an application) running out of jokes (memory)?
• How many tickets (data) are being sold in the village camp versus the city camp?

If you can't see what's happening, you can't fix problems, and the show might crash. This is where the paper comes in. The authors built a super-observant "Ringmaster's Eye"—a monitoring system designed specifically for this chaotic, moving circus.

Here is how their system works, broken down into simple parts:

1. The Scouts (The Agents)

In every single wagon (server node) in your circus, there is a tiny, super-fast scout.

• What they do: These scouts constantly check three things:
  • The Wagon itself: Is the engine hot? Is the fuel low? (Machine metrics).
  • The Performers inside: Is the clown juggling too many balls? Is the acrobat tired? (Container metrics).
  • The Act: Is the magic trick working? (Application metrics).
• The Trick: Instead of shouting out loud all the time (which would waste energy), these scouts quietly write their notes on a small clipboard and wait for a signal.

2. The Health Check (The Signal)

Every few seconds, the main Ringmaster (the Control Plane) sends a "Hello!" signal to all the wagons.

• The Magic: When a wagon replies "Hello!", it doesn't just say "I'm here." It piggybacks its clipboard notes onto that "Hello!" message.
• Why do this? It saves time and energy. Instead of the wagon making a separate, long trip to the Ringmaster just to deliver a report, it delivers the report while saying hello.
• The Catch: If the wagon is too small (low resources), it deletes its notes immediately after sending them. If the Ringmaster loses the message, the data is gone. The authors decided this is okay because keeping the notes on the wagon takes up too much space, and losing a few seconds of data is better than slowing down the whole circus.

3. The Big Board (The Control Plane)

The Ringmaster's tent has a giant digital whiteboard.

• Aggregation: The Ringmaster doesn't just look at one wagon; they look at the whole camp. They take the notes from 50 wagons in "Town A" and add them up to say, "Town A is running at 80% capacity." This gives a big picture view of the whole distributed cloud.
• Storage: They write these stats down in a massive, organized ledger (using a tool called Prometheus) so they can be looked at later.

4. The Reporters (The Clients)

Who needs this information?

• The Managers: They want to see a dashboard to know if the show is healthy.
• The Schedulers: These are like the stage managers who decide where to put the next act. If they see a wagon is full, they send the new act to a different wagon.
• The AI: If you are training a robot to run the circus, it needs to read all these past reports to learn how to make better decisions in the future.

How People Get the Data

The system offers two ways to get the reports:

1. The "On-Demand" Menu (REST API): You ask, "Show me the stats for the wagon in Paris from 2:00 PM to 3:00 PM," and the system hands you a specific report.
2. The Live Stream (Streaming API): You subscribe to a channel. As soon as a wagon updates its stats, the Ringmaster pushes the new number to your screen instantly. This is great for live dashboards where you need to see problems the second they happen.

Why is this special?

Most monitoring systems are built for a static office building where everything stays in one place. But a Distributed Cloud is like a moving parade.

• Old systems would get overwhelmed trying to track a parade that keeps changing its route.
• This new system is built to be lightweight (so it doesn't slow down the wagons) and flexible (so it can handle wagons joining or leaving the parade instantly).

The Future

The authors admit that if the circus gets too big (thousands of wagons), the Ringmaster might get too busy. In the future, they plan to let the wagons talk to each other directly (like a gossip network) to share the load, so the Ringmaster only needs to check in on a few key wagons. They also want to make the system smarter at compressing data so it takes up less space, and add an alarm system that screams if something breaks.

In a nutshell: They built a lightweight, flexible "health monitor" that lets you keep your eye on a cloud of computers that is constantly appearing, disappearing, and moving around, ensuring the show always goes on without a hitch.

Technical Summary

Here is a detailed technical summary of the paper "A monitoring system for collecting and aggregating metrics from distributed clouds."

1. Problem Statement

The traditional centralized cloud model is increasingly inefficient for modern applications requiring real-time processing of large data volumes, leading to the emergence of Distributed Cloud (DC) models. DCs consist of dynamically formed and discarded ad-hoc clouds with resources tailored to specific user needs (location, latency, privacy).

However, the dynamic nature of DCs—characterized by nodes constantly joining or leaving and resource constraints on edge hardware—presents significant challenges for observability:

• Lack of Native Solutions: Existing monitoring tools (e.g., Prometheus, Kubernetes metrics pipelines) are designed for static or federated clusters, not the fluid, ephemeral nature of DCs.
• Resource Constraints: DC nodes often have limited hardware resources, making heavy monitoring agents impractical.
• Data Aggregation Needs: Users and automated systems (schedulers, autoscalers) require not just individual node metrics but also aggregated views of the entire DC state to make informed decisions.
• Observability Gap: Without a robust monitoring system, DCs lack the necessary visibility for fault detection, performance tuning, and automated scaling, hindering their viability in real-world scenarios.

2. Methodology

The authors propose a native monitoring system designed specifically for the DC architecture, implemented as part of the open-source Constellations (c12s) platform. The system follows a hybrid architecture separating components running on nodes from those in the control plane.

System Architecture

• Node-Level Components:
  • Collectors: Separate agents for Machine-level (using Node/Windows exporters), Container-level (using cAdvisor), and Application-level metrics. These expose metrics via a standardized HTTP GET /metrics endpoint compliant with the OpenMetrics format.
  • Metrics Agent (Starometry): Responsible for polling collectors, temporarily storing data on the local file system, and transferring data to the control plane.
• Control Plane Components:
  • Processor (Protostar): Monitors node health via a ping protocol. Crucially, it piggybacks metric data onto these health-check responses to minimize network overhead. It aggregates node-level metrics (e.g., summing CPU/RAM) to create DC-level views.
  • Metrics Storage: Uses Prometheus as the time-series database for long-term persistence.
  • Reader: Provides APIs for clients to retrieve data.

Data Flow & Protocols

1. Collection: The Metrics Agent periodically polls local collectors.
2. Transfer: Instead of a dedicated push/pull for metrics, the system piggybacks metric data onto the health-check protocol. When a node replies to a "ping" from the Processor, it includes the collected metrics.
3. Persistence: The Processor writes received metrics to the Prometheus storage.
4. Aggregation: The Processor periodically aggregates node metrics (sum/average) to generate DC-level metrics.
5. Access: Clients access data via:
   • REST API: For historical data retrieval (time-range queries).
   • Streaming API: Using NATS as a message broker, allowing clients to subscribe to specific topics (e.g., metrics.nodes.* or metrics.dc.X) for real-time dashboards and autoscaling decisions.

Implementation Details

• Language: All components are written in Go.
• Messaging: NATS is used for the streaming API and internal communication.
• Storage: Prometheus is used for its native OpenMetrics support and query capabilities.
• Testing: Validated in a laboratory environment using simulated nodes in virtual machines.

3. Key Contributions

• Native DC Monitoring Design: A comprehensive architecture specifically tailored for the dynamic, resource-constrained nature of Distributed Clouds, distinguishing between node-level and DC-level observability.
• Efficient Data Transfer Strategy: The introduction of a piggybacking mechanism where metrics are transmitted alongside health-check pings. This reduces the number of network requests and minimizes overhead on resource-limited nodes.
• Multi-Level Observability: The system unifies metrics from three distinct layers:
  1. Machine/Hardware: CPU, memory, disk.
  2. Container/Isolated Environment: Pod/container resource usage.
  3. Application: Custom, user-defined metrics generated by applications.
• Aggregation Logic: A built-in mechanism to aggregate raw node data into high-level DC statistics, providing a holistic view of the system state without requiring clients to query every individual node.
• Open-Source Implementation: The full implementation (Starometry, Protostar) is integrated into the Constellations (c12s) project, providing a practical reference for DC monitoring.

4. Results

• Functional Validation: The system was successfully implemented and tested in a simulated laboratory environment. It demonstrated the ability to collect, aggregate, and serve metrics via both REST and Streaming APIs.
• API Completeness: The system provides a comprehensive set of endpoints (detailed in Table I of the paper) allowing retrieval of metrics for specific nodes, containers, applications, and entire DCs over specific time ranges.
• Scalability Proof of Concept: While tested on a small scale, the design explicitly addresses scalability through aggregation and the separation of concerns between the control plane and nodes.

5. Significance and Future Work

Significance:
This work addresses a critical gap in the Distributed Cloud ecosystem. By providing a lightweight, native monitoring solution, it enables the operational viability of DCs. It empowers users to manage complex, ephemeral infrastructures and supports advanced automation (autoscaling, scheduling) and machine learning applications that rely on high-quality training data derived from system metrics.

Limitations & Future Directions:

• Centralization Bottleneck: The current design relies heavily on a centralized control plane. The authors acknowledge this may not scale to thousands of nodes without causing traffic congestion.
  • Future Work: Implementing a Peer-to-Peer (P2P) aggregation strategy within DCs (e.g., gossip protocols) to distribute the load.
• Data Loss: The current "fire-and-forget" approach (node deletes data after sending) risks data loss if the control plane fails to persist it.
  • Future Work: Implementing an acknowledgment mechanism where nodes retain data until a confirmation is received.
• Data Reduction: The paper suggests future integration of lossless compression and lossy dynamic sampling (discarding stable data points) to further reduce storage and network requirements.
• Alerting: Developing a robust alerting mechanism is identified as a necessary next step for a production-ready system.