The environmental impact, carbon emissions and sustainability of computing in the ATLAS experiment

This paper outlines the ATLAS experiment's ongoing strategies to evaluate and mitigate the environmental impact of its massive, globally distributed computing infrastructure in preparation for the significantly increased resource demands of the High-Luminosity LHC era.

The Gist

The ATLAS Experiment's Green Computing Journey: A Simple Guide

Imagine the ATLAS experiment at CERN as a massive, global detective agency. Its job is to solve the universe's biggest mysteries by smashing particles together at nearly the speed of light. But to do this, it needs a superpower: computing.

Right now, ATLAS uses a "digital brain" spread across about 100 computer centers around the world. It has over a million CPU cores (the brains of computers) running at the same time and stores enough data to fill over a million hard drives.

However, there's a catch. As the Large Hadron Collider gets upgraded to become even more powerful (the "High-Luminosity" era), ATLAS will need 3 to 4 times more computing power by the 2030s, and 10 times more by the 2040s.

The Problem: More computing means more electricity, which usually means more carbon emissions (pollution). If they just buy more computers and plug them in, the experiment's "carbon footprint" would explode.

The Solution: This paper is a roadmap on how ATLAS plans to grow its digital brain without choking the planet. They are treating sustainability like a new rulebook for their detective work. Here is how they are doing it, explained with some everyday analogies.

---

1. Making Everyone Aware: The "Energy Bill" Notification

Imagine you turn on your lights and get a text message saying, "This light bulb used as much energy as boiling a kettle for 10 minutes." That's what ATLAS is doing for its scientists.

• The Analogy: They built a feature into their workflow system (called PanDA) that acts like a carbon receipt. Every time a scientist runs a job (a piece of code to analyze data), the system calculates how much carbon that job produced.
• Why it helps: Just like seeing your electricity bill makes you think twice about leaving the TV on, seeing the "carbon cost" of a bad code loop encourages scientists to write cleaner, faster, and more efficient code. They aren't shaming anyone; they are just making the invisible visible.

2. Changing the Rules: The "Library" Strategy

ATLAS has policies that act like a smart librarian managing a massive library.

• The "Data Carousel" (Tape vs. Disk): Imagine you have a huge library. Keeping every book on the main shelves (Disk) is expensive and takes up space. ATLAS uses a "carousel" system: if a book (data) hasn't been read in a while, they move it to a compact, low-energy archive (Tape). Tape uses way less electricity than active hard drives. It's like putting books in a basement storage unit instead of keeping them all on the front desk.
• The "Just-in-Case" Problem: Sometimes, scientists keep data "just in case" they might need it later. This is like hoarding old newspapers. ATLAS realized that instead of hoarding, they can re-print the newspaper (reproduce the data) if needed. It turns out, printing a new copy is often cheaper (in energy) than keeping the old one stored for years.
• Compression: They are squeezing their data files tighter, like packing a suitcase. By compressing data, they save massive amounts of storage space. The tiny bit of energy needed to squeeze the suitcase is nothing compared to the energy saved by not needing a bigger suitcase.

3. Fixing the Leaks: The "Auto-Pilot" System

Computing is full of leaks. Sometimes jobs fail, or computers sit idle but still use power.

• The "HammerCloud" Test: Imagine a security guard who constantly tests the doors of a building to make sure they lock. If a door is broken, the guard locks it off immediately so no one wastes time trying to get in. ATLAS uses a tool called HammerCloud to test computer sites. If a site is broken, it's automatically taken offline so energy isn't wasted on failed jobs.
• The "Scout" Jobs: Before sending a massive army of jobs to a site, ATLAS sends a few "scouts" first. If the scouts get lost or fail, the army doesn't go. This prevents wasting millions of computing hours on a broken path.

4. Smart Timing: The "Surfing" Strategy

The power grid isn't the same all day. Sometimes the sun is shining (solar power), or the wind is blowing (wind power), making electricity clean and cheap. Other times, the grid relies on coal or gas.

• The Analogy: Think of the power grid like the ocean. Sometimes the waves (renewable energy) are huge and perfect for surfing. Sometimes the water is flat.
• The Strategy: ATLAS wants to schedule its heavy computing jobs when the "waves" are high (when renewable energy is abundant). They are also looking at slowing down the CPU slightly. Just like driving a car at a steady, moderate speed uses less fuel than speeding up and braking, running computers at a slightly lower speed can actually be more efficient per unit of work done.

5. Building Better: The "Green House" Concept

When building new data centers, ATLAS is thinking about the "embodied carbon"—the pollution created just by making the building and the computers.

• Cooling: Computers get hot. Usually, we use fans to blow air to cool them, which is noisy and energy-hungry. ATLAS is switching to liquid cooling (like a car radiator). Water is much better at carrying heat away than air. This allows them to run the computers hotter (which is fine for the chips) and use less energy to cool them down.
• Waste Heat Reuse: Instead of letting the heat from the computers escape into the sky, they are piping it into nearby buildings to heat offices or warm water. It's like using the heat from your oven to warm your kitchen in winter.

6. The Big Picture: Why This Matters

The paper concludes that the future of physics depends on being green.

• The "Jevons Paradox": Usually, when technology gets more efficient, we just use more of it. ATLAS is fighting this by ensuring that every efficiency gain translates to a real reduction in their total environmental impact, not just a faster way to do the same amount of work.
• The Goal: By 2040, ATLAS will be doing 10 times more computing. Without these changes, the environmental cost would be astronomical. With these changes, they hope to double their scientific output while keeping their carbon footprint manageable.

Summary

ATLAS is realizing that to understand the universe, they must also protect it. They are moving from a "plug it in and hope for the best" approach to a smart, sustainable ecosystem. They are teaching their scientists to be efficient, their computers to be smart about timing, and their data centers to be green. It's a massive, global effort to ensure that the quest for knowledge doesn't cost the Earth its future.

Technical Summary

1. Problem Statement

The ATLAS experiment at the Large Hadron Collider (LHC) relies on a massive, globally distributed computing infrastructure (the Worldwide LHC Computing Grid, or WLCG) involving nearly one million CPU cores and over 106 TB of managed data. With the upcoming High-Luminosity LHC (HL-LHC) upgrade, computing demands are projected to increase by a factor of 3–4 by the 2030s and by an order of magnitude by the 2040s.

This exponential growth poses a significant risk of increasing the experiment's carbon footprint unless mitigated. The challenge is multifaceted:

• Scope 2 Emissions: Operational emissions from electricity consumption vary wildly across ~100 global sites due to different grid carbon intensities and Power Usage Effectiveness (PUE).
• Scope 3 Emissions: Embodied carbon (manufacturing, transport, and disposal of hardware) is becoming increasingly significant as grids decarbonize, yet it is often overlooked.
• Inefficiency: Waste occurs through failed jobs, unused data, suboptimal hardware configurations, and inefficient cooling.
• Lack of Awareness: Users and site administrators often lack visibility into the environmental cost of their computational choices.

2. Methodology

The paper employs a multi-pronged approach combining policy analysis, technical benchmarking, and operational data modeling to assess and mitigate environmental impact.

• Carbon Accounting Framework: The study utilizes the Greenhouse Gas (GHG) Protocol, focusing on Scope 2 (location-based electricity emissions) and Scope 3 (embodied carbon of hardware and construction). It uses kgCO2eq as the standard unit.
• Benchmarking & Power Modeling:
  • HEPScore: The ATLAS-specific benchmark suite is used to measure CPU throughput.
  • Power Correlation: Power consumption is measured against HEPScore to determine "score per Watt." The study analyzes the impact of CPU frequency governors (performance, ondemand, powersave) and load scaling.
  • Data-Driven Estimation: The PanDA workflow management system is integrated with Electricity Maps to estimate real-time grid carbon intensity (kgCO2eq/kWh) for every job.
• Scenario Modeling: The authors model future scenarios (e.g., 2030 and 2035) for grid decarbonization (specifically in Germany and the US) to evaluate the efficacy of temporal shifting of workloads.
• Cost-Benefit Analysis: Mathematical models are used to calculate the "break-even" point for strategies like data reproduction vs. storage, and the amortization period for new data center construction based on PUE improvements.

3. Key Contributions and Strategies

The paper categorizes mitigation strategies into four groups:

A. Improving Awareness and Education

• PanDA Carbon Reporting: A new feature in the PanDA workflow system provides users with an estimated carbon footprint for their jobs. It uses a global average grid intensity to incentivize code optimization (faster code = lower footprint) rather than encouraging job migration to "greener" sites (which could cause network congestion and idle resources elsewhere).
• Failure Analysis: Educational programs target "failing jobs" (which waste resources). The system now tracks failure rates and limits retries to prevent massive waste from infinite loops of failed submissions.

B. Experiment Computing Policies

• Data Compression & Formats: Adopting the new ROOT RNTuple format and optimizing compression algorithms can save 15–20% of storage space.
• Data Carousel: Archiving infrequently used data to tape (lower operational and embodied carbon per TB) rather than keeping it on disk.
• On-Demand Reproduction: Instead of retaining "just-in-case" derived analysis objects (DAODs) on disk, the system can reproduce them on demand. The study calculates a break-even point of 16%: if data is accessed less than 16% of the time over a year, reproducing it is more environmentally friendly than storing it.
• Automatic Waste Reduction: Tools like HammerCloud automatically exclude problematic sites to prevent job failures. "Scout" jobs run before full tasks to detect configuration errors early.

C. ATLAS-Specific Site Actions

• Frequency Scaling: Reducing CPU frequency (underclocking) significantly improves "HS23 score per Watt" because ATLAS workloads are often memory-bound rather than CPU-bound. Running CPUs below design frequency yields better energy efficiency.
• Checkpointing: Implementing software checkpointing allows sites to pause jobs during grid brownouts or maintenance, preventing the need to restart long-running tasks (12–24 hours) and wasting energy.
• Memory Allocation: Adjusting queue configurations to allow 1GB memory jobs to run on 2GB cores (instead of wasting a core) reduces idle power consumption.

D. General Site Actions & Infrastructure

• Storage Optimization: Shifting background tasks (deduplication, scrubbing) to times of low grid carbon intensity can reduce storage operational emissions by up to 82.4% in high-carbon grids (e.g., CAISO) and 96.4% in future 2035 grids.
• Cooling Systems: Transitioning from air cooling to liquid cooling is recommended. It allows for higher operating temperatures (reducing fan energy) and enables waste heat reuse for district heating.
• Data Center Construction: The paper provides a formula to calculate the amortization time for building new, efficient data centers. In high-carbon regions (e.g., California), a PUE improvement from 1.2 to 1.1 can pay for the embodied carbon of construction in ~6 years.

4. Results

• Energy Efficiency: Reducing CPU frequency improves throughput-per-Watt. For example, running at lower frequencies can yield a 15% energy saving without compromising data throughput.
• Storage Impact: Temporal shifting of background storage tasks can reduce carbon emissions by 60–80% depending on the grid and constraints.
• Data Reproduction: The 16% break-even threshold provides a clear policy guideline: storing data for more than a year is only efficient if the access rate exceeds 16%.
• Cooling & Heat Reuse: A pilot project at Brookhaven National Laboratory (liquid cooling + heat reuse) is projected to save 700 MWh/year and 150,000 kgCO2eq/year.
• Failure Rates: Systematic improvements have reduced the wall-clock time lost to failures in non-user jobs from 10% to 6%.

5. Significance

This paper represents a comprehensive shift in how High Energy Physics (HEP) computing approaches sustainability. Its significance lies in:

1. Holistic Modeling: It moves beyond simple "power consumption" metrics to include embodied carbon (Scope 3) and grid dynamics, acknowledging that as grids decarbonize, hardware efficiency and lifecycle management become paramount.
2. Actionable Policy: It provides concrete, quantifiable policies (e.g., the 16% data retention rule, frequency scaling guidelines) that can be immediately implemented by site administrators and users.
3. Scalability: The strategies outlined are not unique to ATLAS but are applicable to the broader WLCG and general High-Performance Computing (HPC) sectors, offering a blueprint for reducing the environmental impact of large-scale scientific computing.
4. Future-Proofing: By addressing the massive resource growth expected for the HL-LHC, the paper ensures that the physics program can expand without a proportional explosion in carbon emissions, aligning scientific progress with global climate goals.