💻 computer science

Architectures and physical limits of fast and sustainable artificial intelligence

This critical integrative review establishes a decision framework for sustainable AI by analyzing the interplay between semiconductor devices, memory hierarchies, and data center infrastructure to identify physical limits where increasing arithmetic throughput fails to improve latency or reduce environmental impact, emphasizing that true acceleration requires minimizing data movement and embodied carbon rather than maximizing raw compute power.

Original authors: Baldo Alberto Luigi Dalporto, Sabine Mary, Santiago Gallur²

Published 2026-10-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Baldo Alberto Luigi Dalporto, Sabine Mary, Santiago Gallur²

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a factory where the speed of production is not determined by how fast the machines can assemble a product, but by how quickly raw materials can be delivered to the assembly line. In the world of artificial intelligence, the "machines" are the powerful computer chips that perform complex calculations, and the "raw materials" are the vast amounts of data they must process. For years, the industry has focused on building faster machines, assuming that more calculation power would automatically mean faster and more efficient AI. However, a new analysis suggests that this approach is often like trying to speed up a factory by adding more assembly robots while ignoring the slow, clogged conveyor belts that feed them. The real bottleneck is often not the calculation itself, but the movement of data between memory and the processor, the heat generated by these movements, and the energy required to keep the entire system running.

This critical review, authored by researchers at the Instituto Tecnológico de Santo Domingo, challenges the common belief that simply increasing the raw speed of AI chips is the solution to making artificial intelligence faster and more sustainable. Instead of building new hardware or running new experiments, the authors assembled a comprehensive framework to understand how different layers of an AI system—from the microscopic physics of the transistors to the large-scale cooling systems of data centers—interact. They examined how different types of work, such as training a new model versus using an existing one to answer questions, place entirely different demands on the system. Their goal was to create a clear decision-making guide for engineers and institutions to determine when adding more power actually helps, and when it merely adds cost and environmental burden without improving performance.

The researchers found that the limiting factor for AI speed changes depending on the specific task. When a computer is learning from a massive dataset, it needs enormous amounts of memory to hold the information it is studying. When it is answering a question in real-time, it often gets stuck waiting for data to arrive rather than waiting for the calculation to finish. The study highlights that a processor can be incredibly efficient at performing math, yet still waste energy if it spends most of its time idle, waiting for data to arrive from a distant memory bank. This phenomenon means that doubling the speed of a chip does not necessarily halve the time it takes to complete a job. If the data cannot move fast enough to keep the chip busy, the extra speed is useless. The authors demonstrate that for many common tasks, the speed of the connection between the memory and the processor is far more important than the raw calculation speed of the processor itself.

A significant portion of the analysis focuses on the physical limits of the hardware. The researchers explain that as chips get smaller, they face fundamental laws of physics that make them harder to cool and more prone to energy leaks. They point out that the energy required to move a single bit of information is often much higher than the energy required to calculate with it. This means that the most energy-efficient AI system is not necessarily the one with the most powerful chips, but the one that moves the least amount of data. The study also introduces the concept of "embodied carbon," which accounts for the pollution generated during the manufacturing of the hardware. They argue that replacing old equipment with new, faster chips is not always environmentally friendly if the new chips do not save enough energy during their use to pay back the carbon cost of making them. There is a specific point where the energy savings from a new chip finally outweigh the pollution created to build it, and the authors provide a way to calculate when that point is reached.

The paper also scrutinizes the role of software and system design. It explains that the way a program is written and how it organizes its data can have a bigger impact on speed than the hardware it runs on. For instance, a clever software technique can reduce the amount of data that needs to be moved, effectively making a slower chip perform as well as a faster one. The authors emphasize that simply stacking more chips together does not guarantee better performance if the network connecting them cannot handle the traffic. They show that in large systems, the time spent waiting for different parts of the system to synchronize can become the biggest delay, rendering the extra computing power ineffective.

In a specific example using the specifications of a well-known high-performance chip, the researchers calculated the theoretical limits of its performance. They found that for certain types of tasks, the chip's speed is capped not by how fast it can calculate, but by how fast it can read its own memory. Even if the calculation speed were doubled, the time to complete the task would remain almost the same because the data delivery rate would still be the bottleneck. This finding serves as a powerful reminder that hardware improvements must be matched by improvements in data movement and system architecture. The study concludes that true progress in artificial intelligence requires a holistic view that balances calculation speed, memory capacity, data transfer rates, and environmental impact. It suggests that the path to faster and greener AI lies not in chasing higher numbers on a spec sheet, but in designing systems that minimize unnecessary movement and ensure that every unit of energy contributes to useful work.

Technical Summary: Architectures and Physical Limits of Fast and Sustainable Artificial Intelligence

Problem Statement
The paper addresses a critical disconnect in artificial intelligence (AI) deployment: high peak arithmetic throughput does not guarantee low response time or environmental efficiency. AI services often suffer from delays caused by data waiting in memory, remote communication dependencies, or queuing, while hardware may operate with poor energy efficiency due to low utilization or unnecessary data movement. The central problem is identifying which resource—arithmetic, memory capacity, bandwidth, or communication—actually limits useful AI computation in specific workload contexts, and determining under what conditions hardware improvements translate into tangible performance or sustainability gains.

Methodology
This work is a critical integrative review and analytical tutorial, not a systematic review or meta-analysis. It relies on a literature-based synthesis of established analytical relationships, cited author results, published specification ceilings, and new analytical deductions.

Scope: The analysis focuses on physically implemented classical AI systems, particularly deep learning and transformers. It explicitly excludes exhaustive catalogs of all processors, experimental benchmarks, prototype measurements, or field data.
Approach: The authors connect layers ranging from transistor physics and semiconductor scaling to memory hierarchies, interconnects, compilers, and data center infrastructure.
Evidence: Sources include peer-reviewed papers, official manufacturer specifications (e.g., NVIDIA H100, PCIe standards), and consortium documents. The paper distinguishes between "established analytical relationships" and "deductions made here."
Analytical Framework: The core methodology involves deriving conditional mathematical expressions (crossover conditions) to determine feasibility bounds. A worked example uses published H100 SXM specifications strictly as idealized hardware ceilings to demonstrate analytical derivations, explicitly stating these are not measured application performances.

Key Contributions
The paper proposes a decision framework that integrates performance bounds with feasibility constraints and lifecycle accounting. Its primary contributions are:

Workload-Dependent Resource Identification: It distinguishes between training, prompt processing (prefill), and autoregressive decoding, showing how each phase stresses different resources (e.g., memory bandwidth vs. arithmetic intensity).
Crossover Conditions: The authors derive mathematical conditions identifying:
- When increasing arithmetic throughput yields zero latency improvement (because the system is memory or communication bound).
- When batching encounters a memory capacity limit.
- When hardware replacement fails to recover its embodied carbon within a specific horizon.
Hierarchical Performance Modeling: It extends the "Roofline" model to include capacity gates and exposed communication costs, treating memory bandwidth, interconnect latency, and thermal constraints as coupled but distinct limits.
Sustainability Accounting: It separates operational energy from embodied carbon and water usage, providing a break-even analysis for hardware replacement decisions based on lifecycle emissions rather than just runtime energy.

Results and Analytical Findings
The paper presents several analytical results derived from stated assumptions:

Arithmetic vs. Bandwidth: For a dense decoder, the ideal batch size crossover ( $b^*$ ) is derived as $b^* = p_w C / 2B$ (where $p_w$ is bytes per weight, $C$ is compute capacity, and $B$ is bandwidth). If the batch size is below this threshold, increasing compute power does not reduce latency; the system remains bandwidth-bound.
The "Zero Speedup" Phenomenon: The paper demonstrates algebraically that if a system is limited by a non-arithmetic resource (e.g., memory bandwidth), doubling the arithmetic capacity ( $k=2$ ) results in a speedup of exactly 1.0.
Memory Capacity Constraints: A model is presented showing that if the required memory for weights, KV cache, and activations exceeds usable capacity ( $M_{usable}$ ), the system must partition, compress, or offload, fundamentally altering the performance model.
Interconnect Latency: A decomposition of message latency ( $T_{msg}$ ) into setup, queueing, serialization, and propagation terms reveals that for small message sizes, reducing fixed latency is more effective than increasing bandwidth.
Lifecycle Break-even: A simplified formula ( $N_{break} = \Delta C_{emb} / (\Delta E \cdot \gamma)$ ) determines the number of useful tasks required to offset the embodied carbon of new hardware. If the expected task volume is lower than this break-even point, replacement increases total environmental burden.
Utilization Efficiency: An analytical curve shows that while consolidation improves energy-per-task, excessive utilization can violate tail-latency constraints, and idle power penalties significantly degrade efficiency at low utilization.

Significance and Claims
The paper modestly claims its significance lies in the organization and coupling of existing concepts (Roofline analysis, queueing theory, lifecycle assessment) into a traceable, cross-layer decision sequence.

No New Laws: The authors explicitly state they are not proposing new performance laws or discovering new physical limits. The mathematical results are conditional deductions from stated assumptions.
No Prototypes: The study reports no new devices, prototypes, experimental benchmarks, or field measurements. The "worked example" using H100 data is strictly an idealized calculation to illustrate the framework, not a validation of specific hardware performance.
Practical Utility: The framework is designed to help hyperscale operators and universities select infrastructure by identifying the dominant resource bottleneck and evaluating trade-offs between speed, quality, energy, and carbon footprint.
Reproducibility: The paper emphasizes that its analytical derivations can be reproduced using only published specifications and the provided equations, without needing proprietary data or new experiments.

In conclusion, the paper argues that sustainable acceleration requires minimizing useful work and transferred bytes while maintaining task quality and feasible power density, rather than simply chasing higher peak arithmetic throughput.

More like this