A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

In the rapidly evolving world of artificial intelligence, a new generation of computer programs has emerged that can understand and generate human language with startling fluency. These systems, often called foundation models, are built by feeding vast amounts of text into complex mathematical structures known as neural networks. The prevailing idea in this field has been that bigger is better: the more data a model sees and the larger its internal structure, the smarter it becomes. This drive for scale has led to models with billions of parameters, capable of performing tasks without specific training for them, simply by recognizing patterns in the text they have consumed. However, this race for size comes with a hidden cost. The massive computing power required to train these models consumes enormous amounts of electricity, and the environmental impact of that energy use has become a pressing concern. As the world grapples with climate change, understanding the true carbon footprint of these digital giants is no longer just a technical detail, but a necessary part of evaluating their value to society.

A team of researchers from the Technology Innovation Institute in Abu Dhabi and LightOn in Paris decided to look beyond the usual numbers. Instead of just counting the electricity used to train a single model, they conducted a complete, end-to-end accounting of a project called Noor. This project aimed to build the largest language models ever created for the Arabic language, with sizes ranging from 1.5 billion to 13 billion parameters. The researchers wanted to know the total environmental price of bringing these models into existence, from the very first step of gathering data to the moment the models might be used by people in the future. They tracked every source of energy consumption, including the storage of massive datasets, the preliminary experiments needed to get the design right, the main training process, and even the travel required for the international team to collaborate. Their goal was to paint a full picture of the carbon emissions generated by such an ambitious undertaking.

The journey began with data. To teach the models the Arabic language, the team had to assemble a custom collection of text containing 150 billion words. This involved downloading huge amounts of web data, filtering out low-quality content, and organizing it for the computers to read. Storing and moving these terabytes of information required significant energy, even before the actual training started. The researchers calculated that the energy used just to store and transfer this data, along with the preliminary work of cleaning and preparing it, accounted for a substantial portion of the project's total energy use. They found that the effort to get the data ready was not a minor side task but a major contributor to the overall environmental cost.

Once the data was ready, the team moved to the training phase, where the computers learned to predict the next word in a sentence. This is the most energy-intensive part of the process. The team trained four different models of increasing size on a powerful supercomputer. They discovered that the carbon emissions from this training depended heavily on where the computers were located. When they used a data center in Luxembourg, which runs on very clean electricity, the emissions were surprisingly low. However, when they used their own high-performance computing cluster in the United Arab Emirates, the emissions were much higher because the local electricity mix included more carbon-intensive sources. The study showed that the location of the computing work is just as important as the amount of work being done. In total, the training of the four models consumed the most energy, but it was not the only factor.

The researchers also looked at the human element of the project. The team consisted of scientists working in both France and the United Arab Emirates, who needed to travel between the two countries to hold workshops and brainstorming sessions. They calculated the carbon emissions from these flights and found that they made up a significant chunk of the total footprint. In fact, the travel alone accounted for nearly one-fifth of the project's total carbon emissions. This finding challenged the common assumption that the only thing that matters is the computer time. The study highlighted that in a world where data centers are becoming more efficient and electricity is getting cleaner, the emissions from human travel and other indirect costs could become the dominant part of a project's environmental impact.

Looking ahead, the team also estimated what might happen when these models are actually used by people. They calculated the energy required for the models to generate text in response to user prompts. They found that if these models are used widely, the energy consumed during this daily use could eventually surpass the energy used to train them in the first place. This is a crucial insight because the training happens once, but the use can happen millions of times. The researchers noted that if the models are deployed on computers powered by clean energy, the impact would be low, but if they run on grids powered by fossil fuels, the environmental cost could be very high.

The final tally for the Noor project was 36.5 tons of carbon dioxide equivalent. To put this in perspective, the average person in the United States generates about 20 tons of carbon emissions per year, meaning this entire research project produced the equivalent of a little over two years of emissions for one person. While this might seem small compared to industrial pollution, it is significant for a single research effort. The study concluded that the biggest driver of this footprint was the carbon intensity of the electricity used for training. By choosing to run their computations in a location with cleaner energy, the team estimated they could have reduced the total footprint by more than half.

The paper does not suggest that we should stop building large language models, but it argues that we must be more thoughtful about how we build them. The researchers recommend that future projects should systematically track all sources of emissions, not just the training time. They suggest that choosing data centers with clean energy, minimizing unnecessary travel, and using more efficient computer hardware can drastically lower the environmental cost. They also emphasize that as the technology improves and the training process becomes more efficient, the focus must shift to the other costs, such as travel and the long-term use of the models. By taking a holistic view, the scientific community can continue to advance artificial intelligence while keeping a careful eye on its impact on the planet.