AI training models take as much energy as a trans-American flight
Bender et al’ 21 [Emily, Professor, Linguistics at University of Washington, Timnit Gebru founder and executive director of the Distributed Artificial Intelligence Research Institute (DAIR), Angelina McMillan-Major, Senior computer specialist at University of Washington, Shmargaret Shmitchell, computer scientist who works on algorithmic bias and fairness in machine learning. March 01 2021, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? “ https://dl.acm.org/doi/abs/10.1145/3442188.3445922]
Strubell et al. recently benchmarked model training and development costs in terms of dollars and estimated 2 emissions [129]. While the average human is responsible for an estimated 5t 2 per year,2 the authors trained a Transformer (big) model [136] with neural architecture search and estimated that the training procedure emitted 284t of 2 . Training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to require as much energy as a trans-American flight. While some of this energy comes from renewable sources, or cloud compute companies’ use of carbon credit-offset sources, the authors note that the majority of cloud compute providers’ energy is Not sourced from renewable sources and many energy sources in the world are not carbon neutral. In addition, renewable energy sources are still costly to the environment, 3 and data centers with increasing computation requirements take away from other potential uses of green energy,4 underscoring the need for energy efficient model architectures and training paradigms.