Why AI Uses So Much Electricity
1. Quick Summary
Every interaction with a large model involves multiplying enormous matrices of numbers. Each word generated requires a pass through billions or hundreds of billions of parameters, and each multiplication consumes a small amount of energy. Multiply that by billions of requests a day and the total becomes a serious load on the grid.
Training is a different order of magnitude again. A single large training run keeps tens of thousands of accelerators at high utilisation for weeks or months, which is comparable to the continuous draw of a small town over the same period.
2. What It Means
The computation itself is dominated by matrix multiplication. A neural network layer takes a vector of numbers, multiplies it by a matrix of learned weights, and adds a bias. Doing that for billions of parameters per token means billions of multiply-accumulate operations per generated word, and modern chips perform them at extraordinary rates, which is exactly why they draw hundreds of watts each.
Chips are only part of the bill. A data centre also powers storage, networking between racks, power conversion losses and, significantly, cooling. Facility efficiency is usually reported as a ratio of total facility power to computing power, and well-run centres now operate at ratios not far above one, meaning overhead is a modest fraction rather than the dominant term.
Inference, the stage that serves actual requests, is where the ongoing cost sits. Training is a one-off capital expense per model; inference recurs with every user, and for a widely deployed system the cumulative energy of serving can exceed the energy of the original training within a surprisingly short period.
3. Why It Happens
Scale is the direct cause. Model capability has tracked increases in parameters and training data, and both translate into more arithmetic. A model an order of magnitude larger requires roughly an order of magnitude more computation per token at the same efficiency.
The hardware is power hungry because it is built for throughput. Accelerators pack enormous numbers of arithmetic units onto a die and run them in parallel, which is what makes training feasible at all, and the price of that density is heat that has to be removed continuously.
Utilisation matters as much as capacity. A chip sitting idle still draws power, and large training runs are sensitive to communication delays between nodes, so a system that cannot keep its accelerators busy burns energy without producing useful work. Much of the engineering effort goes into keeping utilisation high.
4. Real Examples
The comparison people find most intuitive is against search. A conventional web query retrieves indexed pages and costs a small fraction of a watt-hour, while generating a long model answer runs billions of operations and costs substantially more per interaction, even after all the efficiency work of recent years.
Water use gets attention for a related reason. Many large facilities use evaporative cooling, which consumes water in addition to electricity, and in dry regions that has become a local planning issue independent of the carbon question.
Siting decisions follow the physics. New facilities cluster near cheap and reliable power, and recent announcements include long-term agreements for dedicated generation, including nuclear, because a multi-decade power contract is the only way to guarantee supply at that scale.
5. How It Affects Us
Efficiency work is the main lever, and it is moving quickly. Smaller models that distil a larger one’s behaviour, quantised arithmetic that uses fewer bits per number, and mixture-of-experts designs that activate only part of the network per token all cut energy per request substantially.
There is a rebound effect to be honest about. When inference becomes cheaper, it gets used more, and in practice efficiency gains have historically expanded total consumption rather than reduced it.
For users the practical version is simple: the cost is real but it is not evenly distributed. Long answers, large models and repeated retries each multiply energy use, and the difference between a well-scoped query and a sprawling one is larger than most people would guess.
6. Key Takeaways
- The cost comes from billions of multiplications per token, multiplied across very large numbers of requests.
- Training is a large one-off cost; inference recurs with every query and can overtake it over time.
- Efficient designs such as distillation, quantisation and sparse activation cut energy per request substantially.
- Cheaper inference tends to increase total use, so efficiency alone does not guarantee lower consumption.