AI data center cooling system with blue-lit liquid cooling manifolds and flexible hoses running into a dense GPU server rack
|

AI Cooling Is the Next Bottleneck: Why Data Centers Can’t Keep Up With AI Hardware

ByImran Valiani

As AI GPUs become faster and more power-hungry, the biggest challenge is no longer compute — it’s cooling. Here’s why liquid cooling, thermal management, and next-generation data center design are becoming essential for the future of artificial intelligence.

AI data center cooling has quietly become the hardest engineering problem in the industry — harder, in a lot of ways, than building the chips themselves. NVIDIA’s Blackwell B200 GPU draws up to 1,000 watts per chip in its HGX B200 baseboard configuration. A single GB200 NVL72 rack — 72 of these GPUs wired together — pulls around 120 kilowatts, per NVIDIA’s own rack documentation. The newer Blackwell Ultra generation pushes further still: NVIDIA’s own enterprise reference architecture specifies a full GB300 NVL72 rack as requiring up to 142 kilowatts.

That’s roughly the continuous electrical draw of 100 average U.S. homes — my own estimate, based on EIA’s reported 2024 average household consumption of about 865 kWh per month, or roughly 1.2 kW of continuous draw per home. In one rack.

Nearly all electrical power consumed by the rack ultimately becomes heat that must be removed from the equipment and rejected by the facility. And that’s the real story here. It’s also the same story reshaping how utilities plan for AI’s electricity demand in the first place — a demand curve steep enough that it’s reviving interest in nuclear power as a baseload source for data centers.

Why AI Data Center Cooling Fell Behind Compute

I’ve been following data center infrastructure for years now. To be honest, the pace of the last two generations still catches me off guard.

Here’s the thing about chip design: transistor counts, parallelism, memory bandwidth, and package-level power density have scaled rapidly. Cooling has not scaled at the same rate. It can’t — physics doesn’t care about your roadmap. It’s the same underlying tension I’ve written about before when it comes to AI’s broader energy wall and why brain-inspired chip architectures are getting a second look — the industry keeps solving the compute problem and running straight into the power and thermal problem behind it.

The GPU power trajectory tells the story:

  • H100 (Hopper): up to 700W TDP (SXM configuration)
  • B200 (Blackwell): up to 1,000W in the HGX B200 baseboard configuration
  • B300 (Blackwell Ultra): up to 1,400W per GPU in the rack-scale, mandatory-liquid-cooled GB300 NVL72 configuration — NVIDIA’s own materials describe this as the Blackwell Ultra maximum power envelope.

The form-factor distinction matters more than it sounds: the same GPU family carries a different power envelope depending on which system it ships in, and mixing those numbers up is exactly how a facilities team ends up under-sizing a cooling loop.

That represents a doubling of the maximum GPU power envelope from H100 SXM to Blackwell Ultra. Per Introl’s infrastructure analysis, this progression is exactly what organizations must now plan facilities around — not the other way around.

Conventional air cooling cannot handle the densest rack-scale AI systems on its own. Uptime Institute’s 2024 survey found no single industry-wide cutoff where air cooling stops working — responses were spread across a range of thresholds, with the largest single group (29% of respondents) placing the point where liquid cooling becomes necessary at 20-29 kilowatts per rack. That survey range illustrates how far current 120-142kW rack designs have moved beyond conventional rack densities.

Air Cooling Hits a Wall

Traditional data centers use CRAC or CRAH units — big fans pushing chilled air through a raised floor, across server racks, back out again. It’s worked fine for decades.

Not anymore. At least not on its own — not for rack-scale systems operating above 100 kilowatts.

Liquid beats air on heat transfer by a wide margin, but the exact number depends on which property you’re measuring. Water’s thermal conductivity is about 23 times air’s (0.58 vs. 0.026 W/(m·K)). Its volumetric heat capacity — how much heat a given volume of it can absorb — is around 3,450 times greater than air’s at room temperature. That shorthand you’ll see in marketing copy — “liquid moves heat 3,000x better than air” — comes from that volumetric heat capacity figure specifically, not from a claim that a complete liquid-cooling system performs 3,000 times better overall; actual system performance also depends on flow rate, pumping power, and heat-exchanger design. Worth knowing the difference if you’re the one specifying the cooling loop.

Why air struggles at these densities:

  • Fans need enormous airflow volume to remove kilowatts of heat, which means noise, floor space, and energy overhead
  • Hot spots form inside the rack itself, especially with tightly packed GPUs
  • Cooling capacity doesn’t scale linearly — double the heat load, and you don’t just need double the airflow, you often need a fundamentally different approach.

Talk to anyone running colocation facilities today, and you’ll hear the same thing: H100-density racks already made operators nervous. Blackwell-density racks are a different conversation entirely.

Liquid Cooling Isn’t Optional Anymore

So the industry is shifting. Fast. This is the shift that defines modern AI data center cooling: from air as the default to liquid as the requirement.

As of Uptime Institute’s 2024 Cooling Systems Survey, about 22% of respondents said their organizations were making some use of direct liquid cooling. Nearly half of those — 48% — reported that fewer than 10% of their racks actually used it, mixed into an otherwise air-cooled facility. But that’s changing as AI workloads multiply.

Two main liquid cooling approaches dominate the conversation:

Direct-to-Chip (Cold Plate) Cooling

Cold plates sit directly on the GPU and CPU. Coolant runs through channels in the plate, pulls heat off the chip, and carries it to a Coolant Distribution Unit (CDU) — the piece of hardware that manages the secondary loop connecting the rack-level coolant to the facility’s chilled water or refrigerant system. The board these components sit on matters too — the same signal-integrity pressures pushing AI servers toward denser, hotter designs are also driving the shift to advanced high-speed PCB laminates like Megtron 6, Megtron 8, and Tachyon 100G.

This is the most common approach for retrofitting existing facilities. Also the least disruptive to rack design — you’re not submerging anything.

Immersion Cooling

Full submersion. Servers sit in a tank of dielectric fluid — engineered specifically to be electrically non-conductive — and the fluid absorbs heat directly from every component, not just the GPU. Immersion comes in two flavors: single-phase, where the fluid stays liquid throughout, and two-phase, where the fluid boils off the hot components and recondenses, carrying away far more heat per pass but adding real complexity around vapor containment and fluid handling.

Microsoft’s own engineering team has documented a production two-phase immersion deployment, built with partner Wiwynn, running in the company’s Quincy, Washington datacenter. Microsoft’s internal testing found that two-phase immersion cooling reduced power consumption for any given server by 5% to 15%.

The Water Problem Nobody Talks About Enough

Quick tangent. Bear with me.

Cooling doesn’t just cost electricity — it costs water, and that’s a second front in the AI data center cooling problem that gets a lot less attention than the power story. Traditional evaporative cooling towers consume enormous volumes of water to reject heat. As AI data centers cluster in water-stressed regions, that’s becoming its own controversy, separate from the power grid conversation.

Microsoft’s own reported figures put a hard number on this: the company projects that its newer closed-loop, chip-level cooling architecture — distinct from immersion cooling — can avoid more than 125 million liters of water use annually at each data center where it’s deployed, by eliminating the need for potable water in the cooling process during normal operation. That’s a projected design comparison against Microsoft’s older evaporative-cooling facilities, not a claim that every existing Microsoft data center already consumes that much. Microsoft began incorporating the design into new builds starting in 2024 and reported in January 2026 that it was already deployed at sites including Wisconsin and Georgia.

European regulations are pushing in the same direction. Under the EU’s Energy Efficiency Directive (2023/1791), specifically Article 26(6), data centers with total rated power above 1 megawatt must assess the feasibility of recovering their waste heat — and implement recovery unless a cost-benefit analysis demonstrates it’s technically or economically unfeasible. It’s a feasibility mandate with an escape hatch, not a flat requirement, but it’s pushing facility design in the same direction as the hyperscalers’ own sustainability targets.

What This Means for AI’s Future

Here’s where it gets uncomfortable for anyone planning AI infrastructure. Compute isn’t the bottleneck anymore.

Power and cooling are.

A few practical implications, if you’re trying to actually understand this landscape:

  • Facility retrofits are expensive and slow. You can’t just drop a 120-142kW rack into a building designed for 5-10kW racks. The electrical distribution, cooling loops, and facility plumbing will generally require substantial upgrades.
  • Cooling isn’t the only high-stakes physical constraint — power delivery is too. These racks run on low-voltage DC busbars, and NVIDIA’s own engineering disclosures for the GB200 NVL72 describe a purpose-built “enhanced high-capacity busbar” rated for 1,400 amps — double the amperage of the previous rack-power standard it replaced. That’s a genuine electrical engineering problem sitting right alongside the thermal one, not a footnote to it.
  • Grid capacity is becoming a real constraint. Large AI deployments can be delayed by utility interconnection queues, switchgear availability, cooling-plant construction, network installation, and system commissioning. How long that takes varies widely by site — there’s no defensible universal timeline between hardware delivery and full operation, though a well-documented 2024 case involving a Blackwell production delay illustrates how facility and power readiness, not just chip availability, can hold up a deployment.
  • Location decisions are shifting. Power availability and water strategy are becoming major site-selection factors alongside fiber connectivity and land cost.
  • Liquid cooling maturity is still catching up to demand. Even as adoption accelerates, Uptime Institute’s research continues to flag reliability concerns, maintenance complexity, and coolant leaks as real barriers operators report — not solved problems. Uptime’s reporting has also noted that achieving high mission-critical resiliency remains more complex and costly with direct liquid cooling than with conventional air-cooled designs, which helps explain why adoption outside AI training clusters remains slower.
  • Standardization is coming. AI training is acting as a stress test for cooling technology, and lessons learned here will likely filter into conventional enterprise IT cooling over time — though that enterprise adoption curve looks slower than AI’s.

Where This Leaves Beginners Trying to Understand AI Infrastructure

If you’re new to this space, here’s the simplest way to think about it. For years, the AI story was “who has the best chips.” That story is incomplete now.

The real question is closer to: who can actually power and cool those chips at scale?

A chip that can’t be cooled properly gets throttled. Throttled chips run slower, which defeats the purpose of buying the fastest hardware on the market in the first place. So cooling isn’t a background engineering detail — it’s directly tied to how much useful AI compute actually reaches the world. It’s worth noting this is very much a data-center-scale problem — on the other end of the spectrum, edge AI chips are solving a completely different set of constraints by keeping inference small and local instead of dense and centralized.

To be honest, this part of the industry doesn’t get nearly as much attention as new model releases or benchmark scores. But among the people actually building this infrastructure, it’s increasingly treated as the real constraint on how fast AI capability can grow over the next few years. Not algorithms. Not chip design. Thermal physics.

That’s a strange place for the AI industry to find itself — bottlenecked by something as unglamorous as how well you can move heat out of a room. But that’s exactly where AI data center cooling sits right now: not a footnote, but the constraint everything else has to design around.

About the Author

Imran Valiani | Sales Director, PCB Electronics Manufacturing

20+ years working with major Bay Area and global tech clients. Founder of Silicon to Software, where I write about the hardware layer — PCB fab, AI gear, autonomous systems, and cyber — the stuff most tech writers have never touched. Literally.

Follow: X @SiToSoftware | LinkedIn

This post was written with AI assistance. See my full AI disclosure.

Sources

  • NVIDIA — “NVIDIA Contributes NVIDIA GB200 NVL72 Designs to Open Compute Project,” Oct 2024 (GB200 NVL72 rack power, busbar spec): https://developer.nvidia.com/blog/nvidia-contributes-nvidia-gb200-nvl72-designs-to-open-compute-project/
  • NVIDIA Enterprise Reference Architecture documentation (GB300 NVL72 rack power): https://docs.nvidia.com/enterprise-reference-architectures/nvl72-ai-factory/latest/components.html
  • NVIDIA H100 product page: https://www.nvidia.com/en-us/data-center/h100/
  • NVIDIA HGX B200 PCF Summary: https://images.nvidia.com/aem-dam/Solutions/documents/HGX-B200-PCF-Summary.pdf
  • NVIDIA Blackwell Ultra technical blog: https://developer.nvidia.com/blog/inside-nvidia-blackwell-ultra-the-chip-powering-the-ai-factory-era/
  • U.S. Energy Information Administration, 2024 residential electricity data: https://www.eia.gov/todayinenergy/detail.php?id=65244
  • Uptime Institute 2024 Cooling Systems Survey: https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2024.Cooling.Survey.Report.pdf?version=3
  • Uptime Institute Journal, on liquid cooling and mission-critical resiliency: https://journal.uptimeinstitute.com/liquid-cooling-will-not-outgrow-its-high-density-niche/
  • Microsoft News, on two-phase immersion cooling: https://news.microsoft.com/source/features/innovation/datacenter-liquid-cooling/
  • Microsoft, “Building Community-First AI Infrastructure,” Jan 2026: https://blogs.microsoft.com/on-the-issues/2026/01/13/community-first-ai-infrastructure/
  • Microsoft zero-water cooling design announcement, Dec 2024: https://www.microsoft.com/en-us/microsoft-cloud/blog/2024/12/09/sustainable-by-design-next-generation-datacenters-consume-zero-water-for-cooling/
  • EU Energy Efficiency Directive (EU) 2023/1791, Article 26(6): https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32023L1791

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *