Learnings: Designing and Building AI-Ready Data Centers

It is tempting to size an AI network the same way you size any other data center network: pick the fastest link speed available and move on. That approach misses what actually happens during a large training run.

A distributed job spends much of its time with accelerators talking to each other, not to the outside world. That east-west traffic between compute nodes needs predictable, low-latency delivery across the whole fabric — not just a high number printed on a switch datasheet.

Topology choices, how much oversubscription you accept, and how congestion is handled all shape whether a cluster actually reaches the throughput its hardware is capable of.

Average network performance is a comfortable number to report, and a poor guide to how a distributed job will actually behave.

Synchronous training jobs move at the pace of their slowest participant — one degraded GPU, one noisy link, or one storage path with intermittent retransmissions can hold back an entire job, even while every other metric looks fine.

That is why infrastructure teams need to watch the tail of the distribution, not just the mean: p95 and p99 latency, jitter, drop and retransmission rates, and how long those conditions persist under sustained load.

AI hardware moves on a shorter cycle than most enterprise infrastructure. A fabric that comfortably serves today’s cluster can become the binding constraint the moment a newer accelerator generation arrives with substantially higher per-node bandwidth needs.

Planning for that shift means leaving real headroom rather than sizing exactly to today’s requirement: switch fabrics that can be upgraded rather than replaced, a realistic view of where Ethernet speeds are heading, optics that can be swapped in without re-cabling, and spine capacity with room to grow.

None of this means picking a single bandwidth number and calling it the requirement — the right figure depends on model size, cluster architecture, accelerator generation, and the specific workload, and it will keep changing.

An AI data center produces telemetry from many layers at once — GPUs, CPUs, memory, switches, links, storage, power, cooling, and the applications and training jobs running on top of all of it.

Collecting that data is the easy part. The value shows up when a team can correlate what is happening at the infrastructure layer with what is happening to the workload — for example, noticing that a drop in GPU utilization coincides with rising network congestion, which is slowing collective communication and stretching out each training iteration.

Treating observability as something bolted on after deployment usually means every layer reports its own numbers in its own dashboard, with no way to connect cause and effect when something slows down.

Past a certain cluster size, purely manual operations stop scaling. Rerouting around congestion, balancing workloads, spotting a failing link, adjusting cooling policy, and opening an incident — these need to happen faster and more consistently than a human operator can manage across a large environment.

The pattern that works is a closed loop: observe conditions, analyze what they mean, decide on a response, act on it, and validate that the action actually helped before moving on.

Automating a response is not the same as giving up control over it. The actions with real consequences — isolating infrastructure, enforcing policy changes, scaling resources — need clear policies, guardrails, an audit trail, and a way to roll back if the automation gets it wrong.

AI platforms introduce security boundaries that a conventional enterprise environment doesn’t have to think about in the same way — models, training data, accelerator clusters, the APIs sitting in front of them, and the storage and management networks underneath.

It works better to design those boundaries into the architecture from the start than to add them once the platform is already running: workload identity, service-to-service authentication, network segmentation, encryption in transit, tightly scoped secrets management, and least-privilege access as the default.

East-west traffic inside the cluster deserves the same scrutiny as traffic crossing the perimeter — that includes watching for signs of data exfiltration, separating administrative networks from everything else, keeping an audit log of who touched what, and controlling access to models and datasets specifically, not just to the servers they sit on.

Accelerated racks draw meaningfully more power than the general-purpose racks most facilities were originally built around. That difference needs to be part of the plan before hardware is ordered, not discovered afterward.

There is no single figure that applies everywhere — actual rack power depends on the accelerator generation in use, how densely servers are packed, the cooling architecture behind them, and how the cluster is laid out. What matters is planning in ranges: conventional enterprise racks sit at one end, GPU-accelerated racks draw substantially more on a sustained basis, and large-scale AI systems push density further still.

That planning has to cover redundant power feeds, PDU design, UPS and generator capacity, rack-level monitoring, and enough headroom to absorb transient spikes — worked out at the cluster level, not rack by rack in isolation.

Cooling is often the thing that actually limits how far an AI deployment can grow, more than floor space or even power supply. As rack density climbs, conventional air cooling can become difficult to sustain, and exactly where that becomes true depends heavily on the facility itself.

There is a range of approaches available, each suited to a different density: standard air cooling, hot-aisle/cold-aisle containment, rear-door heat exchangers, direct-to-chip liquid cooling with coolant distribution units, facility water loops, and immersion cooling at the far end.

It is not useful to quote a fixed kW-per-rack number where air cooling "stops working" — the real limit depends on airflow design, rack layout, ambient conditions, and how the facility itself was engineered.

AI infrastructure can draw a substantial amount of power, which makes efficiency a financial question as much as an operational one.

Efficiency is worth measuring across the whole system, not just at the accelerator: power usage effectiveness, how much of that GPU capacity is actually being used versus sitting idle, cooling efficiency, how workloads get scheduled, and where compute is simply being wasted.

Where it is practical, renewable-energy integration, heat reuse, and energy-aware scheduling all contribute — the point in each case is getting more useful work out of the energy already being consumed, not just reducing the number on the utility bill.

AI hardware, networking standards, and workload patterns all move faster than the refresh cycle most data centers were designed around. Building the entire environment for one accelerator generation, or one model architecture, tends to age badly.

Modular rack designs, compute platforms that can be swapped rather than gutted, network fabrics with room to scale, more than one viable cooling option, and software-defined infrastructure all make it possible to expand or change course without a full redesign.

Ownership model matters here too, and there isn’t one right answer: owning hardware outright, managed infrastructure, colocation, cloud GPU capacity, consumption-based models, and hybrid combinations of these all trade off differently on cost, control, and flexibility depending on the organization.