Every node earns its way in

A rack of new GPUs is not a cloud

New GPU hardware arrives broken more often than anyone admits. Not visibly broken. It racks, it cables, it powers on, it enumerates. Then three weeks into a customer's run a link flaps, a GPU falls off the bus, and a job spanning hundreds of nodes restarts from its last checkpoint.

Each new generation of silicon ships denser, hotter, and closer to its physical limits. In the field that shows up two ways. The first is infant mortality: a node that survives its first weeks under sustained load will usually run for years, but a real fraction of new nodes do not survive those weeks. The second is silent degradation: a node keeps reporting healthy while it quietly gets worse.

The failure modes are familiar to anyone who has run a large fleet:

• Xid errors that only appear under sustained load
• ECC page retirement creeping upward on one HBM stack
• NVLink and PCIe links that flap, retrain, and settle at reduced speed
• Thermal throttling from a bad heatsink mount or a miscalibrated fan curve

None of these stop a node from passing a casual smoke test. Any of them will ruin a tenant's month. Years of managing large GPU clusters in other people's data centers taught us one lesson above the rest: hardware lies.

The acceptance gate

We treat acceptance as a gate, not a checklist. Before a node enters the fleet it passes, in order:

1. Inventory match. Every GPU, DIMM, NIC, and drive matches the purchase manifest and the registered configuration, serial by serial. Wrong part, wrong slot, wrong count: rejected before anything else runs.
2. Firmware uniformity. BMC, BIOS, GPU VBIOS, NIC, and retimer firmware converge to the fleet's pinned matrix. A fleet with mixed firmware is a fleet you cannot reason about.
3. GPU diagnostics. Full device-level diagnostics on every GPU. A driver handshake is not a diagnostic.
4. Fabric at rated speed. NVLink and PCIe bandwidth measured, not assumed. A link that trains at half width passes enumeration and fails this gate.
5. Collectives against a reference. The node runs NCCL all-reduce and the results are compared to the known-good envelope for its SKU. Slow is a failure, even when nothing errors.
6. Burn-in. Hours of sustained mixed load across compute, memory, fabric, and storage. Temperatures, clocks, power draw, ECC counts, and link retrains are retained as a time series, not sampled once at the end.

The time series is the point. A single healthy reading proves the node was fine for one instant. The curve shows whether it was drifting toward a failure it had not reached yet.

Billing starts when the gate opens

The rule is simple. A node starts billing reserved GPU hours when it passes acceptance, and not an hour before. Every node earns its way in.

A node that fails goes back to the vendor with the diagnosis attached: which gate, which counter, which time window, which serials. That turns an RMA dispute into a document review. A hot spare takes its place through the same gate, under the same rules. The whole workflow is idempotent and audited, so a node that fails at step four can retry from step four, and the record shows every attempt.

Evidence both sides can rely on

Acceptance produces an artifact, not a green checkmark. The report captures the image digest the node booted, the firmware matrix it converged to, the hardware inventory, every diagnostic result, and the full burn-in time series. It is signed and stored in the same evidence ledger that later backs SLA reporting.

Both sides use it. The fleet owner holds vendors to their warranties with it. The customer knows the reserved capacity they are paying for started from a proven baseline, and can verify when. When capacity is tight, the temptation is to skip the gate and start the meter. The platform does not allow it. A node cannot be marked billable without the evidence attached.

A rack of new GPUs is not a cloud. It becomes one node by node, and every node earns its way in.