What we measure before a GPU reaches a customer
One harness, many gates
A GPU fleet does not fail at one moment in its life. It fails at intake, after a firmware push, on a routine reboot, and slowly across months of load. Validation that happens once cannot catch that. So Lilac runs one validation harness, with one report schema, at every point where a node's state can change:
| Gate | When it runs | What it settles |
|---|---|---|
| Image CI | Before an image is released | The build boots and validates on reference hardware |
| Intake burn-in | When hardware arrives | The node is worth racking |
| Acceptance | Before billing starts | The node meets the envelope for its SKU |
| Preflight | Every boot | The node is still the node we accepted |
| Selfcheck | After every firmware or image update | The change broke nothing |
| Health sweep | On a schedule, for life | The node is not drifting |
Same checks, same thresholds, same output. Every run is keyed by image digest, SKU, firmware matrix, and hardware serials. That is what makes fleet-wide questions answerable in one query: which nodes ran this image, which SKU regressed after this firmware, which serial has failed this check before. A harness built for a 10,000-node fleet has to produce records a database can answer for, not logs a human has to read.
"It enumerated" is not proof
The lowest bar in GPU operations is the driver listing the expected number of devices. Clearing it proves almost nothing. A node can enumerate eight healthy-looking GPUs while one of them trains its links at half width, carries a growing count of retired memory pages, throttles a few minutes into sustained load, or intermittently stalls collectives.
So the harness measures behavior, not presence. Link bandwidth at rated speed, measured. Device diagnostics at full depth, not the quick pass. NCCL all-reduce compared against the reference envelope for the SKU, because one slow node in a ring drags the whole ring. Sustained clocks under load, because idle clocks flatter everyone. Every node earns its way in covers how these checks anchor the acceptance gate.
Rebuild in minutes, or the records are wrong
Nodes boot immutable A/B images. An update lands in the inactive slot, the node reboots into it, and rollback is a reboot back into the previous slot. Identity, registration, and evidence live outside the image.
The consequence: rebuilding a node from its registered state is a routine operation with a measurable duration, and we treat that duration as a platform metric. It should be minutes. If a rebuild takes an afternoon of hand-assembled state, the registered state was incomplete, and the day you find out will be the worst available day. Rebuilding continuously, as part of normal operations, is how you find out on a calm Tuesday instead.
Drift becomes a scheduling block, not a 2am page
Fleets drift. A vendor tech updates firmware during a service call. A BIOS setting changes on one node in a rack. A replacement NIC arrives carrying different firmware than the card it replaced. A device drops off the bus and nobody notices because nothing was scheduled there that hour. In most fleets, drift is discovered by a customer's job, at night.
Continuous validation inverts that. Preflight and health sweeps compare each node's observed state, meaning firmware versions, BIOS settings, device set, and link topology, against its registered state. A mismatch does not wake anyone. It marks the node unschedulable, automatically. New work stops landing on it, the divergence is written to the node's record, and the fix runs through the same idempotent, audited convergence workflow that built the node in the first place. When the node validates clean, the block lifts.
Real hardware failures will still page a human. Drift should not. The harness does not make hardware reliable. It makes hardware honest: which check, which node, which image, which firmware, on the record, every time.
← All news