NVIDIA's NVCRE runs the NCCL test that Slurm gets from a single srun command

Three custom resources carry the design. You create a Certification that names the target nodes and the categories to run. It creates one Workflow per category, and each Workflow creates one Job ^[11].

The Workflow applies catalog, platform and GPU overrides, manages iteration count, and sets the orchestration target. The Job runs the workload on its node group, monitors node health, and records measurements and failures ^[10]. Results then travel back up: the Job records which nodes failed and why, the Workflow reports the category result, and the Certification groups results by category ^[11]. A degraded fabric link comes back as a node name and a reason code.

This got built because of a gap in Kubernetes. On Slurm the readiness test is one srun command, and Kubernetes has no built-in equivalent ^[8]. The same test there needs GPU and RDMA resource requests, NCCL settings matched to the network fabric, a large enough shared-memory volume, and a way to make all the pods start together ^[9]. Platform teams have been writing that themselves, usually shell scripts wrapped around NCCL tests, with a runbook or spreadsheet tracking which nodes cleared which stage ^[7].

The sample in the post is an example of the report format. That run fails communication/nccl-all-reduce after 42 minutes 18 seconds at full scale, eight nodes per job, with gpu-node-07 flagged ThresholdViolation and gpu-node-12 flagged HardwareFailureDetected ^[13]. Two of eight is 25 percent of the nodes in that job ^[18]. For the runtime to mean anything on your hardware you would need the same GPUs per node, the same fabric and the same iteration count, and NVCRE derives the first and third of those itself: it detects GPU architecture and cloud platform from the target nodes, then sets GPUs per node, the NCCL environment and platform networking accordingly ^[15]. The failure the post opens with is a 512-GPU training job ^[1], and the sample it prints covers eight nodes.

Pass and fail criteria are Common Expression Language expressions evaluated against the measurements ^[16]. Those expressions define what "ready" means, and the bar moves by stage: bring-up, burn-in, preproduction and production each set a different one, and a node that passes a smoke test is not necessarily ready to join a 512-GPU run ^[6]. The post does not print the threshold values.

The catalog covers five NCCL communication variants, the DCGM level-4 diagnostic suite, and NeMo pretraining with Nemotron 5 at 8B and 56B parameters, which is eight entries across three domains ^[14][19]. Because every resource is a CRD, the whole progression is inspectable with kubectl and drivable from GitOps ^[12]. NVIDIA's post says "Readiness becomes a proven property of the cluster rather than an assumption." On the published evidence it is a measured property of the nodes you paid to test, for the categories you chose, and the example manifest is still apiVersion nvcre.nvidia.com/v1alpha1 ^[20][17].