Product1 publisher2 min readPublished
A crash-looping plugin hid Atlassian's RDMA fabric from Kubernetes for 271 days
Atlassian's platform team says the device plugin that advertises its fast GPU network had never worked in production, and the training jobs that quietly fell back to TCP sockets kept running with no error at all.
The Product Desk · Product desk
What happened
- Atlassian's training jobs outgrew single machines as models reached the tens of billions of parameters, with one node no longer able to hold the model, its optimizer state and a workable batch size at once.
- The Kubernetes device plugin that advertises the RDMA fabric had been crash-looping on every fabric-capable production node in one of its clouds since deployment, and the team found it 271 days later by accident during an unrelated GPU operator upgrade.
- The cause was a mirrored container image that resolved to a single-architecture manifest not matching the nodes, so the plugin failed at exec, and a stopgap applied to staging months earlier never reached production.
- On current-generation GPU hardware, the team says a socket-based path can leave a multi-node job running at roughly half the speed the same hardware delivers over RDMA.
- Before the RDMA and Lustre work, communication and storage delays showed up as low GPU utilization, uneven step times and runs that took far longer than they should have.
Compiled by The Product DeskSomething wrong?How this is made
Why it matters
- cost A run at roughly half throughput needs about twice the GPU hours, so on affected jobs about half the accelerator time bought no training progress, and the invoice reads as a busy cluster.
- constraint Restart counts and pod age, the two numbers most on-call dashboards show first, cannot separate a plugin that broke last night from one that never worked, once nodes autoscale.
- decision If the platform owns RDMA and Lustre so ML teams never touch them, the platform team also owns proving on every run that the fabric is the path being used.
An ML engineer submits a multi-node training job on the internal platform that sits behind product experiences such as search and ranking [2]. It schedules. It finishes. "Before RDMA and Lustre, distributed jobs ran. How fast they ran was anyone's guess," Atlassian's platform team wrote in a post on the CNCF blog [20].
The reason a dead fabric is hard to see is in how the request works. A job that explicitly asks for the fabric and cannot get it stays pending indefinitely, which the post calls loud and easy to diagnose. A job that does not ask runs over sockets with no error, and the post calls that both the default and the expensive case [10].
So the crash loop had no complainant. 271 days is about nine months of fabric-capable production nodes advertising nothing to the scheduler [16]. "A capability with no consumers is invisible, however expensive it was to buy," the authors wrote [11]. No alert existed for a DaemonSet that had never been ready, and the post says that absence was the real defect [12].
Staging passed. The post describes a healthy environment beside a broken one as a trap, because staging passing told the team nothing about production [19].
What the account does not give is a bill. There is no count of jobs that fell back to sockets across those 271 days, no cost figure, and no benchmark cited for the half-speed comparison, which is the team's own characterisation of current-generation GPU hardware [18].
Here is the grid worth drawing against your own cluster. One axis: does the job request the accelerator fabric, or take the default. The other: is the fabric actually healthy. Three of the four cells are either fine or fail in a way an on-call engineer will see within an hour. The fourth cell, default request on broken fabric, runs to completion at reduced speed and reports success. A pre-flight check that fails the run when collective traffic is not on the fabric moves that cell into the loud column, and it is the only intervention in the post's account that touches the default path.
That check is the difference between owning GPUs and running a platform. Atlassian's argument is that once training spans more than one node, inter-node communication, shared storage, placement, topology and validation stop being things a platform can treat as second-order [1]. Its own framing of the standard: "For distributed AI, performance is not just optimization. It is part of correctness." [4]
What to watch
- Whether Atlassian publishes the pre-flight validation gate itself, including what condition fails a run that is not using the fabric.
- Whether Kubernetes GPU operator or device plugin distributions ship a default alert for a DaemonSet that has never reached ready.
- Whether an independent benchmark on current-generation GPU hardware supports the roughly two-to-one socket-versus-RDMA gap.