Modal's gang scheduler places multi-node GPU training jobs from a shared serverless pool

A conventional scheduler can place jobs one machine at a time. Distributed training cannot begin until every node it asked for is placed together ^[15]. Modal's four-node example therefore needs 32 B300s free at the same moment ^[1]. According to Modal, the scheduler also keeps the gang inside one availability zone and network ^[5]. That shrinks the set of free nodes that qualify for any one request.

Clusters draw on the same capacity as Modal's other workloads, according to the company ^[6]. I think sharing one pool is the right design for utilisation. It is what lets Modal bill by the second with no cluster held between jobs ^[7]. When the pool runs short, Modal says the scheduler adds capacity ^[5]. Co-founder and CEO Erik Bernhardsson said in a 2022 account of starting Modal that he wanted software that could take code on a developer's computer and launch it in the cloud within a second ^[14]. For a gang job, launch time also includes finding every node.

RuntimeWire describes the bet as making distributed workloads feel like ordinary cloud functions without removing the operational demands of keeping a cluster reliable ^[18]. Modal did not publish start times by cluster size, or say what happens to a running job when one node in the gang fails.

The two GLM 4.7 timings ^[9] check out as division. A 717 GB payload is 5,736 gigabits. At 50 Gbps that takes about 115 seconds, the "nearly two minutes" in the example ^[2]. Finishing in under two seconds needs at least 2.9 Tbps of effective throughput, about 45% of the 6.4 Tbps per node that Modal advertises ^[3]. RDMA matters when GPUs repeatedly synchronize weights during training or transfer large caches during inference ^[17]. For a gap that size to show up in a team's own job, its wall-clock time has to be dominated by node-to-node transfers of that kind.

I'd rate the gVisor work as the best engineering in the release. Direct access to networking hardware improves performance, while a provider sharing machines among customers still needs isolation between them ^[16]. Modal extended its sandbox to carry RDMA traffic ^[10], so the fast path runs inside the same secure, multi-tenant containers as the rest of the platform ^[19]. The scheduler is only one piece of the stack: the RDMA setup is configured for PyTorch and NCCL, and Clusters work with Modal's Volumes, Cloud Bucket Mounts and Queues ^[4].

Among its customer examples, Modal says Decagon fine-tuned open models of up to one trillion parameters with the Miles framework ^[13]. The company raised $355 million in May at a $4.65 billion post-money valuation, in a round led by General Catalyst and Redpoint Ventures ^[11]. It reported annualized revenue above $300 million ^[12].