Mixtral

The sparse-model bill arrives at serving time, and it is paid in collectives

A CUDA and ROCm tutorial ends on the unglamorous half of Mixture of Experts: two all-to-all collectives per layer, optimizer state pushed onto PCIe, and launch overhead you have to design away.