Rohan Pinto

Microsoft's 1-bit BitNet reopens the question of which inference needs a GPU

Rohan Pinto says BitNet b1.58 runs a 100-billion-parameter model at five to seven tokens a second on a single CPU. That is enough for overnight batches and air-gapped servers, and short of what a cloud tier needs.