Ivy

Perplexity fills its embedding GPUs by counting tokens rather than requests

Perplexity's 4 September thread splits request preparation, batch scheduling and CUDA execution across two languages so each can change alone, with a 512-token fill threshold deciding when its batcher is worth the hop.