Google's vLLM TPU embedding numbers: 83,996 tokens/s, and a 0.999 cosine gate

Divide the two published figures and you get the number that actually sizes a cluster: 83,996 tokens/s against 5.13 requests/s is roughly 16,400 tokens per request ^[6]. That matches the 16K-class sequences in the configuration ^[2][3], and it is why request-rate accounting is the wrong instrument for this workload. Here a request is a document, not a query.

Run that rate against a real corpus. At 5.13 requests/s, one pool in that configuration needs about 226 days of continuous work to embed 100 million long documents at one request each ^[7]. The write-up puts production retrieval at hundreds of millions of chunks, plus images, reindexing jobs, online queries and multiple tenants ^[8]. That arithmetic, not elegance, is why Google's design makes TPU the primary pool and holds GPU capacity as secondary fallback for bursty indexing ^[5].

The more interesting engineering sits in correctness. Chunked prefill lowers peak memory by splitting a long input across steps ^[9], but an embedding model still has to emit one pooled vector covering the whole sequence ^[10]. If pooling state is not accumulated correctly across chunk boundaries, the vector comes out wrong with no obvious failure ^[11]. Google's hybrid StepPool keeps that state across chunk boundaries and request preemption using cached request metadata ^[12]. A wrong vector does not page anyone. It surfaces as slightly worse recall, weeks later, usually blamed on the model.

Two further constraints are hardware-shaped. TPU matrix units impose strict divisibility rules during tensor-parallel sharding, so vocabulary padding was added to keep sharded execution hardware-safe while preserving logical output ^[13][14]. And because TPU serving leans on compilation, the recommended pod lifecycle puts compilation warm-up ahead of the health check, so the first real request is not the one paying the JIT cost ^[15].

Then the parity numbers, which are the part worth arguing about. The stated targets are cosine similarity of at least 0.999 for text and at least 0.995 for multimodal ^[4]. In drift terms that is 0.001 against 0.005, five times the permitted deviation on the multimodal path ^[16]. That is also the path where the current vLLM-TPU design chunks only the text portion of the prefill ^[17]. Looser tolerance and the thinner implementation land on the same workload, which is where the evaluation budget belongs. The dev.to write-up's own advice is to measure vector parity, Recall@K, NDCG and top-K overlap before moving hardware ^[18], and clearing 0.999 on text tells you nothing about the image side.

One caution on the headline throughput: the write-up says plainly that this is a specific benchmark point rather than a universal TPU number ^[19]. Treat it as a single measured configuration until someone reproduces it on their own corpus.