Divide 512 by the token count of a typical request and you have the number of requests the batcher must collect before the GPU is actually busy [22]. The threshold comes from the thread itself: runtime for the smaller embedding models is driven mainly by token count rather than by the number of individual queries, and roughly 512 tokens can be enough to fill the hardware [7][8]. Short requests mean a lot of gathering, and gathering is what Tulip exists to do [6]. A single 512-token document fills the batch on its own, at which point Tulip is scheduling a batch of one across a gRPC boundary [22].
The other half of batching is the wait. Tulip merges small requests until it has enough tokens to use the hardware efficiently, while trying not to pay the latency penalty of waiting too long for a batch to fill [9]. The retrieval design bounds what that wait can cost: Perplexity's published search architecture starts retrieval through both lexical and semantic modalities and merges the results into a hybrid candidate set, so the pipeline is not wholly blocked on the query embedding [14]. Semantic retrieval still sits on a latency-sensitive path for every live search [15]. The same models also serve indexing and scoring, where the workload is large and throughput is the thing that matters [16].
The language boundary is placed where it costs least. Ivy and Tulip are Rust, and the only Python is ROSE, the layer that runs forward passes and manages CUDA graph execution [23][10]. Batch size is adjusted twice on the way down, since Ivy splits requests that arrive too large and Tulip merges the ones that arrive too small [24].
Inside ROSE, the two workload families share more than they differ. The engine reuses many of the same kernels for large language models and embedding models [11]. Divergence is confined to the cache and the attention path: embedding skips the key-value cache used in autoregressive generation and relies on ragged attention instead of paged attention, and the engine can pick an attention backend from model shape and sequence length [12][13]. That confinement is what makes one engine plausible for both, rather than two runtimes with a shared changelog.
The scheduling instinct fits the house. Founding president Andy Konwinski helped create Apache Mesos and Spark before co-founding Databricks [20]. The resourcing fits too: TechCrunch reported in September 2025 that Perplexity had raised another $200 million at a $20 billion valuation, taking reported funding to roughly $1.5 billion [19], which is the budget that makes bringing embedding, ranking, classification, scoring and language model serving in-house a reasonable thing to attempt [17]. The 512-token number is a fact about Perplexity's smaller embedding models and its own traffic mix. On traffic whose requests already saturate a batch, the same three services add a hop and change nothing.
