LinkedIn's job-search ranker queries its teacher models live during training

Hundreds of thousands of queries per second is the constraint that picks the model size ^[4]. A 0.6B student is what fits, and that is why the inference figures matter as much as the training ones. Structured pruning and context compression took ranking throughput from about 290 to over 2,000 items per second per GPU, roughly 6.9x ^[15][1]. Items per second is not queries per second. Each query scores a candidate set, and InfoQ's account does not say how many candidates retrieval hands the ranker.

The eightfold figure is a product of separate wins, so multiply them. Multi-node training is credited with up to 3.5x, FSDP2 with a further 20%, and H200 multi-node clusters with up to 30% on top of that ^[9][10][11]. 3.5 x 1.2 x 1.3 is about 5.5x ^[2]. Reaching eight needs another 1.5x, and the candidates on the list are LiGer's 2x larger batch sizes and the 3x that came from serving local teacher replicas across nodes ^[8][6][2]. Every one of those multipliers is reported as an "up to", so 5.5x is a ceiling on the three named wins, not a typical case.

Copy the teacher serving. An asynchronous client queries the SGLang-hosted teachers during training, processes their outputs and adds them to the distillation losses, and the framework loads teachers of different sizes while managing tensor-parallel and data-parallel setups ^[5]. The 3x came specifically from putting teacher replicas on each node ^[6]. So every training node needs GPU memory for teacher weights as well as the student, and if the teachers live elsewhere the async client crosses the network once per training example ^[3].

Offline mode precomputes teacher outputs, writes them to HDFS or NFS, and training reads them instead of querying live ^[7]. Those cached outputs hold only until someone swaps a teacher. InfoQ's guidance is to query online while teacher choices are still moving, then switch to offline caching once the teachers stabilize and query volume rises ^[19].

The quality number needs care. NDCG@10 for job searches went from 0.7583 to 0.9432, reported as a 24.48% gain ^[14]. Dividing the endpoints gives 24.38%, so something rounds differently somewhere in the chain ^[3]. For 0.94 to mean anything on another system you would need the same query mix, which here is natural-language job search for LinkedIn's US users, the same candidate generator, and the same relevance labels ^[16].

Other teams can take the FP8 finding straight off the page. LinkedIn evaluated FP8 mixed precision and found no benefit for models under 8B parameters, because casting overhead outweighed the compute savings ^[13]. InfoQ puts the main contribution on the systems work that makes iteration fast enough, rather than the distillation recipe ^[22], and contrasts it with Pinterest's June 2026 post on near-linear training scalability, which scaled the training framework to build larger teachers and distilled them into students for Homefeed and Related Pins ^[20].