One unset flag accounts for three orders of magnitude in a LiteLLM gateway benchmark

The harness isolates exactly what it was built to isolate. maximhq/bifrost-benchmarking ships a mocker, a fake LLM provider, so the reported numbers are gateway overhead with real provider latency removed ^[3]. The post reports a p50 gap of 5.8x at 100 RPS, widening to 17.6x at 500 RPS ^[5]. Bifrost's p50 at 500 RPS was 0.75 ms ^[8]. If both figures come from the same 500 RPS runs, LiteLLM's p50 there was about 13.2 ms ^[1]. Whether 13 ms of gateway overhead is visible in your traces depends on what the model call behind it costs. Between 500 and 1,000 RPS LiteLLM dropped 5% of requests and its p99 reached 33.8 seconds, while Bifrost served every request with a p50 under 1 ms ^[6]. Bifrost's p50 went from 0.75 ms at 500 RPS to 0.79 ms at 1,000 RPS, a rise of about 5% while the request rate doubled ^[8][2]. The author says he did not find Bifrost's ceiling, and that 1,000 RPS was the highest rate he ran ^[9]. The tool is maintained by the Bifrost team at Maxim AI, which the author discloses, and he says he did not build his own ^[2][3]. He alternated runs, one gateway then the other, because running all of one gateway's rounds in a block would hand a long-session slowdown to whichever went second ^[10]. Each number is the median of three 20-second runs ^[11], which is 60 seconds of load per data point ^[3]. The hardware holds the absolute figures down, and the author says so ^[14]. It was a shared-CPU VPS: 4 vCPU on an AMD EPYC 9354P, 15.6 GiB of RAM, Ubuntu 26.04.1 LTS, Docker 29.8.0 ^[12]. He set no CPU or memory limits, and the load generator, both gateways and the mocker shared those four cores ^[13]. Maxim's own published overhead figures are 59 microseconds on a t3.medium and 11 microseconds on a t3.xlarge at 5,000 sustained RPS ^[15], a 5.4x spread between two instance sizes inside one vendor's table ^[4]. The author states his numbers are not directly comparable to those ^[16]. The official LiteLLM image does not set --num_workers, and the CLI default is 1 ^[17]. The first comparison was therefore one Python process against a Go server with all four cores available, and it produced a ratio in the tens of thousands ^[18]. He reran with --num_workers 4 and confirmed four multiprocessing-fork children inside the container ^[19], which moved the result by three orders of magnitude ^[20]. Anyone running LiteLLM in production who has never set --num_workers, he wrote, should go check that before reading any further ^[21]. The post does not state which worker count produced the 5.8x and 17.6x table. Both gateways ship with authentication for inference disabled ^[22]. In Bifrost, the field is EnforceAuthOnInference in framework/configstore/tables/clientconfig.go, declared with gorm:"default:false" and pinned there by a test ^[23]. In LiteLLM, master_key is None unless it is set under general_settings in the config file or LITELLM_MASTER_KEY is in the environment, and the config file wins ^[24]. A gateway holds the provider keys you configure, so until that line is flipped, anything that can reach the port can spend them ^[25]. For the ratios to transfer, three things have to hold on your side: the gateway does not share cores with your load generator, your LiteLLM runs more than one worker, and gateway overhead is a real share of your p99. The 500 RPS cliff is a property of four shared cores as much as of Python; more cores move that rate up. The post's verdict is Bifrost for enterprise use, and stay on LiteLLM if your team extends it in Python or runs far below saturation ^[1].