Sluice puts a doorman in front of vLLM, because vLLM cannot tell who paid

A developer has published the design notes for Sluice, an admission and shaping proxy that sits in front of self-hosted vLLM and decides, per request, which tenants get through untouched, which get trimmed, and which get shed ^[1][2]. The argument underneath the code is the part worth arguing with: on a GPU pool of fixed size, a tier table in a contract is a promise that no component in the usual stack is positioned to keep ^[3].

The failure mode is specific. Once KV-cache capacity comes under pressure, arrival order and backend scheduling decide who suffers, and every tenant, whatever they were sold, gets the same degraded latency, queueing, or failure ^[4]. Nothing in the default stack says preserve Guaranteed, do less work for Standard, and drop Best-Effort first ^[5].

Sluice's inputs are three live signals per decision cycle: KV-cache pressure and waiting-request depth scraped from vLLM via Prometheus, plus an SLA violation rate computed from completed requests in a PostgreSQL decision ledger ^[6]. Tenants map to tiers in a YAML file, with the example mapping a support bot to Guaranteed, a coding assistant to Standard, and batch analytics to Best-Effort ^[7].

The layer-boundary reasoning is the strongest section, and it is mostly an argument by elimination. According to the post, Kubernetes can place pods, restart replicas, and autoscale, none of which helps when GPU capacity is fixed or slow to provision, and it has no concept of a tenant or a tier at the request level because it schedules pods, not chat completions ^[8]. vLLM's scheduler is conceded to be the right layer for token-level execution, batching, and KV-cache management, and Sluice deliberately stays outside that loop ^[9]. Envoy or Kong give you authentication, routing, retries, and rate limiting, but the author's claim is that a rate limit cannot express "degrade this tier's max_tokens before rejecting that tier's requests, based on live GPU pressure" ^[10].

What I like is the restraint in the surface area. Only POST, PUT, and PATCH requests to /v1/chat/completions or /v1/completions are eligible for shaping at all, and everything else passes through untouched ^[11]. Even then, exactly two fields are ever rewritten: max_tokens, and model for fallback routing ^[12]. The routing stage carries its intent in its own docstring, "QoS route between pools, never utilization/load balance", with Guaranteed and Standard on on-demand capacity and Best-Effort preferring spot unless spot pressure or a health failure forces eviction ^[13][14]. That is a service policy stated as one, rather than a load balancer pretending to be one.

Two things to watch. The SLA targets are hard-coded in sluice_proxy/app.py as Guaranteed 2000, Standard 1000, Best-Effort 500, while tenant-to-tier mapping lives in config ^[15]. That is a split that will bite the first time an account manager negotiates a number, and the post gives no unit for those values or the direction they run in, so a reader cannot tell whether 500 is a tighter budget than 2000 or a looser one ^[16]. Guaranteed's figure is twice Standard's and four times Best-Effort's, which is a suspiciously tidy ratio for a production target ^[17]. Second, the post is written by the project's own author and contains no measurements of what Sluice does to tail latency under contention ^[18]. The design reasoning stands on its own; the effect does not yet.