Zalando's durable agentic engineering win was a proxy, not a model

Zalando's engineering blog has published a look back at 2.5 years of LLM work across more than 250 engineering teams, and the thing it credits first is not a model or a coding agent ^[1][2]. It is a LiteLLM-based API proxy that the company's ML platform team deployed in January 2024 to give engineers API access to models from multiple providers, now OpenAI, AWS Bedrock and Google Vertex ^[3].

The stated payoff is unglamorous and compounding: one team got a single point at which to measure adoption by monthly and weekly active users, by model, and by User-Agent ^[4]. Everything else in the post hangs off that. Post-call hooks do anonymized cost tracking ^[5]. Pre-call hooks enforce client version upgrades by restricting proxy access based on the User-Agent header ^[5]. The post is blunt about why that mechanism exists: for self-managed client installations, blocking access is the only effective measure, and the same applies to retiring models, because there is always a long-tail group of users who do not adjust local configurations or follow new model releases ^[6]. A proxy is not just a router. It is the only place where refusal is possible.

The same chokepoint absorbs mistakes made by users. Zalando enabled auto-injection of prompt caching checkpoints, which the team says reduced costs for custom agents while their authors were still learning about prompt caching ^[7]. The operational numbers are modest. To work around LiteLLM stability and memory leak issues, the team forces restarts after 20,000 requests via `--max_requests_before_restart`, and says this lets it serve 2,000 monthly active users on six small pods specced at 2k CPU and 4 GB ^[8][9]. That is roughly 333 monthly active users per pod ^[10]. The team says it is looking forward to LiteLLM's expected Rust rewrite for performance and stability ^[11].

Zalando has been a GitHub Copilot user since the days when it offered IDE autocomplete, and describes the proxy as complementary to that ^[12]. The in-house surface area on top is deliberately cheap: a chat UI forked from a now-unmaintained open source codebase, and a CLI built with pydantic-ai ^[13]. The chat UI still sees high adoption, which the authors call a surprise given how many more capable IDE plugins and CLIs now exist ^[14]. The CLI began at an August 2024 hackathon, before coding agents existed, which puts it about seven months after the proxy went live ^[15][16]. It has since accumulated agent mode with MCP support and automatic Bearer token injection for internally hosted MCP servers, an HTTP-to-stdio MCP proxy, and a command that installs safe configurations for Claude Code, opencode and pi ^[17]. The point of the token injection and MCP proxy, per the post, is that no secrets need to be hardcoded in configuration files, which matters as the user base spreads beyond engineering to people with varying intuition for security ^[18].

Two frictions are worth tracking, because they are industry-wide rather than local. Tools too often send a generic User-Agent, which defeats client identification at the proxy; Zalando's own agents carry a name, repository and version, and for third-party tools the team requests or contributes upstream changes ^[19]. And tools lack support for custom auth commands for token generation, supporting only static credentials ^[20]. Until vendors fix both, the metering and enforcement story above stays partly blind.