LiteLLM's Lens moves agent-failure analysis into the self-hosted gateway that routes model calls

A trace records one run: the model calls, the tool use and the intermediate steps between them ^[11]. When an agent fails, the cause can sit several steps before the bad output, so a single request-response record often does not show it ^[11]. Lens works across runs. It groups similar problems and links each group back to the traces that produced it ^[6]. Investigations run on demand or on a schedule, and the individual traces stay available in the proxy's logs interface for manual inspection ^[10].

I think the storage design is the right one for a self-hosted product. The step-by-step trace record goes to ClickHouse, and the investigation results that point into it go to PostgreSQL ^[7]. A separate analyzer worker connects to the proxy and runs on the customer's infrastructure ^[7][8]. The analyzer is also a gateway client. It calls a model the customer selects, routed through LiteLLM ^[9]. Every investigation therefore adds model traffic to the proxy ^[7][9].

The rate-limit pitch follows from where the data lives. LiteLLM's stated benefit is direct access to customer-controlled trace data without the rate limits of hosted tracing platforms ^[24]. A coding agent running SQL against that ClickHouse instance is limited by the hardware the customer gave it ^[7][24].

Jaffer said Lens is being built for a future in which agent swarms generate "200K+ traces" ^[4]. The launch image illustrates it with hundreds of agents making thousands of model and tool calls, each run a trace carrying steps, tokens and cost ^[15]. The figure is a design target ^[4]. For it to tell an operator anything, LiteLLM would need to publish an ingest rate into ClickHouse, how many runs the analyzer can investigate per hour, and what one investigation costs in model calls. The documentation does not establish general availability, Lens pricing or measured performance at the announced scale ^[14].

Much of the feature list already exists elsewhere. LangSmith offers trace monitoring with automatic clustering and failure analysis ^[18]. Arize Phoenix and Braintrust combine tracing, monitoring and evaluation ^[20]. LiteLLM's claim to difference is position. Jaffer described the gateway as a chokepoint for enterprise AI traffic ^[3]. Few vendors pick that word for their own product, and Runtimewire treats it as company positioning ^[3]. Runtimewire also notes a commercial incentive in the expansion: the open-source gateway is free to self-host, while the enterprise offering adds governance, security and support ^[21]. LiteLLM is a Y Combinator Winter 2023 company and raised a $1.6 million seed round from Y Combinator, Gravity Fund and Pioneer Fund ^[23].