Serving RAG retrieval as a cacheable GET lets an edge proxy absorb repeat lookups

An HTTP cache builds its key from the request method, the path, the query parameters and a few selected headers ^[11]. It treats POST as unsafe and non-idempotent and forwards it straight to the origin. Many RAG frameworks send retrieval as a POST with a JSON body ^[12]. So the first change is to the API. The write-up moves query, top_k and threshold into the query string of a GET, and a shared proxy can then store the response ^[12].

The author, writing on dev.to, rules out the in-app options first. With 12 API pods, an in-process lru_cache becomes 12 separate caches with low hit rates. A rolling deploy empties all 12 at once and sends the full load back to the vector database ^[8]. Redis fixes the fragmentation. It also adds a stateful service to patch and scale, plus hand-written key serialization and invalidation that has to keep up with document updates ^[9].

The header in the example is `Cache-Control: public, s-maxage=300` ^[13]. With a 300-second shared-cache lifetime, one key can go back to the origin at most 288 times a day on a given cache node ^[1]. For the twenty phrases in the anecdote, that caps origin retrievals at 5,760 a day per node, however many users forget their password ^[2]. A phrase asked less often than once every five minutes at a node misses every time ^[6].

The route is `/v1/tenants/{tenant_id}/retrieve` ^[13]. I think putting the tenant in the path is the right design. The proxy keys on the path, so an entry cannot cross tenants the way a hand-built cache key can when someone leaves out the organization ID ^[10][11]. The cost is hit rate. The same question from two tenants makes two entries ^[3]. A single-tenant bot pays nothing for that. The purge tag `kb:{tenant_id}` is tenant-wide as well, so one edited document clears every cached retrieval that tenant holds ^[4].

The 38% is a share of lookups ^[2]. The post does not give the invoice amount, its split between query fees and storage, or a hit rate after the change. A hit skips embedding, vector search and chunk fetch, 110 to 280 ms in the post's own figures, along with the embedding charge and the vector query fee ^[6]. For that to reach the invoice, query fees have to be a large part of the bill. Questions also have to cluster the way the author says support and documentation traffic does ^[15].

Generation is left alone. The streaming chat endpoint stays a POST ^[14], and the model still writes a fresh response for every conversation ^[4]. For chat traffic to see the savings, the chat service's own retrieval call has to go out through the gateway as the cacheable GET. A direct call inside the cluster never touches the cache ^[5].