Build1 publisher2 min readPublished
Ollama's five-minute idle default triggered 214 model reloads in a day
A dev.to writeup instrumented 1,180 local requests over 24 hours and counted 214 cold model loads, including a summarizer on a 10-minute cron that came up cold every time.
The Engineer · Build desk
What happened
- Ollama unloads an idle model after five minutes by default, so the next request re-reads gigabytes of weights from disk and re-allocates VRAM before it generates a token.
- A dev.to author timed every request for 24 hours and logged 1,180 requests against 214 model load events, which he works out as one reload for every 5.5 requests.
- On his box a cold load cost 11.4 seconds to first token against 0.9 seconds warm, and 18.1 percent of requests were cold.
- Pinning two models that do not both fit in VRAM with keep_alive: -1 triggered partial CPU offload and dropped generation from 42 tokens per second to 6.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- constraint Any polling job whose interval is longer than the idle timeout can never be measured warm, so the timeout sets its latency, whatever model it calls.
- decision Teams that standardised on the OpenAI SDK cannot fix residency in application code; the change has to go into server configuration or deployment.
- cost 214 cold loads at 10.5 extra seconds each is about 37 minutes a day of one developer waiting on disk reads, on a single box with one user.
- contradiction The post's own figures do not reconcile: with 81.9 percent of calls warm, the median call sits in the 0.9-second spike, so the 3.1-second p50 must be measuring more than time to first token.
A cold request is three operations before the first token: read gigabytes of weights off disk, allocate VRAM, then start generating [1]. The chat model on the author's box is about 4.9GB, and the cold path took 11.4 seconds [10][4]. That is roughly 430 MB per second end to end [25]. Disk read accounts for only part of the 11.4 seconds, because allocation and runner startup sit inside the same number [1].
The obvious fix made things worse. Pinning both models with `keep_alive: -1` puts `llama3.1:8b` and `qwen2.5-coder:14b` resident together, about 13.9GB of weights on a 16GB card, leaving 2.1GB for context and overhead [20]. Ollama offloaded part of the work to CPU and generation fell from 42 tok/s to 6, a sevenfold loss on every token to avoid one 11-second load [7][21].
What worked was server-side. The author set `OLLAMA_KEEP_ALIVE=24h` as an environment variable on the server, kept one resident model per GPU, and moved embeddings to a separate CPU-only Ollama instance [8]. Load events went from 214 a day to 9, about 96 percent fewer [8][22].
Before that he tried the application-level version, adding `keep_alive` through the OpenAI SDK's `extra_body`, and the next day's load count was still in the 200s [14]. He noticed only because he was still counting loads [14]. He reports the field doing nothing on `/v1/chat/completions` on his version, with the native `/api/chat` endpoint honoring it [6].
Two checks tell you whether this is your problem. `ollama ps` says what is resident now, and the `UNTIL` column is the effective expiry, whatever your config says [15]. Counting loads means grepping the server log for the line that appears exactly once per load, and the author warns that the wording drifts between versions, so read your own log by hand first [16]. The distribution to look for is two spikes with nothing between them, at 0.9s and 11.4s in his case [13]. That is a residency state, not a slow model.
The 11.4 seconds is a measurement of one box: 16GB of VRAM, weights on NVMe, three clients, one of them a note summarizer on a 10-minute cron [9][11]. That job fires 144 times a day, each run following ten minutes of silence against a five-minute timeout, so it was cold on every call [26][12]. For weeks the author read its 12 seconds as the cost of local summarization [12].
Of the default itself, he wrote: "This is a completely reasonable default for a laptop. It is a terrible default for anything with bursty traffic, which is every side project ever built." [17]
What to watch
- Whether a later Ollama release honors keep_alive on /v1/chat/completions; the author reports the field being ignored on his version only.
- Whether the UNTIL column in ollama ps agrees with OLLAMA_KEEP_ALIVE after a server restart on your own version.
- A published cold-load time for larger weights on slower storage would show how much of the 11.4 seconds is disk read and how much is VRAM allocation.