Fine-tuning on your own docs bakes the drift in: a parameter that has returned 400 for years

The operator of dailymeteo.com sat down to retrain the fine-tuned GPT model behind its natural-language chat endpoint and found that the training data had been teaching the model to call an API parameter that returns 400 ^[1][2][3]. The training set had been written from the API documentation, and, as the dev.to write-up puts it, documentation is a claim about a system rather than the system ^[4][5].

The service is a daily meteorological archive for Europe, gridded at 1 km, running from 1961 to roughly five days ago, with a chat endpoint that turns a plain-language question into R code against the archive ^[1]. Behind that sits a fine-tuned model trained on a few hundred question-and-code pairs ^[2]. The plan for the day was hyperparameters; the day went instead to measuring what the API actually returns, and three documented claims turned out to be wrong ^[6][7].

The clearest case is `slp`, sea-level pressure. The documentation lists it as available ^[8]. There are 21,916 daily rasters on disk covering 1961 to 2020 ^[9], which is within a day of complete daily coverage for those 60 years ^[10]. The API will not serve any of them: an allow-list of `['tmax','tmin','tmean','prcp']` guards every entry point, and the continent mapping points at a newer data folder that never received the pressure rasters, so there are two independent reasons for the same 400 ^[11][12]. The error message returned on failure still lists `slp` among the valid options, because the message is older than the list ^[13]. Two training examples asked for it, so on any pressure question the model emitted confident, well-formed, non-functional code ^[14][15].

The other failures are the same shape. A request for 1960 returns "There's no data for date range" because the archive begins in 1961, while the recent end moves: daily to about five days ago, monthly to the previous month, annual to the last complete year ^[16][17]. Any fixed end year written into a system prompt is silently wrong within twelve months ^[18], and several training examples hardcoded one ^[19]. The measurement also surfaced a better pattern: out-of-range requests are not errors, they clip, so `ann 1961 -> 2035` came back as 1961 to 2025 and `mon 1961-01 -> 2030-12` came back through 2026-07 ^[20][21]. Generated code can ask wide and read the real extent out of the response instead of guessing ^[22]. Those two clipped values are internally consistent with a measurement taken in August 2026 ^[23].

Two more, both about conventions nobody wrote down. For long-term means the returned timestamp carries a climate period, not a date, so it reads `1961-1990` or `25.07.1991-2020`, and code doing `substr(timestamp, 1, 4)` to pull a year gets nonsense, which some of theirs did ^[24][25]. Long-term means had no description in the docs at all, and across 23 calls in the training set they were invoked seven different ways, two mutually contradictory, so the model learned that no convention exists ^[26][27]. Separately, 28 examples pinned an arbitrary year the question never specified, such as autumn in Prague fetched as 1999-09-01 to 1999-11-30 ^[28][29].

Worth checking on your own stack: whether your error strings and your allow-lists were last edited in the same year, and whether your examples encode extents that your API will happily report back if asked.