Scattered pass rates across 93 versions break the cutoff-to-version shortcut

A usable boundary would show up as a slope. Older releases pass, pass rates sag as the tasks approach the estimated line, then everything past it fails. The Microsoft developer blog post says the results should have had a recognizable shape if the model's own version estimates were worth anything, and that instead the author got noise ^[19]. Dev Proxy 0.3.0 passed four of five tested tasks while 0.4.0 passed none ^[13]. Much later, 2.3.4, 3.0.0 and 3.1.0 each passed one of two, followed again by releases with no passing tasks ^[14].

Some of that is sample size. The 336 Dev Proxy tasks spread across 53 versions average about 6.3 per version ^[4], and the 413 SPFx tasks across 40 versions about 10.3 ^[5]. A version with two tasks can only score 0, 50 or 100 per cent. Pooled, the two runs are steadier: 749 tasks, 122 passes, 16.3 per cent ^[1][2][3].

The instructive failures sit nowhere near any boundary. One SPFx 1.11.0 task asked about using a third-party module loader in the local workbench, and the model produced plausible advice about SPFx's module-loading model, AMD, webpack and iframe isolation while missing that SystemJS support had been removed from the local workbench in that release ^[16]. A Dev Proxy 0.5.0 task asked which executable to invoke. The model answered devproxy, when the required answer for that version was mgdp, from the period the product was still named Microsoft Graph Developer Proxy ^[17].

For the 18 per cent to say anything about your product, a lot would have to hold. It is one model under test, driven through the GitHub Copilot SDK, against two Microsoft developer products ^[6]. Documentation and web search were removed ^[10]. The task prompts and rubrics came from other models in the same family, with GPT-5.6 Sol extracting changes from changelogs and GPT-5.6 Terra judging each result ^[7][9], so the score carries the judge's error as well as the model under test's. Retrieval was off by design. So 18 per cent is a floor for an agent that ships docs, and production behaviour is a separate question.

The author notes that even a local product installation can give the agent access to the answer and invalidate the measurement ^[11]. To find out what a model remembers, you have to strip the product off the machine the agent works on. No extension will ever run on that machine.

The hypothesis under test was simple enough to state in two sentences. "The model knows the product up to roughly version X. Anything introduced after that needs additional context," the post's author wrote ^[18]. OpenAI lists Luna's cutoff as February 16, 2026, and Dev Proxy 2.1.0 from January 28 was the latest stable release available by then ^[2][3]. The model put its own limit at Dev Proxy 0.29.0 and SPFx 1.21.1 ^[4], and 0.29.0 shipped on June 25, 2025, almost eight months earlier ^[5]. Conservative in the direction you would want, and still not usable for budgeting, because passing tasks keep appearing well past it ^[12].

A cutoff date is a ceiling on what the training data could have covered ^[1]. I would ship the docs for every supported version and spend the context.