Portability of an eval comes down to which parts of it live where. The guide's own workflow has five components: define the objective, collect the dataset, define the metrics, run and compare, then continuously evaluate [7]. One of those five is described as running on OpenAI's infrastructure, with the worked example telling you to use the Evals API to create and run evals in the OpenAI dashboard [5]. The other four are already yours [17]. If your objective, your held-out set and your grader definitions sit in version control, what you are replacing is a runner.
The part worth auditing is the part the notice does not enumerate. Existing evals content stays available during the transition window [1], and read-only arrives on October 31, 2026 [2], so anything visible only in the dashboard, run history in particular, needs somewhere else to live while writes still work.
Then the arithmetic. Shutdown is November 30, 2026 [3], which is 30 days after the read-only date [13]. Those 30 days are not migration budget for anyone whose release gate creates eval runs. Creating a run is a write, read-only removes writes, so the gate fails on the earlier date [14].
That matters more because of what the same page asks for. Its last workflow step is continuous evaluation: run evals on every change, monitor for new cases of nondeterminism, grow the eval set over time [6]. Advice like that puts the hosted platform in the deploy path rather than beside it. A best-practices guide that carries its own end-of-life notice is at least specific about the dates, and it points to the deprecations page for the current timeline [4].
Now the number. The summarization example asks for ROUGE-L of at least 0.40 and a coherence score of at least 80% under G-Eval, measured on a held-out set of 1000 reference transcripts and summaries [8]. Read that as a threshold for someone else's corpus. For it to transfer, your transcripts need comparable length, your reference summaries need one consistent house style, and your G-Eval judge needs pinning, because a judge fetched over an API moves under your baseline when the model behind it changes. The guide is not naive about this. It tells you to calibrate automated scoring against human feedback [10], and it names eval datasets that do not faithfully reproduce production traffic patterns as an anti-pattern [11]. It also lists what to draw on: synthetic, domain-specific, purchased, human-curated, production and historical data [9].
The advice that survives the shutdown is the cheapest thing on the page. Pairwise comparison, classification, or scoring against explicit criteria beats judging open-ended generation, because models discriminate better than they compose [12]. That holds on whatever runner you move to, and it is the reason the whole exercise exists: the same input can produce different output, which is what makes ordinary software testing insufficient here [15].
In my context, the cheapest order is graders and datasets out of the dashboard first, runner swapped second, baselines re-established last, all of it while the old platform still accepts writes. Teams that only ever drove the API and kept their own scores may find the port is a weekend. Teams that read their release history off the dashboard have a different job.
