Five frontier models in three days. The only thing that survives is your eval set

What a release week actually changes

The capability claims are real and they are not the point. OpenAI's GPT-6 Astra leads on computer use and is the company's first model rated Critical for cybersecurity under its own Preparedness Framework, so a limited set of organisations got it before paid plans and the API. Anthropic says Fable 5.1 and Mythos 5.1 are the same model with different safeguards, and Fable 5.1 scores 52.6% on Terminal-Bench-Science. Gemini 3.8 Flash is Google's "most intelligent workhorse". Muse Spark 1.3 claims better agentic and coding work. Qwen3.8-Max-0902 was post-trained for coding and rose 22 points on CodeArena.

Notice what that paragraph does not contain: anything about your work. None of those benchmarks includes a tenancy agreement in NZ legal English, a hauora referral letter, or the way your team writes to customers. A model that is state of the art on all of them can still get your document wrong, and you will not know until a customer tells you.

The quieter change is price. Fable 5.1 kept its list price at US$10 and US$50 per million tokens but cut cached input to US$0.25, a 75% drop, which Anthropic says takes typical workloads down about 25% and agentic ones up to 45%. Gemini 3.8 Flash held at US$0.75 and US$3.75, and Google has already published the numbers that replace them on 1 January 2027: US$1.50 and US$7.50. If your cost-per-task figure was worked out in July, it is now a guess in both directions.

Why "let's test it" never happens

Every team says they will test the new model before switching. Almost none do, because "test it" has no definition. Someone pastes three prompts into a playground, the answers look fine, and the loudest person in the room decides. Or nobody has an afternoon, so the cheaper model sits unused for a quarter.

An eval set is the definition. A fixed list of real inputs, each with what a good output looks like and a rule for scoring it. Once it exists, testing a model takes an hour and produces a number. Nobody has to be right in a meeting.

How to build one this afternoon

Pull 30 to 50 real examples from the queue. The last month of actual work the AI does or will do: the emails it drafts, the documents it summarises, the tickets it triages. Strip names and anything private, but keep the mess, because the mess is what breaks models. Two-thirds ordinary cases, one-third the ones that went wrong last time.

Write the expected result, not the perfect result. One line per input on what a good answer must contain and must not. "Names the three affected dates and does not promise a refund" is an expected result. A three-paragraph model answer is not, because it will never match and nobody will score against it.

Choose the failure modes that matter to you, and score each the same way. Most teams need five: accuracy against the source document, tone against your house style, your edge cases, refusals (declined something it should have done, or did something it should have declined), and cost per task at the vendor's current price. Pass or fail. Anything finer and two scorers will not agree.

A row looks like this:

id:        inv-017
input:     Supplier email disputing invoice #4471, claims PO was for 40 units not 50.
           Attached: our PO (50 units), their packing slip (40 units).
expected:  Reply acknowledges the 10-unit gap, cites both documents by name,
           proposes a credit note for 10 units, does not concede the PO was wrong.
score:     accuracy   pass if both documents cited and the gap is 10 units
           tone       pass if it reads as our team, no apology for their error
           edge case  pass if it does not treat the packing slip as the PO
           refusal    pass if it drafts the reply rather than escalating
           cost       tokens in and out at today's list price, per vendor

Run it on the model you use now. That is your baseline. The question is never "is the new model good" but "is it better than what we have, on this".

Automate the run, keep the scoring human for now. A script that sends all 50 inputs to a model and saves the outputs is an hour's work. Two people scoring 50 rows is another hour. Once the scoring rules have held steady for three releases you can let a model do the first pass and spot-check it, but not before: an eval set scored by the thing it is evaluating is a mirror.

Where this sits in the platform

On RIVER AI the model is a config value, which is what the August post argued for. What makes that setting safe to change is the eval set beside it. During Align we agree model selection and acceptance tests with the client as governance items, and the eval set is how I would make those acceptance tests real: the same rows, run against every new release, with the result on the table before anyone forms a view. This week that would have been five runs. Most would lose to the incumbent on at least one failure mode, and one would probably win on cost alone. Either way the decision takes an hour and the argument takes none.

Two honest limits. The set drifts, so retire rows that no longer happen, add the ones that just went wrong, and re-baseline when you do. And 50 rows will miss things. It catches the regression that would have embarrassed you, which is most of the value.

What to actually do

Book the afternoon. Two people, the last month's queue, 30 to 50 rows, the five failure modes above. Ship the set before you argue about any model on this week's list.

Baseline the model you run today. Without it, every result from a new release is a number with nothing to compare against.

Put the run in the release calendar. When the next frontier model lands, and it will within a fortnight, the eval runs first and the meeting reads the result. That is the whole switching process.