The first post in this series found that field order sometimes changes output quality, and it ended with a promise to check whether that holds on a small model that doesn’t think, from a family other than Gemini. The last post was the failed version of that experiment, where Vertex alphabetized my schemas before the model ever saw them and the independent variable never made it out of the request. This is the run that worked.

Dusting off the server

I have spent about a year using models through APIs, mostly Vertex and Bedrock, without paying much attention to what was running underneath them. That was fine until field ordering turned out to be something the serving layer might rewrite on its way past, at which point I wanted a stack I could completely control.

I tried the MacBook Air first, an M4 with 24GB, and it was too slow to be worth using for a run this size. So I went and turned on the machine in my home lab, an RTX 3080 with 10GB of VRAM, which had been powered off for almost a year. Half a day of patching and driver updates got it healthy again. I started with Ollama, found that it was also reordering my fields, and did not chase that any further than confirming it, because what I wanted at that point was fewer layers between me and the model rather than a better-configured one. Everything below came off that box running llama.cpp directly.

Two models, no thinking

I ran Qwen3-4B-Instruct-2507 first and then Qwen3-14B with reasoning disabled, using the same six schema variants and the same 72 cases as the original post, judged for encounter reachability the same way as before.

The effect is there on both models, and on the 14B the gap between the best and worst ordering is close to fourteen-fold. Nothing changed between those runs except which order the fields were declared in.

The comparison I care about most is alpha_nested against nested_narrative, because those two schemas are identical in every respect apart from the order of their fields, down to the nesting and the field names. On the 14B one of them breaks better than a quarter of all encounters and the other breaks under three percent of them. Every other comparison in this series has had some structural difference mixed in with the ordering, and this one doesn’t.

The ranking is not the same as Gemini’s. There, append_order was the worst variant at minimal thinking and flat_alpha was near the bottom, while here flat_alpha is the worst by a wide margin on both models and append_order sits in the middle. What does hold across all of them is the other end of the chart, where nested_narrative and grouped_by_type are at or near the floor on Gemini, on the 4B and on the 14B.

Room for thinking

Qwen3-14B is a hybrid checkpoint, so the same weights that produced the numbers above will also produce reasoning traces if you ask for them, which made the thinking run cheap to set up. It was not cheap in memory. Reasoning tokens live in the KV cache along with everything else, and finding room for them next to a 14B at Q4_K_M on a 10GB card took some arithmetic that renting the model by the token had spared me from ever doing. It fit, with less headroom than I would have liked.

Thinking collapses the three worst orderings into single digits and leaves the two best ones roughly where they were, which takes the spread across the six variants from something like fourteen-fold down to under three. That is the same pattern I found on Gemini, and it means the same thing here: a bad ordering can be rescued by spending reasoning tokens on it, and a good ordering does not need them, so the saving is in never spending them.

One variant goes the other way. ui_contract is worse with thinking on than either non-thinking run, and it is the only one that does this. I don’t have an explanation and I am not going to invent one. It resembles the non-monotonic behaviour I saw at low thinking budgets on Gemini, where a few orderings got worse before they got better, and I would treat both as unexplained rather than as evidence of anything in particular.

So what?

Four models now, across two families, with thinking and without, and what survives all of it is fairly narrow. Which orderings are bad turns out to be specific to the model, so the worst variants from one model are not a guide to another. The orderings that are good stay good, and nested_narrative and grouped_by_type have been at or near the floor everywhere I have looked, with a gap to the worst variant on the same model that is wide enough to matter if you are shipping something.

So the advice from the first post holds, with a different reason behind it. Order your fields by decision structure, not because any particular alternative is universally bad, but because that ordering has been reliably close to the floor on every model I have tried, and it costs nothing to do.

Scope and limits

The 14B is Qwen3-14B rather than an Instruct-only checkpoint, so this is not a clean comparison against the 4B and I am not making any claims about model size. Both are quantized to Q4_K_M, which is a difference from the full-precision Gemini runs. The thinking checkpoint truncated levels more often than the others, and the reachability judge responded by inventing the missing encounters instead of reporting the gap, so I patched it and re-ran the judging for that column only, rather than spend another six hours of judging per column on the two I had already scored. A small number of judge calls were genuinely ambiguous on manual review, all of them in alpha-ordered variants, which is too few to read a pattern from but worth mentioning. Encounter counts per variant, including the levels that came back short, are in the repo.

All of this is directional. There is one generation per case and no repeats, so the ends of the chart and the size of the gap between them are worth something, and the difference between two variants sitting a point apart is not.

Run it yourself: the harness is in the repo.

Field order sometimes changes output quality. That was the finding in the first post in this series, and the follow-up I said I would run was whether the effect holds on a small model that doesn’t think at all, from a different family than Gemini.

I picked Qwen3-Next-80B-A3B-Instruct, served through Vertex AI’s Model-as-a-Service endpoint. It ships as two separate checkpoints, Instruct and Thinking, rather than as one model with a switch, and the Instruct build doesn’t emit thinking traces at all, which makes it a cleaner baseline than a reasoning model dialed to zero. Six schema variants, the same six from the first post, 72 cases each, judged for reachability the same way as last time. The run completed and the judging finished without anything looking obviously wrong.

What came back

Five of the six variants land between 16.7% and 23.3%, close enough together that at this sample size I wouldn’t read the ordering among them as meaning anything. The decision-ordered schema, nested_narrative, sat at or near the floor in every column of the Gemini work and comes out mid-pack here. The only variant that clearly separates is flat_alpha, and it separates by being the worst, which is not where it landed on Gemini either.

A result where five different treatments produce roughly the same number is either a real finding about the model or a sign that the treatments weren’t different, and the second is much easier to check than the first, so I went and read the raw output instead of the summary.

Where the order went

The nested variants were all coming back with their keys in alphabetical order, whatever the schema had declared. That also explained flat_alpha, which is the one variant whose declared order is already alphabetical, so it was the only case where what I asked for and what came back agreed.

If the schema was being alphabetized somewhere between my process and the model, then five of my six conditions were the same condition and the experiment had never run. The question was where the sorting happened, because that determines whether it’s mine.

Pydantic. The schemas are Pydantic models and the request carries model_json_schema(). I had been using Pydantic the same way against Gemini for months without seeing this, so I doubted it was the culprit, but it’s two lines to check and it’s the layer closest to me:

>>> list(NarrativeLevel.model_json_schema()['properties'])
['theme', 'setting', 'player_weapon', 'player_weapon_damage',
 'player_health', 'encounters', 'boss', 'objective', 'reward']

>>> list(AlphaNestedLevel.model_json_schema()['properties'])
['boss', 'encounters', 'objective', 'player_health', 'player_weapon',
 'player_weapon_damage', 'reward', 'setting', 'theme']

Both come out in declared order, one narrative and one alphabetical, which is what the variants are supposed to be. Whatever is sorting them is downstream of this.

My own results files. This is the part that had kept me from noticing earlier. The harness writes results with model_dump_json(), which serializes the parsed object using the field order declared on the Pydantic class rather than the order the response arrived in. The model returned alphabetical, my client parsed it into an object where order is a property of the class, and the file on disk came out in the order I expected to see. Nothing short of reading resp.choices[0].message.content, the response string before anything touches it, shows the ordering the model actually produced.

The wire. Two variants with genuinely different declared orders, run on the same case:

Two different declarations, one output ordering. The same held at every nesting level, inside each encounter object and inside each encounter’s enemy object.

Properties order against required order. A JSON Schema carries field order in two places, the properties object and the required array, and it seemed worth ruling out that one survived while the other didn’t. I built a small schema where the properties order, the required order, and alphabetical order are three distinguishable sequences, at two levels of nesting:

Alphabetical at both levels, matching neither declared sequence.

The SDK. Everything up to this point had gone through the openai package and Pydantic, so the last thing to rule out was my client stack in its entirety. I wrote the request body by hand into a file, sent it with curl, and read the response without parsing it:

curl -s -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  --data @request.json \
  "https://aiplatform.googleapis.com/v1/projects/${GCP_PROJECT}/locations/global/endpoints/openapi/chat/completions"

The model’s output, pulled out of the response and indented for reading:

{
  "alpha_field": "zorblax",
  "child_object": {
    "bravo_field": "flimble",
    "charlie_field": "quixnar",
    "yankee_field": "snorgle"
  },
  "zeta_field": "wibblet"
}

Same alphabetical ordering at both levels, from a request with no Python, no SDK and no Pydantic anywhere in it.

I spent a while after this looking for a way to ask for a specific order and didn’t find one. Gemini’s API has a propertyOrdering field that does exactly this, and I tried sending it, but it isn’t documented for this endpoint and I couldn’t get it to have any effect, so I can’t tell whether it’s unsupported or whether I had the shape wrong. I also tried reaching a MaaS model through Vertex’s native endpoint rather than the OpenAI-compatible one, and got back a 200 and a response that ignored my schema and invented five demonstration fields, which I take to mean that path isn’t meant for these models. Neither of those is a result, and I’d treat both as me not finding the door rather than evidence there isn’t one.

The model. Google’s documentation for MaaS structured output has a usage example, a calendar event with three fields, so I ran it unmodified except for the model ID:

class CalendarEvent(BaseModel):
    name: str
    date: str
    participants: list[str]

completion = client.beta.chat.completions.parse(
    model="qwen/qwen3-next-80b-a3b-instruct-maas",
    messages=[
        {"role": "system", "content": "Extract the event information."},
        {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
    ],
    response_format=CalendarEvent,
)

{
  "date": "Friday",
  "name": "science fair",
  "participants": ["Alice", "Bob"]
}

The schema declares name, date, participants and the response comes back date, name, participants, which is alphabetical. The documentation page shows the output of this example in declared order. Running the same thing against DeepSeek rather than Qwen gives the same reordering, so it isn’t specific to the model.

What this does to the experiment

Five of the six variants were never different treatments, so whatever those break rates measure, it isn’t field order. The one number that survives as a real difference is flat_alpha’s 32.4%, and that comparison is confounded too, because flat_alpha is a flat schema with individually numbered fields and the other five are nested schemas that all arrived alphabetized, so what separates it might be the flattening rather than the ordering.

The replication question is still open. I don’t have an answer on whether the effect holds on a small non-reasoning model, and this run doesn’t get me closer to one.

The earlier Gemini results are unaffected by this, and I checked rather than assuming. Gemini’s native response_schema path returns fields in the order the schema declares, with alpha_nested arriving alphabetical because that is how it’s written and nested_narrative arriving in decision order because that is how it’s written. The independent variable did reach the model in those experiments.

All of this ran between 2026-07-31 and 2026-08-10, which is worth stating because this is the kind of behavior that gets changed without an announcement.

Why this matters

In March 2024 someone opened an issue against vLLM titled “Order of keys for guided JSON.” Different model, different serving stack, two years before any of this. They reported that the keys came back alphabetical rather than in the order their template declared, and that it had a significant impact on generation quality for their task. They filed it as a defect because it was breaking their work.

That is somebody I have never met, working on something I know nothing about, arriving at the premise of this series independently and reporting it as damage. I find that more convincing than my own numbers.

The same behavior has been reported against Google’s own OpenAI-compatible endpoint. In June 2025 a user showed that gemini-2.0-flash honored propertyOrdering through the native client in five runs out of five and failed to honor it through the OpenAI-compatible endpoint in three runs out of five, with the field confirmed present in the outgoing request. A Google engineer replied that they had reproduced it internally and would discuss it. The user asked for an update four months later, and the thread ends there. Four related threads about key ordering sit underneath it, going back to November 2024.

There is a reason this particular property is the one that keeps getting lost. Schema enforcement gets defended because a violation is loud: a missing required field or a wrong type breaks the parser on the other end immediately, so it gets caught in testing and it stays fixed. An ordering that gets discarded produces valid JSON with every field present and every type correct, nothing raises, and the only symptom is output that is a bit worse in ways nobody is going to trace back to serialization. Properties that fail quietly don’t accumulate tests, and anything in the request path can drop them without a single alarm going off.

So the advice from the first post needs an asterisk that has nothing to do with models or tasks. You can order a schema carefully, ship it, and have the ordering discarded in transit, and everything you can see afterward will look right. In my case my own client library reassembled the response into the order I had asked for, so even the results on disk agreed with me. The only reason I found out is that I went and read bytes I had no particular reason to read.

Scope and limits

One provider, one API surface, two models, tested over ten days. Gemini’s native path preserves declared order and I verified that directly rather than assuming it. I could not find a documented way to control field order through Vertex’s OpenAI-compatible endpoint, which is not the same as there not being one, and if somebody knows the parameter I missed I would like to hear about it. I have no visibility into what runs behind that endpoint, so this is a report of the behavior and not a diagnosis of the cause. The Qwen run and the harness changes are committed, but those numbers are an artifact of the investigation rather than data about Qwen. The reproduction is one request body and one curl command, in the repo.

If you have measured the quality of LLM output at any scale, you have probably built one of these. You write a rubric of a few criteria, you have a strong model score each one from 1 to 5, and you combine the scores into a weighted average. It is the realistic way to grade open-ended output when there are too many outputs to read by hand and no single right answer to check against. I have written this kind of judge many times, for work and for side projects.

It is easy to build and easy to trust. You write the criteria, you run it, it returns numbers, and the numbers look reasonable. Nothing about using it tells you when it is wrong. And a weighted average has two ways of being wrong that never show up in the number it hands you.

I found both in a judge of my own, so the example below is a game-level generator. The lesson is not about games.

A score that survives a broken artifact

The judge was scoring generated levels. Each level has the player fight three encounters, and each enemy has a weakness the player has to exploit with an item. The catch is timing: the item for an encounter is only picked up after that encounter is won, so a weakness has to point at the starting weapon or at something from an earlier fight. Point it at the encounter’s own pickup and the fight is unwinnable, because the player cannot be holding that item yet.

One level had all three enemies weak to their own encounter’s pickup. Every fight unwinnable, the whole level impossible to finish. My judge scored it 4.75 out of 5.

That number should not be able to exist. Getting to why it does took two passes at the problem, and the two reasons are different.

The first way: the weights hide it

To check the rubric I built a second judge that did one thing. It read each level and asked, per encounter, whether the weakness could be satisfied by something the player already had. One rule, a yes or no, close to ground truth. Over the same 64 levels the rubric judge averaged 3.95. The one-rule judge found that one in seven levels had at least one unwinnable encounter. Same outputs, and one judge said fine while the other said a seventh of them could not be completed.

The rubric had six criteria. Two of them, the ones most likely to notice an unwinnable level, I had weighted at the bottom, one point each out of twelve. The criteria that a broken level can still score well on, theme and the like, I had weighted higher. So even when the rubric noticed the problem and marked the relevant criterion down, the score climbed back up on the criteria that did not care about it, and landed near the middle. The one criterion that saw the defect was outvoted by the ones that didn’t.

The honest version of how the weights got that way is the useful one. I did not weight them carelessly. I decided early that some criteria mattered more than others, which was reasonable on its face, and then never checked it. This was a demo, not something anyone depended on, so I set a plausible weighting and moved on without confirming it scaled the judgment I actually wanted. The weighting wasn’t thoughtless. It was unvalidated, which looks the same from the outside and lands in the same place. And the same move, a sensible early weighting that never gets re-examined, walks straight into real systems where the stakes are not a demo. That is the first failure mode, and it is the tamer one: the judge saw the problem and the arithmetic buried it.

The second way: the criterion drifts

The tamer one assumes the judge saw the problem. Often it doesn’t, and it doesn’t fail the same way twice.

I ran the rubric judge over the levels twice, two independent passes. On the broken level above, one pass scored it 3.33 and the other scored it 4.58. Same level, same rubric, two and a half points apart. On the low pass it caught the timing problem and marked the reference criterion down to a 2, noting every enemy was weak to a pickup from its own encounter. On the high pass it called the identical pattern a minor issue, scored that same criterion a 4, and wrote that no names were wrong.

The reason is in the rubric. The reference criterion asked whether names existed and matched. It never stated the rule that actually defined the failure, that a weakness has to point at something the player already has. So when the judge met a level built entirely on that failure, it was grading against a criterion that never ruled on the question, and it improvised. One run treated the timing break as fatal. The next treated it as a stylistic quirk worth a point. Both are defensible readings of a criterion that was never fully written.

This is the failure mode worth sitting with. It is not bias, and it is not the lossy average, though both are real here. It is that a criterion you left underspecified gets interpreted differently every run, and the overall score hides which reading you got. One pass caught the defect and the weights buried it. The next pass never caught it. The average looked about the same either way. You cannot tell, from the number, whether the judge saw the problem and discounted it or missed it entirely.

Why this is easy to miss

Put the two together and the trap is clear. A weighted rubric is the only practical way to score a lot of this output, so you build one. It returns steady, reasonable numbers, so you trust it. And the number is steady even when the thing underneath is not: a real defect can sit inside a mid-range score, and a run-to-run coin flip on an underspecified criterion averages out to something that looks stable. The output that should alarm you looks exactly like the output that shouldn’t.

The weights you set by feel and the criteria you left half-written are deciding what the judge can see, and the aggregate will not tell you what they decided. A clean score from a judge you have not checked is the most dangerous thing in the pipeline, because it is indistinguishable from a clean score that is earned.

The first thing to say is that a lot of what a rubric judge does badly should never have been the model’s job. Reachability is a rule. A weakness has to reference the starting weapon or an earlier pickup, and you can check that in code that walks the inventory and compares names, no model in the loop, the same answer every run.

Anything you can write as a rule, write as a rule, and let the code judge be right the way the rubric judge can’t. My second judge was an LLM only because I wanted a loose match on item names and didn’t want to spend the afternoon on the exact version; for a real system I’d have written the check, and it would have caught every broken level here.

The model judge is for what genuinely resists a rule: whether the level is fun, whether the tone holds. There a rubric is the only thing that scales, and scale is exactly why it has to be calibrated. Most of the published work on judge reliability goes after bias in the model. Very little goes after the step where you crush a rubric into one number, which is where mine actually broke.

The habit that survives all of it is dull and it works: take a handful of outputs you already understand, and see what your judge says about them. If it disagrees with you on the cases where you know the answer, the number it gives you on the cases where you don’t is worth nothing.

Scope and limits

One rubric judge against one targeted judge, over 64 generated levels, scored twice each. The one-in-seven break rate is a fact about this set, which was built to stress exactly this kind of interdependency, not an estimate of how often it happens anywhere. The rubric judge’s swing across passes is large, and that is part of the point, not a caveat around it: an instrument that scores the same broken artifact 3.33 and 4.58 is telling you what its number is worth. The judgments and both prompts are in the repo. The useful move is not to trust any of this. It is to run your own judge against cases you already understand and see whether you believe it.

In one of the levels the model generated, the final boss is weak to an item the player never has a chance to pick up. Every field reads as plausible and nothing is malformed, but the weakness points at something the player cannot be holding by the time they reach the fight, so anyone following the level hits a wall and stops.

A level can be fun or flat, and that is a judgment call, but a reference either resolves or it doesn’t, and that is the only thing I judged this round. The prompt asks for something specific:

💡 Enemy weaknesses must reference the player weapon or a pickup from a prior encounter. The boss weakness must reference a specific pickup the player acquired during encounters. Each encounter trigger must set up the next encounter or the boss fight.

A level that violates one of those rules is broken in a way I can check, so this time I judged exactly that and nothing else: not whether the level was fun or coherent, only whether its references resolve. A level counts as broken if any encounter in it points at something the player can’t be holding yet, and the share of levels with a broken reference is the break rate the charts below plot, where lower is better. A judge with one explicit rule to check is fast and close to ground truth, where the soft rubric I leaned on last time was neither.

None of this matters at my scale. The side project runs a handful of generations a day, and fifty percent more tokens or fifty percent fewer comes out as the same rounding error. The schemas aren’t real either; they’re six orderings of the same fields, built to crank field interdependency as high as it goes so that a small effect gets big enough to see. I pulled the thread anyway, because a schema handed to a model is a construct with a cost, and a construct with a cost is the kind of thing I check without deciding to, the same reflex that won’t let me ship a query without reading its plan. It is part of the prompt, and right now it tends to get set by accident.

The earlier post showed that field order changes output quality, with the effect worst at minimal thinking. The obvious move is to turn thinking on, and this post is whether that works and what the budget costs.

Re-judging across the swap

Google pulled the preview model out from under this between posts, which is its own story, so I re-ran minimal on the stable release and put both minimal sets, old and new, through the new judge. The point was to confirm that the effect survives the swap and that the judge catches it in both sets, which it did. From there I walked the stable model up the thinking ladder, from minimal through low, medium, and high, at 72 generations per cell. The harness is in the repo.

The preview-minimal bar carries over as a reference to the first post rather than as a model comparison; there isn’t enough here to say one release beats the other at minimal, and I’m not claiming it. Because the schemas are built to exaggerate, every error rate below is a ceiling, so the real-world numbers would run lower. And at 72 generations per cell the small differences are noisy, so the direction the numbers move is worth more than the size of any single gap.

Thinking clears it

At minimal thinking the badly-ordered schemas break, the worst stable ordering failing on roughly one level in fifteen. The decision-ordered schema (nested_narrative in the table below) sits at or near the break-rate floor in every column and never needs the budget at all.

Turning thinking on closes the gap. The schema that was worst at minimal, append_order, reaches zero by medium and stays there; flat_alpha, less broken at minimal, turns out to be the stubborn one and doesn’t clear until high. Across the bad orderings, medium and high drive the error to roughly nothing. More budget gives the model room to keep a later field consistent with an earlier one instead of committing blind, and on this task that is enough. What I can show is where the number went, not what the model did to get there, since I have error rates and token counts rather than thinking traces, so the mechanism stays a guess and I will leave it one.

One result I can’t explain: low thinking did worse than minimal on two of the schemas, repeatably across reruns. A little thinking was worse than none. I don’t have a story for it, and I won’t invent one to cover the gap.

What it costs

Thinking isn’t free, and the cost barely moves with the schema. At high budget every ordering spends roughly the same number of thinking tokens, clean and broken alike, so a good schema doesn’t earn cheaper thinking so much as it earns you the option of skipping the budget entirely, clearing the bar at minimal where thinking is zero.

That reframes the saving as something other than fewer thinking tokens. The saving is in never spending them.

Here is the cheapest setting that gets each ordering to roughly zero reference errors, by total tokens:

The clean orderings clear the bar at minimal, while the bad ones climb the ladder, most to medium and the worst all the way to high, paying two to four times as much to land in the same place. The task, the model, and the output are identical across the rows; what changes is the order of the fields, and that is the root cause that forces the bad schemas up the ladder, charged on every call. The cheapest ordering isn’t even one I built to be good, which is the point worth keeping: clearing the bar takes nothing clever, only an ordering that doesn’t work against the model.

At a handful of calls a day this stays a curiosity, but wrap the same schema in a tool surface that fires a million times and the order of the fields becomes a line on the bill that nobody chose. The specific saving I measured here is contrived, but the variable behind it sits in every structured-output tool, usually inherited from a REST API that ordered its fields for a database or a UI, neither of which is the thing generating against them.

Scope and limits

One model, one task, six orderings built for dense interdependency, a single judge on a single property. Tasks with looser field dependencies will show less. The token figures are breakeven on a toy tuned to be hard, so treat them as a ceiling rather than a quote. The mechanism behind the compensation is unmeasured, error rates and token counts with no traces. The low-thinking dip is real and unexplained. Still in progress: whether the ordering effect holds on other small models without thinking.

QA is being cut on the theory that AI can test now. The theory is mostly right. But the role bundles two jobs under one title, and the cut can’t tell them apart. One of them AI just finished. The other just got more valuable.

The first job is execution. Running cases, checking paths, regression suites, filing bugs. This was being automated for a decade before the current models, and the current models finish it. Cutting it is fair. A machine does it as well or better, and I would not argue otherwise.

The second job is a stance toward the product: probing how it will actually be used rather than how it is supposed to be, and finding where that breaks. Those are different questions, and the gap between them is where the value lives.

The field that worked

I shipped a text input that had to be uppercase. The spec said uppercase. My validation enforced uppercase. Every case passed. Our QA flagged it anyway: users will type lowercase out of habit and get bounced by a field that is doing exactly what it was told. I pushed back. It’s working as intended, nothing’s broken.

She was right and I was wrong, and it took me a while to see why. Nothing was broken. Nothing was even unconventional, which is why neither I nor a model reading the spec would flag it. She knew, from watching this specific population in other software, that people type how they type, and that being blocked by a field behaving correctly does not read to a user as their own mistake. It reads as the product being broken, and they leave. The field was perfect, which is why it never read as a correctness problem at all. What broke was the fit between a correct field and how these particular people behave, and that is visible only to someone holding that context. The fix was trivial once seen: accept lowercase, uppercase it ourselves, ship. But someone had to see it coming.

The field is the easy version, because it is one rule on one screen. The stance matters more where no single screen is wrong. Picture two features in the same release. One logs you out after ten minutes idle. The other is a multi-step form that saves only on final submit. Each is correct on its own, and each passes. The problem only exists for a population that gets interrupted constantly: they fill three steps, get pulled away, and come back to a login screen and an empty form. Nothing is broken. Neither feature violates anything. The failure lives in the interaction between two correct things and a real fact about how these particular users spend their day, and it is invisible to anyone reasoning about either feature alone.

That is the shape of the work. Every piece is right and the experience is still wrong, and the job is to go find where.

The adversary

This gets mistaken for product’s job, or design’s. It is a different stance. Product and design are generative. They reason about what the thing should be and how a user ought to move through it, and by role and incentive they advocate for that vision. The adversary asks the opposite question: how it will be used rather than how it should be, and where that fails.

A good one is faintly annoying, and the annoyance is the role working correctly. They don’t just tell you the build is wrong, they tell you the requirement was wrong, that the thing everyone agreed to ship should not ship as written. That is more irritating than a bug report and worth more than one, and you learn to love it. Fold quality into “product owns it” and you keep the advocate and lose the adversary, because nobody paid to make the vision succeed is positioned to attack it.

Why AI does not fill the gap

This is the part I expected to go the other way, because I spend my days pointing models at structured work and they are good at it. They are not weak here. Two separate things stop them, and I don’t think either is a capability gap a better model closes, though I hold that loosely with the models moving the way they are.

The first is aim. A model will answer any question you ask about your users. It will not raise the one you did not think to ask. You can make it adversarial, tell it to attack, play the confused user, hunt for the collision between two features, but only if you already know where to point it. The aiming is the scarce thing, and the aiming is exactly the judgment being cut.

The second is incentive. Like the builder, a model is an advocate. It executes the intent you give it and optimizes the goal you set, and it is not positioned to attack that goal. Point it at a known failure mode and it will probe that mode at a scale no team can match. The model is a multiplier on the person who already knows where users fail, who can now run those modes at scale. Cut the judgment to save headcount and you keep the multiplier and lose the thing it multiplies.

Same on paper

Here is the hard part. A good QA and a useless one look the same on paper. Same tickets, same cases run, same bugs closed. The difference shows up only when you work with them: the good one keeps finding things you would never have considered, and can tell you why each one matters, for whom and under what real condition. That is legible to a peer in an afternoon and invisible to a spreadsheet. It does not compress into a metric, so it does not travel up an org chart.

The obvious objection is that this is exactly what someone with no value would say: my contribution does not show up in the numbers, so trust me, it is there. Fair, and usually correct. But it is testable. The person with nothing makes claims that are vague and after the fact, because vague and after the fact cannot be caught being wrong. The real adversary makes the opposite kind of claim: specific, about behavior you can watch, made in advance, against pushback. Users will type lowercase and leave is falsifiable the second it is said. The engineer who disagreed could have been right. I was that engineer once, and I wasn’t. What sets the valuable one apart is that the calls can be checked, and they’re right at a rate that isn’t luck. That’s the part of this I can least prove on paper, which is sort of the whole problem.

The evidence erases itself

But look at what that test needs. To know someone is right at a rate that isn’t luck, you have to accumulate the calls. And the calls don’t accumulate.

They erase themselves twice. Once because nothing breaks: a prevented failure leaves no ticket, so the best work produces no artifact at all. And again because when the adversary is right, the team concedes on the spot. Oh, you’re right, change it, ship. Within minutes the catch is relabeled obvious, of course we handle lowercase, and the fact that one specific person saw it when nobody else did has dissolved into common sense. There is no incident to point back to, no record it was ever in question. The value was real, and it is gone from the account.

So the rate is computable in exactly one place: the head of a peer who was in the room, watching the calls get made and conceded before they evaporated. That is why the value is legible to a colleague in an afternoon and invisible to the file. I mean that mechanically. The afternoon is the only window in which the evidence exists, and the file is built after the window closes.

Why a rational org cuts them anyway

This is why competent organizations cut these people. The local manager often knows exactly who they are, so it isn’t blindness. The cut is made one level up, on the only signal that scales: headcount, tickets, “AI reduced testing effort by sixty percent.” The distinction that matters was never in that signal. And it is worse than ordinary invisibility, because the best adversaries are the hardest to see in the record. Their best work is the failure that never happened, and a failure that never happened leaves nothing behind. The track record undercounts exactly the people most worth keeping.

You can’t fix that by tracking harder, because the thing you would track is destroyed at the moment of concession. You can’t trust the metric, because it is silent on the only thing that matters. You can’t trust the artifacts, because the best work is locally testable and globally invisible. And you can’t trust the local manager either, because the incentive runs the wrong way. A smaller team reporting the same tickets is a win on the manager’s own scorecard, and defending an adversary means defending a headcount with a story, against a number. The one person positioned to know is the one person paid not to say.

A sorting problem

The implication is not “don’t cut QA.” Much of it should go. The execution half is genuinely done, and AI finishes it. But this is a sorting problem dressed as a budget cut, and the sort can’t be done from a distance. The failure is cutting while you believe the number in front of you tells you what you are giving up. Whether you ultimately cut or keep is downstream of that. An organization that knows it’s blind here will at least hesitate. One that reads headcount and tickets as the whole picture will cut clean, report a tidy efficiency gain, and never learn what it lost.
I was the engineer who said it was to spec. I was right that nothing was broken. That was never the question.

A preview model is a deal. You get something good early, you accept the rough edges like throttling, inconsistency, and surprise behavior, and you accept the clock. The deal is worth taking when the preview model is the best thing available for what you are doing. It is not worth taking blind.

Google has been particularly aggressive on the clock lately. They launched Gemini 3.1 Flash Lite Preview, deprecated it with almost no warning, and then shipped the stable version with a deprecation date one year out and no announced replacement. Born to die. In a year. I run a different side project on that model, and the clock is already ticking with nothing obvious to move to when it runs out. This is GCP’s normal posture toward things you build on, and it is the reason this post exists.

Gemini 3.5 Flash went GA on May 19, and it is the obvious successor to Gemini 3 Flash Preview, which is what one of my side projects (a tiny playable adventure generator for kids) runs on today. Google has not announced a deprecation date for 3 Flash Preview yet, but the right time to decide whether to migrate is before they do. I was on the preview model because it was clearly better than Gemini 2.5 Flash at the task I needed: better outputs, more consistent across runs, worth the throttling I had to build resilience around. That is the only reason to take the preview deal in the first place.

The harness

I do not swap models on a vibe. Before I touch production, I run my real workload against both models with a stronger model judging the output on a rubric, and I look at the numbers. The numbers in this post come from a public demonstrator of that approach, the autoregressive-schemas repo from the previous post, extended to 72 input cases. Same setup: the decision-ordered nested schema that won the earlier study, same prompt, same thinking_level: minimal, 144 generations per model, judged by Claude Sonnet 4.6 with extended thinking on a six-criterion weighted rubric. My actual production harness is structurally the same but written in a different language and using its own prompt, schema, and rubric. The shape of the validation is what matters, not the specific code.

What the run showed

Quality. Mean rubric score 4.46 for 3.5 Flash against 4.10 for 3 Flash Preview, and 3.5 won 55 of 72 cases head-to-head with 17 losses and no ties. The gains concentrate on reference integrity, causal chain, balance, and mechanical sense, the consistency criteria, exactly where I want them for coherent generated worlds. Thematic coherence is flat. Completeness slightly favors 3 Flash Preview, and at this n I am not reading anything into it.

Reliability. Zero retries across 144 calls on 3.5 Flash. On 3 Flash Preview, 12.5% of calls needed at least one retry and the worst single call needed five. That’s the difference between a preview model and a GA model showing up in the numbers, and it’s also less resilience code I have to keep paying attention to.

Latency. Close enough to call even. 3.5 Flash is ~250 ms slower at the median but tighter, with about half the standard deviation and a worst case of 8.8 seconds against 12.9 for 3 Flash Preview. I will take consistent-and-slightly-slower over fast-and-jittery for a UI that already shows progress.

Cost. 3.5 Flash is 3x the per-token price of 3 Flash Preview, and produced ~33% more output tokens in this run, putting the real per-call cost ratio at ~4x. The token gap is partly whitespace: at thinking_level: minimal, 3 Flash Preview minifies its JSON about 40% of the time, while 3.5 Flash pretty-prints every time, which is what the bimodal distribution in the chart shows. Whether 3.5 Flash can be steered to minify is an open question I did not chase. For this product the absolute dollars are rounding error regardless.

Migration

The code change was the model ID. The harness is why that was enough.

The point is not that you should run Gemini 3.5 Flash. The point is that the answer to “should I migrate” is not in the changelog, it is in running your own task against both models with the numbers you actually care about in front of you. Preview is preview. Build the muscle before you need it.

Scope and limits

One task, one schema, 144 generations per model, one judge, one day. Quality on personalized game generation is a squishy thing to score and the rubric is mine. The bimodal output behavior on 3 Flash Preview is noted, not characterized; I did not try to steer it. No claim about 3.5 Flash being better than 3 Flash Preview in general, only on this task on this rubric. If you are on preview and considering staying, the right move is to run your own version of this against your own workload. The harness is in the repo.

A prompt firewall is cheap to run in front of a language model because the language model is slow. A sanitization call that adds 80 ms disappears into the second or two a model takes to respond. You can afford to screen every prompt because something slower is always paying the bill.

My architecture does not give Model Armor anything to hide behind. It gates payment. Model Armor screens the input, the charge goes through only if it passes, and the model runs last, on data that has already been through the payment processor. There is no slow inference call in front of Model Armor to absorb a bad moment. Every millisecond it spends is a millisecond a user spends watching a spinner with their card on the screen.

I had been running it like this in production for weeks and I could not have told you what it cost me. The median was fine, because the median is always fine, and the median is the number you see without trying. I had never looked at the rest of the distribution. This is the post where I did.

One note on scope. This is about latency, not availability. Putting a third party in the payment path raises questions about what happens when it is down rather than slow. Those questions are real and I am not answering them here.

What Model Armor is

Model Armor is Google Cloud’s prompt and response screening service. You define a template that turns on some set of filters, call the API with a piece of text, and it tells you whether the text tripped anything. It screens for prompt injection and jailbreak attempts, for sensitive data, for malicious URLs, and for the responsible AI categories. Those last ones are grouped under the label RAI: sexual content, hate speech, harassment, and dangerous content, each with a configurable confidence threshold. Each filter is independent and you can combine them freely.

It sits wherever you put the call. It is built and documented to wrap the model, screening the prompt on the way in and the response on the way back. I put it in front of payment instead. That is the reason this study is about the tail and not the median. Where you call it from decides whether its worst behavior is something you will ever feel.

Whether it fires at all

This is not an efficacy study. A real one would try to trip the filters up, probe the edges, and hunt for the inputs that slip through, and I did none of that. What I wanted before spending any time on latency was just confirmation that the filters were not silently broken, because how fast a filter runs is not worth measuring if it does not work in the first place. So this is a precondition check and nothing more: one input per filter that should obviously trip it, one that obviously should not.

No surprises, which is exactly what I wanted before moving on to the part I actually came for, which was the cost.

A small experiment

The harness calls Model Armor across a set of filter configurations, times every call, and writes the raw per-call numbers out. It runs as a Cloud Run job in us-central1, the same region as the Model Armor endpoint, so the numbers are Model Armor’s own latency with no meaningful network path in the way. The repository provisions everything with Terraform, runs, and tears itself down: https://github.com/jdhornsby/model-armor-latency.

I tested 13 configurations: 7 single filters and 6 combinations, from the one I run in production up to everything turned on at once.

Each configuration ran at 4 prompt sizes, 100 calls per size, for 5,200 calls in total.

Prompts were generated by repeating a fixed sentence until they hit a target size, four sizes from short to long. That is deterministic and reproducible, so the size buckets are consistent across runs, but repeated filler is not representative of natural input and the API does not return the token count it actually billed, so the sizes are approximate and I do not analyze latency against length. Everything below is pooled across all four sizes.

The other thing is sample size. With 100 calls in a cell, the 99th percentile is essentially the second-slowest call you saw, and one slow request moves it a long way. My first charts made specific filters look catastrophically unstable at specific sizes. The raw calls showed three or four slow requests out of a hundred.

Everything below is pooled, 400 calls per configuration, because the pooled view is stable and the per-cell view is not.

The tail

The median is good everywhere. Light filters sit around 75 to 85 ms. Prompt injection moves it to about 127 ms. Every filter turned on at once is about 139 ms. Prompt injection is the only filter that moves the median at all.

Here is every configuration, pooled, sorted by 99th percentile. Read the columns, not the order: the rank between the heavy configurations is not reliable, and there is a clean example of why below.

Across 5,200 in-region calls, 1.75% took longer than 500 ms and 0.75% took longer than 1 second (39 of 5,200). Roughly one call in 130, in region, with no network in the way, takes more than a second, on the day I ran it, against this configuration. The slowest individual calls in the run were over 4 seconds, and those single extremes are exactly the per-cell noise the pooling is built to absorb.

Read the gap between the two dots, not where they sit. It is small and even for the light filters and wide and ragged for anything with prompt injection or heavy RAI.

The tail is not spread evenly.

The URL filter, sensitive data protection, and the hate speech filter, each alone, did not put one call over 500 ms in 400 attempts. The light filters are quiet. Everything with a heavy tail contains prompt injection or one of the heavier RAI categories. Prompt injection on its own is the odd one: its median is the highest of the single filters, but on its own it never put a call over a second in 400. The multi-second calls only show up once it is stacked with something else.

That is as precise as I will get, and the table shows why I will not get more precise. Sorted by 99th percentile, it has “All filters” beating “PI + all RAI”, a configuration outrunning a strict subset of itself, which cannot happen for real. A p99 over 400 calls is just the fourth-slowest call, and at that resolution the order among the heavy configurations is noise. So the finding is coarse and one-directional, and that is all it is: stacking does not touch the median, anything with prompt injection or heavy RAI has a worse and less predictable tail than the light filters, and the ranking within that group is not real. Median flat, tail not.

I do not know why prompt injection behaves this way and the rest do not. The shape is consistent with prompt injection running a heavier model-based classification while the other filters are cheaper checks, but that is a guess about internals I cannot see.

If Model Armor is in front of your model, most of this never reaches your users, because your inference call is slower than its worst day. If it is in a path with nothing slower in front of it, the median is not your number. Your number is one call in 130 over a second, in region, unpredictable, reducible with fewer filters but not removable. I run prompt injection plus sexual content in production. It has a real tail. I keep it because what it protects is worth that to me. That is a decision you make with your own numbers. Make it with the tail in front of you, not the median you see by default.

Cost

Model Armor is free up to 2 million count per month per account, then $0.10 per million. Count is Model Armor’s billing unit, roughly one per token of inspected text. Producing this, development included, cost under a dollar on a dev account. A single clean run sits near the free-tier line, so reproducing it is effectively free. Given everything above, you should probably do that against your own configuration rather than trust mine.

Scope and limits

One service, one region, 13 configurations, 4 prompt sizes, 100 calls per cell. Per-cell counts are too small to characterize a specific filter at a specific size, which is why everything here is pooled, and the pooled counts carry the shape and not a precise number for any one configuration. Prompts were repeated filler, not natural text, so this measures the filters on synthetic input and the size dimension is not analyzed. This is latency under normal operation and says nothing about Model Armor failing or under unusual load. If it matters to your system, run the harness against your own configuration and look at your own tail. That is the number that decides whether this is fine for you, and it is not one you can get from the median.

LLMs are good at producing structured output from messy input. That property is core to agentic systems and to tasks like structured data extraction. Both are work I do day to day, at my job and on side projects. A good prompt goes a long way, but how much does the output schema itself matter?

I started looking at this because of a class of failure I kept hitting. The output would be locally fine, every field reasonable on its own, but globally broken. I ran into it doing structured data extraction from financial documents on another project, and I ran into it again on a personal side project at madejustfor.me that generates personalized story-game adventures for kids. Different domains, but the failures looked similar, so I wanted to understand why.

The eel

Here’s a level the side project produced for an underwater dungeon. The relevant encounter:

{
  "location": "Murky Trench",
  "enemy": {
    "name": "Bio-luminescent Eel",
    "weakness": "Tainted Kelp"
  },
  "pickup": {
    "name": "Abyssal Core",
    "effect": "Stabilizes pressure"
  },
  "trigger": "Eel carcass blocks the path to the ancient gate"
}

The player defeats the eel and the eel’s corpse blocks the path forward. Defeating the enemy creates the obstacle instead of removing it. The pickup from this encounter is the item needed for the final boss, so the player has done all the work and is now soft-locked behind a fish they killed. Each field is reasonable in isolation, but the level is broken.

Why the schema order matters

The model is generating left to right, one field at a time, and each field is conditioned on every field before it. Once “Bio-luminescent Eel” is committed to the output, the trigger field has to say something consistent with there having been an eel. The cheapest continuation that references the eel is the eel itself, which by this point in the output is dead, so the carcass becomes the obstacle.

The model didn’t reason about whether defeating an enemy should produce a blockage. It produced tokens that locally fit the field name and locally referenced what it had just written. That’s how autoregressive generation works. The schema isn’t a shape that gets filled in, it’s a sequence of fields generated in order, and each field constrains what later fields can plausibly say. Field order is part of the prompt whether you intended it that way or not.

This is true whether the model is producing freeform JSON or generating under a forced schema like Gemini’s structured output mode, which is what I was using. Constrained decoding makes the argument sharper, not weaker: the schema is literally part of the decoding loop, walking the model through fields in the order you defined.

The fix is to put fields in an order that mirrors how a person would make the decisions. Setting and theme before specific entities, premise before consequences, anchors first and dependents later. An incident report would start with what happened before listing root cause. A meeting summary would establish purpose before action items. The pattern is the same: commit to the things that frame everything else first, then let later fields depend on them. When the model commits to “the path forward is sealed by ancient magic” before it commits to “the enemy is an eel,” the trigger has somewhere to go that isn’t the eel’s corpse.

A small experiment

I built a testbed to measure this. The model generates a video game level from a short input premise, with the same fields available across all six schema variants. I used Gemini 3.1 Flash Lite with no thinking budget because that’s what the side project runs on. I judged each output on a six-criterion rubric using Claude Sonnet 4.6 with extended thinking, across 64 input cases per variant.

The six variants I tested, each representing a way schemas actually get organized in the real world:

• a flat list - like API query parameters or a database table
• by type - numbers together, then dates, then strings, etc
• appended over time - fields are just added to the end as the schema evolves over time
• the ui contract - ordered to match the UI the API was built for
• alphabetic ordering - maybe with some reasonable structure, but hit by an auto-formatter
• a decision structure - organizes so that decisions come before downstream consequences

The decision-ordered schema came out on top. The gap between best and worst was 0.49 points on a 5-point rubric, and the decision-ordered schema beat each of the worse-ordered ones in 70 to 75 percent of paired comparisons on the same input. The gains show up mostly on reference integrity, causal chain, and thematic coherence, which are the criteria most affected by whether later fields stay consistent with earlier ones, exactly where you’d expect autoregressive conditioning to bite. Numbers, raw outputs, and the striking individual cases are in the repo: https://github.com/jdhornsby/autoregressive-schemas.

When to care

Most CRUD-style tool schemas don’t have this problem. If your tool takes independent parameters from a clear instruction like get_user(user_id, include_email), there’s no consistency pressure between fields and not much for schema order to do.

The failure mode shows up when you ask the model to construct something coherent: plans whose steps depend on each other, tickets where reasoning fields have to match the decision, configs whose later fields have to be consistent with earlier ones, reports that synthesize multiple inputs into a structured artifact. These are the kinds of tools that make agents useful, and they’re the cases where field order actually shows up.

Whether to care depends on two things: whether the model has to keep multiple fields mutually consistent, and whether you’re running at a scale or cost ceiling where you can’t just throw a bigger model or thinking budget at the problem. If both apply, schema design is worth thinking about. The same logic applies when wrapping an existing REST API as an MCP tool: you inherit a field ordering that was designed for human ergonomics, database normalization, or backward compatibility, and none of those are what helps an LLM generate good output. Whether it actually matters in any given case is an empirical question, and the cheap way to find out is to look at your own outputs for the pattern the eel illustrates.

Scope and limits

This was one model, one task, dense field interdependencies, minimum thinking. Tasks with weaker dependencies probably show smaller effects. Bigger models and thinking budgets compensate for a poorly-ordered schema, but you pay for it. Prompting and fine-tuning are bigger levers than schema design; schema design is a smaller one that costs nothing additional at inference time. Two follow-ups in progress, both separate posts: how thinking budget compensates for poor schema order, and whether the effect replicates on other small models without thinking.