How TypeSafe's Jev Might Work: Parallel Constrained Decoding from Scratch

TypeSafe's homepage shows the same classification twice. The LLM version completes in 8.566 seconds. Jev, their first "System One" model, completes it in 0.114. The launch post adds three more claims: Jev "generates all outputs in a single query", output tokens are free, and every decision comes back typed, with a calibrated confidence score.

I have no inside knowledge of how Jev works. TypeSafe describes it as a trained model — calibration comes from a training method they call RLCD — and I cannot reproduce that. But the other claims have a very specific smell. Free output tokens and "a single query" sound less like a faster model and more like a model that never writes its output at all.

So I tried to build that part. This post is the result: an independent reconstruction of the decoding technique I believe sits under Jev-style models, which I will call parallel constrained decoding. I build it up from nothing, including the one piece of transformer plumbing it depends on — the KV cache — then spend most of the post on what went wrong when I implemented it on open models, and finish by checking which of Jev's claims it actually explains.

TL;DR

• In a JSON answer to a closed schema, only about 1 token in 10 is a decision. The rest is scaffolding you could have written yourself.
• Prefill the prompt once, copy its KV cache to one short row per field, and read every field's decision out of one batched forward pass. A warm 28-field request costs 4 forward passes instead of 276, and the output is schema-valid by construction.
• Each field comes back as a probability distribution, plus a candidate mass number that flags when the model wanted to answer something off the menu.
• Most of the engineering is not decoding. It is tokenizer boundaries, quoted booleans, chat templates and label naming — each of which can silently flip answers.

The problem: most of a JSON answer is not an answer

Here is a support-ticket triage task. An angry enterprise customer, a production outage, a double-billed invoice. The schema has 28 fields, every one of them either a boolean or a closed set of labels:

sentiment                 enum     ['VERY_NEGATIVE', 'NEGATIVE', 'NEUTRAL', 'POSITIVE']
severity_tier             enum     ['SEV_0_CRITICAL', 'SEV_1_MAJOR', 'SEV_2_MODERATE', 'SEV_3_MINOR']
sla_breach_risk           boolean  [True, False]
target_resolution_hours   enum     ['1_HOUR', '4_HOURS', '12_HOURS', '24_HOURS']
... 24 more

This is exactly the kind of task TypeSafe pitches Jev at. The standard way to do it with an ordinary LLM is to describe the schema in the prompt and let the model write the object. Qwen2.5-1.5B does it in 273 tokens, which cost 276 sequential forward passes through the model. Roughly 2.4 seconds on a laptop.

Now count what those tokens actually say.

The model writes "severity_tier": "SEV_0_CRITICAL",\n. Of that line, you already knew the indent, the key, the colon, the quotes, the comma and the newline — the schema fixed them before the model ran. You also knew that once the model commits to the token 0, the rest of _CRITICAL follows, because no other allowed label starts that way.

One token in that whole line is a decision. The rest is scaffolding.

Do this across all 28 fields and you get 29 decision tokens out of 285. About 10%. The model spent roughly 90% of its generation time typing text you could have typed for it.

And the text it did write was not reliable. For a ticket whose subject line reads CRITICAL P0: Production US-East API 502 Gateway Outage, the model wrote:

severity_tier          'SEV_3_MINOR'
sla_breach_risk        False
churn_risk_level       'LOW'

It also drifted into a run of eleven false values out of fifteen booleans, and then stopped before the last key, so the object was missing ticket_priority entirely. Valid JSON. Not a valid answer to the schema.

Keep that SEV_3_MINOR in mind. We will come back to what the model actually believed there.

The piece of plumbing you need: the KV cache

To see the fix, you need one fact about how transformers generate text.

A transformer reads by having every token look at every token before it. To make that possible, each token computes two vectors per layer, called a key and a value. Think of them as a summary card the token writes about itself and leaves on the table for everyone who comes after.

When the model is about to produce token 274, it needs the cards of tokens 1 through 273. It already computed them when it read those tokens. Recomputing them every step would make generating n tokens cost n² work. So it keeps them in a pile. That pile is the KV cache.

For the triage prompt, the pile is concrete: 1,154 tokens × 28 layers × 2 (keys and values) × 2 KV heads × 128 dims × 2 bytes ≈ 32 MiB.

Here is the property that matters, and it is the whole post in one sentence: a token's card depends only on the tokens before it, never on the tokens after it. That is what the causal mask guarantees. Token 50's card is finished the moment token 50 is read, and nothing that happens later can change it.

So the pile is not just a speed optimisation. It is a snapshot of a prefix. And a snapshot can be copied.

Copy the pile once per field, hand each copy to a different continuation, and each continuation sees exactly what the model would have seen had it written that continuation itself. Not an approximation. The same arithmetic.

Where the analogy breaks: these are not notes a reader consults at leisure. Each copy is a live, mutable 32 MiB buffer that the model writes into as the continuation grows. Copy it a few dozen times and the memory is real — we will come back to that.

The idea: parallel constrained decoding

Put the two facts together. You know every token of every field's line up to the point where the labels first disagree. And the prompt's KV cache can be copied to as many continuations as you like, each one exact. So:

1. Prefill the shared prompt once. Instructions, the list of allowed values for every field, the ticket, and an assistant turn that already opens the JSON object with {. This is the expensive part, and every field shares it.
2. Build one short row per field. For severity_tier, that row is the tokens of "severity_tier": "SEV_ — the longest common token prefix of all four of its labels. The row ends exactly where the labels part ways.
3. Run all the rows in one batched forward pass, each on its own copy of the prompt's KV cache. Read the next-token distribution at the end of each row. Restrict it to the tokens that start an allowed label, renormalise, and that distribution is the decision.

The rows the compiler produces look like this:

row                     ··"sentiment":·"  → compare 'NE', 'NEG', 'POS', 'VERY'
row             ··"severity_tier":·"SEV_  → compare '0', '1', '2', '3'
row                 ··"sla_breach_risk":  → compare ' "', ' false', ' true'
row            ··"primary_department":·"  → compare 'BILL', 'ENTER', 'INF', 'LEGAL', 'SEC'

Notice what disappeared. There is no generation loop. There is no sampling. The model is never asked "what comes next?" — it is asked "of these five specific tokens, which do you believe in?" Twenty-eight times, side by side in one batch.

Step 1 is cheaper than it looks. Most of the prompt — the instructions and the catalog of allowed values — is identical for every request with the same schema, so its KV state is computed once and cached. Every request after the first, a warm request, only has to prefill the ticket itself.

In practice there is one refinement, and it is one of the findings below: the schema's first field is decided on its own, and its answer is written into the prompt before the others are asked. So a warm request costs four forward passes — prefill the ticket, decide the first field, write its answer, decide the other 27 in one batched pass. Four, against 276.

The output has every key, in schema order, with real booleans, and every value is one you allowed. Not because it was validated afterwards — because nothing else could have come out.

This is where the first two Jev claims click into place. "All outputs in a single query" is one batched pass over the fields. "Free output tokens" is what you would charge if you never generated output tokens: the cost is in the prompt, and the answer is read, not written.

How this differs from the "constrained decoding" you may already know. Libraries like Outlines, llguidance and XGrammar also guarantee schema-valid output, by masking illegal tokens at each step of the generation loop. That fixes validity but keeps the loop: you still pay one sequential forward pass per token, scaffolding included. Parallel constrained decoding removes the loop. It is the difference between checking each word as it is written and realising you only ever needed 28 words.

And you get confidence for free

Because each decision is a renormalised distribution rather than a sampled token, every field comes back with a number attached. Two numbers, actually, and the second is the more useful one:

• probability — how the model splits its belief among the labels you allowed.
• candidate mass — the raw, unrenormalised probability that the model's next token starts any allowed label.

Low candidate mass means the model wanted to write something else entirely. That is a hallucination detector that costs nothing extra, and in the next section it catches failures that nothing else does.

Remember SEV_3_MINOR? Asked in parallel, the same model puts 0.380 on SEV_3 and 0.315 on SEV_0. It is not smarter than the autoregressive run — it makes the same call, and the call is close to a coin flip. The difference is that the parallel decoder tells you it is close to a coin flip, and the generated JSON hides that behind a confident string.

What goes wrong when you implement it

The idea fits in a paragraph. Making it correct on real tokenizers and real models is where the findings are.

Everything below was measured on the same test bench: three 28-field schemas — the support triage you have already seen, a fintech fraud check and a code security review — plus a four-field stress test where one field picks from 255 customs categories. The main model is Qwen2.5-1.5B; where a finding showed up more clearly on another model, I say which.

Every trap in this section has the same shape. The parallel decoder asks the model a question that is subtly different from the one it would face if it were writing the JSON itself — a different token boundary, a different spelling, a different position, a different prompt. The model answers anyway, the decoder renormalises, and you get a clean, confident-looking label. Nothing in the output tells you the question was wrong. Each finding ends with the rule that fixes it.

The prompt every finding starts from

Since every trap is about the question the model is asked, here is the question itself. Every schema gets the same three-part prompt, rendered through the model's own chat template:

• a system turn with one line of instructions and a catalog of every key, its description and its allowed values;
• a user turn with the input — the ticket, alert or code diff;
• an assistant turn that already contains the opening { and a newline. The model never writes the brace; every field's row is appended right after it.

This is the fintech fraud prompt exactly as Qwen2.5-1.5B sees it, special tokens included. The triage and code security prompts are the same template with their own keys and input.

<|im_start|>system
You are a precise classification engine. Read the input and fill in a JSON object with exactly the keys listed below. Every value must be copied exactly from that key's allowed options.

Keys:
- "is_fraudulent": Whether transaction is fraudulent. Allowed: true | false
- "risk_tier": Calculated risk tier. Allowed: "LOW" | "MEDIUM" | "HIGH" | "CRITICAL"
- "recommended_action": Immediate mitigation action. Allowed: "APPROVE" | "CHALLENGE_2FA" | "MANUAL_REVIEW" | "BLOCK_TRANSACTION" | "FREEZE_ACCOUNT"
- "primary_anomaly": Predominant anomaly. Allowed: "IP_GEOLOCATION_MISMATCH" | "UNUSUAL_VELOCITY" | "HIGH_VALUE_TRANSFER" | "NEW_DEVICE" | "TOR_EXIT_NODE"
- "secondary_anomaly": Secondary anomaly indicator. Allowed: "RAPID_DESTINATION_CHANGE" | "MULTIPLE_FAILED_LOGINS" | "CARD_NOT_PRESENT" | "NONE"
- "requires_human_escalation": Whether senior fraud analyst must review. Allowed: true | false
- "can_auto_block": Whether system should execute automated block. Allowed: true | false
- "aml_sar_filing_required": Whether FinCEN SAR report must be filed. Allowed: true | false
- "fincen_flag_threshold": Whether transfer amount exceeds AML reporting threshold. Allowed: true | false
- "sanctions_screening_risk": OFAC sanctions risk. Allowed: "CLEARED" | "POTENTIAL_MATCH" | "HIGH_CONFIDENCE_MATCH"
- "tor_exit_node_detected": Whether connection originates from anonymizing proxy/Tor. Allowed: true | false
- "device_fingerprint_mismatch": Whether device hardware ID differs from cardholder device. Allowed: true | false
- "velocity_score": Transaction frequency score. Allowed: "NORMAL" | "ELEVATED" | "EXTREME_BURST"
- "block_card_immediately": Whether payment card should be deactivated immediately. Allowed: true | false
- "freeze_online_banking": Whether web portal access should be restricted. Allowed: true | false
- "notify_cardholder_sms": Whether urgent SMS alert should be dispatched. Allowed: true | false
- "notify_cardholder_email": Whether notification email should be dispatched. Allowed: true | false
- "requires_identity_verification": Whether government ID biometric verification is required. Allowed: true | false
- "merchant_category_risk": Merchant category code risk level. Allowed: "LOW" | "MODERATE" | "HIGH_RISK_CRYPTO" | "RESTRICTED_GAMBLING"
- "chargeback_probability": Probability of cardholder chargeback dispute. Allowed: "LOW" | "MEDIUM" | "VERY_HIGH"
- "counterparty_jurisdiction_risk": Jurisdiction risk tier of destination merchant. Allowed: "TIER_1_LOW" | "TIER_2_MODERATE" | "TIER_3_HIGH" | "SANCTIONED_ZONE"
- "auto_reverse_transfer": Whether clearing network should attempt immediate reversal. Allowed: true | false
- "fraud_ring_association": Whether signature matches known syndicate patterns. Allowed: true | false
- "alert_clearing_house": Whether inter-bank network should be warned. Allowed: true | false
- "escalate_to_fiu": Whether Financial Intelligence Unit should be notified. Allowed: true | false
- "confidence_score_tier": Model decision confidence bucket. Allowed: "HIGH_CONFIDENCE_FRAUD" | "LEANING_FRAUD" | "BORDERLINE" | "LEGITIMATE"
- "loss_prevention_priority": Queue priority for asset recovery team. Allowed: "IMMEDIATE_P0" | "STANDARD_QUEUE" | "LOW_PRIORITY"
- "account_takeover_suspected": Whether cardholder credentials were compromised. Allowed: true | false<|im_end|>
<|im_start|>user
Input:
TRANSACTION ALERT #TX-98421
Timestamp: 2026-09-15 03:14:22 UTC
Cardholder: Sarah Jenkins (Account ID: US-99120)
Transaction Amount: $14,850.00 USD
Merchant: CyberGold Crypto Exchange (Nicosia, Cyprus)
Cardholder Billing Address: Seattle, WA, USA
Device IP: 185.220.101.5 (Known Tor Exit Node, Frankfurt, Germany)
Device Fingerprint: Unrecognized Linux Firefox 128.0 (First seen 4 minutes ago)
Velocity Check: 3 previous transactions attempted in past 10 minutes from Singapore ($1), London ($1), and Frankfurt ($14,850).
Cardholder Typical Activity: Average monthly spend $1,200. No prior cryptocurrency, wire transfer, or foreign cross-border transactions recorded in 3 years.
Current Account Balance: $15,200.00 USD.
Two-Factor Authentication: SMS 2FA prompt bypassed via Session Hijacking / Cookie Injection anomaly.<|im_end|>
<|im_start|>assistant
{

Everything up to and including Input: is the cached head, identical for every request on this schema. The alert is the only per-request part. A field's row is this whole prompt plus one short line: for the first key, "is_fraudulent":. Each finding below changes one thing about that line, or about the text in front of it.

Tokens are not characters, and this will ruin your day

The obvious way to score a boolean: tokenize the text up to the value, "sla_breach_risk":, append it to the prompt, and compare the probabilities of true and false.

This is wrong, and it is wrong by nearly five orders of magnitude.

Tokenize the whole line and you see why:

key part      │·│·"│sla│_b│reach│_r│isk│":│·│     (9 tokens)
whole line    │·│·"│sla│_b│reach│_r│isk│":│·true│,⏎│  (10 tokens)

In real JSON, the space after the colon belongs to the value: true is one token. Split the string in two and the key part ends with a lone space token, and the value becomes the space-less true. That pair essentially never occurs in text. You are asking the model about a string it has never seen.

Measured across the triage schema's 15 booleans: P(true) + P(false) came to a median of 1.5 × 10⁻⁵. After the dangling space, the model's favourite continuation was often a digit — a value starting after a separate space token looks like a number to it.

Do it the other way — tokenize the complete line for each candidate, "sla_breach_risk": true,⏎ and "sla_breach_risk": false,⏎, and cut where the two token sequences first diverge. They agree through ": and split at the ninth token, so the row ends one token shorter than the naive split and the things being compared are true and false, space attached, exactly as they appear in real JSON. Asked that way, the same question puts a median of 0.89 on the two.

The decisions were not random noise — renormalising two slivers still picks the same side most of the time, because the slivers carry some of the model's opinion. Three of fifteen flipped. The real damage is that the numbers stop meaning anything. A probability computed from 1e-5 of the mass cannot distinguish a confident answer from rounding error.

Sometimes the correct row really does end in a lone space — Qwen writes the space before a number as its own token, so for an integer field, ending on a space is right. That is why this is a rule and not a special case.

Rule: tokenize the complete line in context, and end the row where the tokenizer would.

Small models quote their booleans, and it cascades

With the boundary fixed, the next question is what counts as an answer. Ask Qwen2.5-1.5B to fill in "is_fraudulent": — the very first key of the fraud schema — and look at what it wants to write. Concretely, these are two rows in the same batch, each the full prompt above followed by:

<|im_start|>assistant
{
  "is_fraudulent":        ← row 1: what comes after the colon?
  "is_fraudulent": "      ← row 2: and if it opens a quote?

after "is_fraudulent":          after "is_fraudulent": "
         ·"  0.868                      true  0.868
     ·false  0.073                     false  0.074
      ·true  0.058                      TRUE  0.035

87% of its belief is on an opening quote. It wants to write "true", the string, not true, the boolean.

A decoder that only compares true against false sees 0.058 versus 0.073, renormalises, and answers False — for a transaction from a Tor exit node that bypassed 2FA. Follow the quote and the model is 87% sure of true.

The fix is to treat true and "true" as two spellings of one label rather than two labels. The quote is not a decision, it is a fork — so follow it and multiply along the path. Quoted true is 0.868 × 0.868 ≈ 0.753; add the bare 0.058 and True holds 0.812. False gets its bare 0.073 plus 0.868 × 0.074 ≈ 0.137. Renormalise those two and the answer is True at 0.855. Same model, same prompt, same batched pass — the quoted spelling is just one more query in the same row. Opposite answer.

And then it cascades. That field is the schema's first field, and its answer gets written into the prompt before the other 27 are decided. Get it wrong and 23 of 28 fields change — the model dutifully follows the false premise it was handed.

Why does the first field get quoted so heavily when later ones do not? Because there is nothing in front of it. Once a line like "is_fraudulent": true, is in the prompt, the model copies its style and the quote all but vanishes.

Rule: a label is a set of spellings. Follow every spelling and add them up.

The first key of a JSON object is a special position

That observation — the first key is different — goes deeper than quoting.

Without a fix, every row in the batched pass is appended straight after the {. Which means every field is being asked as if it were the first key of the object. For the actual first field, fine. For the other 27, it is a position the model rarely occupies while writing JSON.

It showed most clearly on the smallest model I tried, Qwen3-0.6B, on a small seven-field schema and a ticket that opens "Charged twice for my March invoice!!". Here is that prompt in full — note the empty think block that enable_thinking=False inserts, more on that later:

<|im_start|>system
You are a precise classification engine. Read the input and fill in a JSON object with exactly the keys listed below. Every value must be copied exactly from that key's allowed options.

Keys:
- "sentiment": Customer mood. Allowed: "POSITIVE" | "NEUTRAL" | "NEGATIVE"
- "topic": What the ticket is about. Allowed: "BILLING" | "TECHNICAL_ISSUE" | "FEATURE_REQUEST" | "ACCOUNT_ACCESS"
- "wants_refund": Whether the customer asks for money back. Allowed: true | false
- "is_spam": Whether the message is spam. Allowed: true | false
- "deadline": Deadline the customer gives. Allowed: "1_HOUR" | "12_HOURS" | "1_WEEK" | "NONE"
- "plan": Customer's plan. Allowed: "FREE" | "PREMIUM" | "ENTERPRISE"
- "route_to": Team that should handle it. Allowed: "BILLING_TEAM" | "ENGINEERING" | "SALES"<|im_end|>
<|im_start|>user
Input:
Subject: Charged twice for my March invoice!!

Hi, I was billed twice for my March invoice ($49 each). I am really annoyed, this is the third time I write about it. Please refund the duplicate charge within 12 hours or I will cancel. I'm on the Premium plan.
-- Dana<|im_end|>
<|im_start|>assistant
<think>

</think>

{

The test asks for topic two ways. Both rows share everything above; they differ only in what sits between the { and the question:

asked as the first key:          with one line in front:

{                                {
  "topic": "                       "sentiment": "NEGATIVE",
                                   "topic": "

And the answers:

asked as the first key:   topic: FEATURE_REQUEST 0.662, TECHNICAL_ISSUE 0.313, BILLING 0.026
with one line in front:   topic: BILLING 1.000

A double-billing complaint classified as a feature request, with 2.6% on BILLING. Put any line that looks like part of this schema's answer in front of it and BILLING wins — from 0.83 after a boolean key to near certainty after the schema's first key. That holds even when the answer in the line is wrong: "sentiment": "POSITIVE", for this furious customer, still gets BILLING to 0.999. Foreign lines are hit and miss: a customer name helps, a ticket ID or a language code leaves topic wrong.

So the engine decides the schema's first field alone, writes its answer into the prompt, and decides everything else after it. That is why a warm request is four forward passes rather than two, and it costs 6–17% more latency. Is it worth it? On Qwen2.5-1.5B, agreement with the model's own autoregressive answer went from 20 to 26 of 28 fields on the fraud schema and from 25 to 26 on the code security schema — and dropped by one field, 13 to 12, on triage. A clear net win, not a free one. (It also raises the stakes of the previous finding: the first answer is the one every other field reads.)

Rule: decide the first field alone and write its answer in before asking the rest.

Listing the allowed values is not optional

So far the question was malformed at the level of tokens and positions. This one is about the prompt, and it is the one I would have gotten wrong. Your prompt describes each field. Do you also need to list every allowed value? It roughly doubles the fixed part of the prompt.

Yes. It is the whole game.

Without the list, most enum fields have 3% of the model's belief on any allowed label. Nothing in the prompt told it the answer should be "SECURITY_OPS" rather than "Security", so it writes what it would naturally write — or copies a plausible-looking string straight out of the ticket. The decoder still returns a label, renormalised, often with probability near 1.0 — computed from a sliver of the model's actual belief. It looks confident and means nothing. Eight of 28 decisions changed.

Booleans survive, because true and false are the only natural ways to finish a JSON key-value pair. Some enums survive too: severity_tier holds up at 0.79 because the row already contains "SEV_, which shows the model the format.

This is exactly what candidate mass is for, and it is why you want it reported per field. On one fraud schema field, secondary_anomaly, candidate mass came in at 0.24 even with every value listed — and 64% of the model's belief was sitting on labels belonging to primary_anomaly, the field above it. The decoder happily reported a winner at probability 0.551. That 0.551 is a renormalisation of a quarter of the model's opinion, and without the mass number you would never know.

The cost of listing everything? The prompt head doubles (446 → 901 tokens) — but the head is identical for every request with the same schema, so its KV state is computed once and cached. Per request, listing every value costs about 1.06× the warm request time.

Rule: list every allowed value in the prompt, and report candidate mass next to every answer.

When one token is not enough

Everything so far assumed that a single token tells the labels apart. Usually it does. But target_resolution_hours allows both 1_HOUR and 12_HOURS, and both start with the token 1; assigned_agent_tier has TIER_1 and TIER_2, which agree up to the digit. In the race above, target_resolution_hours is the field with two dark cells — 29 decisions for 28 fields. (The model picked DIRECTOR for the agent tier, so that field needed only one.)

The fix is the same move as the quoted booleans: extend the row down the shared branch, still inside the same batched pass, and multiply. For target_resolution_hours, 1 is by far the likeliest first token, and one step later 2 beats _HOUR, so 12_HOURS wins.

You can take this all the way and score every label to the end of its line: add up how likely the model finds each of its tokens, so the tail of the label gets a vote too. I will call that exact scoring. On these schemas it rarely matters — it changed none of the 88 decisions, for about four times the rows — but it walks straight into a second version of the token-boundary trap, this time at the end of the line.

Qwen2.5 writes the end of a JSON line — closing quote, comma, newline — as a single token, ",⏎. Score a label's line without its newline and it ends in the bare token ", instead, which the model almost never writes. Across all 241 labels in the three schemas, that one token made each label look about 1,400× less likely than it really was.

If every label paid the same penalty, it would cancel out when you compare them. They don't: within a field the penalty typically differs by about 2× from one label to the next, and by up to ~18× at worst — enough to reorder close labels. And, as always, the answer is still a perfectly valid label.

Where the newline attaches also depends on the tokenizer:

• Qwen2.5, Qwen3, Llama 3 glue it to the punctuation before: ,⏎
• Gemma, Qwen3.5 make it a standalone token
• SmolLM2 glues it to the indentation after: ⏎·

Which means there is no single JSON layout that works everywhere. You have to lay the object out one way for Qwen2.5 ({\n then "key": value,\n) and the other way for SmolLM2 ({ then \n "key": value,) — and the way to find out which is to try both and check whether the tokenizer merges across the seam. Note that Qwen3.5 broke from its own family here: do not assume, probe.

Rule: when labels share a prefix, follow the branch. When you score a whole label, its line includes its ending — and probe the tokenizer for where that ending goes.

Your label names are a hyperparameter

Shared prefixes get extreme in the stress test: one field, a customs category out of 255 labels shaped like CAT_095_Aerospace_Titanium_Fasteners_&_Bolts.

Qwen spells numbers one digit per token. So all 255 labels share the prefix CAT_ and then diverge over three levels of digits. The model's very first decision on this field is between the tokens 0 (0.532) and 1 (0.455) — a coin flip over a digit that carries no meaning whatsoever.

I tried four ways of reading the field: following the likeliest digit at each level, following it until ten labels remain and then exact-scoring those, following every branch, and exact-scoring all 255 labels. They gave three different answers, none of them right.

Now drop the codes and keep the same 100 category names. The prompt shrinks to 32% of its length, the first token carries actual information, and all four decoders agree. (A 1.5B model still gets it wrong — it is a 100-way choice from a paragraph of text. But it is wrong consistently, with a probability that honestly says how unsure it is.)

The lesson generalises past this technique. I only tested dropping the codes entirely, but the same logic says Aerospace_Fasteners_095 should beat CAT_095_Aerospace_Fasteners. A model that must commit to digits before it reaches any semantics is being asked to guess.

Rule: if you control your label vocabulary, put the meaning first.

A bonus trap: the chat template

The last question-shaped trap is the prompt's wrapper. Qwen3 is a reasoning model. Unless you pass enable_thinking=False, its chat template leaves the assistant turn open, and the model wants to start thinking: it puts 99.8% on <think> at exactly the position where the engine writes {. (With the flag, the template inserts an empty think block and the problem goes away.) The engine overrides it, the JSON continues, and candidate mass looks perfectly healthy — 1.000.

Five of 28 decisions came out different.

Candidate mass only measures the decision point. It cannot see that the prompt as a whole is off-distribution. Nothing in the output flags it.

(Related, and mildly cursed: Llama 3.2's template writes today's date into the system prompt. Harmless for correctness — but it means your cached prompt head silently changes every midnight.)

Rule: render the prompt through the model's own chat template, with the arguments it expects.

The rules, in one place

If you build a parallel constrained decoder yourself, this is the checklist:

1. Tokenize each field's complete line in context; end the row where the tokenizer would.
2. Treat every spelling of a label (true, "true") as the same label, and add their paths.
3. Decide the first field alone, and write its answer in before deciding the rest.
4. List every allowed value in the prompt, and report candidate mass with every answer.
5. Follow shared prefixes; when scoring whole labels, include the line ending the tokenizer actually uses.
6. Put meaning before codes in label names.
7. Render through the model's own chat template, with the right arguments.

The bill: benchmarks against token-by-token generation

With every rule in place, back to the question the post started with: how much faster is it? Same test bench, same model (Qwen2.5-1.5B, 4-bit, MLX), on an M1 Pro with 16 GB, warm requests. "Token by token" is the same model writing the whole JSON object itself from the same prompt.

The shape of the win is clearer if you vary the number of fields. Token-by-token generation pays for roughly ten tokens per field, every one a sequential forward pass. Parallel decoding pays for one more row in a batch:

On PyTorch with Apple's GPU backend (mps) and a 0.5B model in bf16, the ratios are far larger — 12–14× on the 28-field schemas, and even the 255-label preset wins 1.9× — because the autoregressive loop there is dominated by per-step overhead that a batched pass does not pay.

Two things in that table are worth more than the speed column.

First: none of the autoregressive outputs matched their schema. Not one. Two stopped before the last key, two wrote booleans as strings, one invented a label that does not exist. Every parallel-decoded output matched, by construction. That is Jev's third claim — "never makes type errors" — and it falls straight out of the decoder.

Second: the last row. This is not a universal win.

Where it loses

Only closed decisions. Booleans and enums. Free text, numbers and nested objects need ordinary generation. This is the real boundary of the technique.

Few fields, huge label sets. The 255-label row is the worst case: four fields means there is almost no scaffolding to skip, and a 4,600-token prompt means every one of the batched rows drags a 127 MiB KV copy behind it. The model writes 47 tokens; on MLX, that is cheaper.

Fields in the same batch are independent. This is the fundamental trade. The parallel decoder asks each question in isolation. Autoregressive generation lets earlier answers influence later ones — sometimes helpfully, sometimes by dragging the model into a run of false. You can buy some of it back the same way the first field works: decide some fields first, write their answers into the prompt, then decide the fields that depend on them (each extra round costs two more forward passes). But if two fields must be consistent, derive one from the other in code. Telling the model the other field's answer informs it; it does not constrain it — on a double-billing ticket, with topic written in as BILLING, the model still routed it to SUPPORT at 0.56 rather than BILLING_TEAM at 0.17.

Memory scales with rows × prompt length. 27 rows on a 32 MiB prefix peaked at 1.3 GiB above the loaded model. The copies are free until the rows write into them, and then they are very much not free. There is a memory budget knob; going from 27 rows per pass down to 4 bought 3.8× less memory for 1.3× the decode time.

Not bit-exact across batch shapes. In fp16, how many rows share a batch moves the probabilities by up to about 0.01, purely because the GPU adds numbers up in a different order. No decision changed in testing, and in float32 the effect shrinks more than 500-fold. But "same answer as last time" is not a guarantee you get.

The probabilities are not calibrated. They are the model's belief renormalised over your labels — a 0.8 is not right 80% of the time. Use candidate mass to spot the questions the model didn't want to answer, and fit thresholds on real labelled data before you trust any of it. Which is exactly where Jev claims to be different.

So is this what Jev does?

Line the claims up against the reconstruction:

The row it does not explain is the interesting one. Out of the box, these probabilities are just the model's opinion — honest about coin flips, but not calibrated. TypeSafe says Jev is trained to make its probabilities mean something (their RLCD, "Reinforcement Learning for Calibrated Decisions"). If my guess is right, that is the real moat: the decoding trick is a few hundred lines, and a model trained to be calibrated at exactly the positions this decoder reads is not.

Training for this format would plausibly fix the problems that took up most of this post, too. A model trained to answer at exactly these positions should not quote its booleans, lose its footing at the first key, or need the tokenizer archaeology. That part I cannot test from the outside.

What I take away

The headline is the speedup, but the speedup is the least interesting part. Batching is batching.

What is interesting is that the constraint improves the question. Listing the allowed values moves the model's belief from 3% on-schema to 98% on-schema. Asking about one field in isolation gets you a probability distribution instead of a sampled guess. Reporting candidate mass tells you when the model wanted to answer something that was not on the menu — which is the single most useful signal here, and the one an autoregressive pipeline throws away.

And most of the actual engineering turned out to be neither decoding nor batching. It was tokenizer boundaries, spellings, chat templates and label naming. A missing newline token making every label look 1,400× less likely. A model quoting its booleans and flipping 23 fields. A schema's first key being a privileged position in the model's world.

That ratio — one paragraph of idea, six notebooks of details — is not unusual for inference work. It is just unusually well documented in this case.

Try it yourself

The implementation and all six notebooks behind this post are on GitHub: HiGal/parallel-constrained-decoding. It fills schemas for open decoder models on MLX or PyTorch — tested on Qwen2.5, Qwen3, Qwen3.5, Gemma 3 and LFM2.5 — and measures its own forward passes rather than assuming them. Timings are M1 Pro, 16 GB, and will differ on your hardware, but the ratios should not.

If you run it on a model or tokenizer I have not tried, or you get early access to Jev and can put the two side by side, I would love to see the numbers.

This is an independent reconstruction. I am not affiliated with TypeSafe and have no knowledge of Jev's internals beyond their public launch post; every claim about how Jev works is my inference. Numbers in this post were produced by the notebooks linked above.