Claude CodeExperimentsBlogPortfolioAbout me

Experiments / Structured Output

Back to Experiments

Structured Output Reliability

Five ways to make a model return valid JSON, measured as schema complexity climbs. Parse rate lies: a response can be perfect JSON and still break your schema. What actually holds up under load, and what it costs in latency.

Flat schema, 5 fields

Extract 5 fields (name, email, company, role, seniority) from a messy contact blob. Types are strings and one bounded enum. This is the easy case: everyone does well, the gaps are small but they exist.

samples: n=1000model: claude-sonnet-4-6date: 2026-06
Method
Schema-valid
Retry rate
Latency
Notes
  • Naive promptprompt
    94.3%
    parses 96.2%
    5.8%
    baseline
    Baseline: 'return JSON'. Fails on markdown fences (```json), trailing prose after the object, and the odd trailing comma. Parses 96%, but 2pt more break the schema (wrong enum casing).
  • Prompt + 3 examplesprompt
    97.7%
    parses 98.5%
    2.3%
    +90ms
    Three examples kill most of the formatting noise. Overhead is the extra input tokens on every call. Still occasionally wraps the object in prose.
  • Forced tool calltool-use
    99.1%
    parses 99.7%
    0.9%
    +40ms
    tool_choice forces a single function call whose input_schema is the target. Structure is guaranteed by the API layer. Remaining misses are semantic (right shape, wrong enum value).
  • Native json_schemaschema
    99.9%
    parses 100.0%
    0.1%
    +150ms
    Strict response_format with the schema attached. Structurally impossible to return invalid JSON. Overhead is schema compilation on the provider side, amortized under caching.
  • Constrained grammargrammar
    100.0%
    parses 100.0%
    0.0%
    +320ms
    Self-hosted Llama-3.3-70B via vLLM + xgrammar. Token sampling is masked to the grammar, so every token is legal by construction. 100% valid, but the grammar-compile and masking cost real latency.
Takeaway

On easy schemas the naive prompt is already 94% valid, so the question is what the last 6% costs you. If a bad row means a failed enrichment, few-shot is enough. If it means a broken downstream job, force the structure at the API layer: tool-use gets you to 99% for almost no latency.

Nested schema: arrays, enums, optionals

Parse a support ticket into a nested object: an array of line-items, each with an enum status, an optional refund object, and a discriminated union for the channel. This is where prompt-only approaches start to bleed.

samples: n=1000model: claude-sonnet-4-6date: 2026-06
Method
Schema-valid
Retry rate
Latency
Notes
  • Naive promptprompt
    79.2%
    parses 88.3%
    20.8%
    baseline
    Falls apart on depth. Flattens nested arrays, emits null vs omits optionals inconsistently, and invents union tags. 1 in 5 needs a retry. This is the number that quietly kills prod pipelines.
  • Prompt + 3 examplesprompt
    89.7%
    parses 94.1%
    10.3%
    +140ms
    Examples help structure but cannot cover every branch of a union. 10% retry rate is still too high to run unattended without a repair loop.
  • Forced tool calltool-use
    97.2%
    parses 99.2%
    2.8%
    +55ms
    The API enforces nesting and required fields. Remaining 3% are union mis-tags and enum values that are plausible but not in the allowed set. Cheapest way to get to 'good enough'.
  • Native json_schemaschema
    99.7%
    parses 100.0%
    0.3%
    +180ms
    Strict mode with $defs and enum constraints. Nesting, unions and enums are all enforced during decoding. The 0.3% gap is semantic-only (valid shape, wrong choice).
  • Constrained grammargrammar
    99.9%
    parses 100.0%
    0.1%
    +480ms
    vLLM + xgrammar compiled from the same JSON Schema. Enum and union tags are structurally unreachable if invalid. Overhead grows with grammar size: deep schemas mean bigger masks per token.
Takeaway

Depth is where 'the model usually returns JSON' becomes a lie. Naive prompting drops to 79% schema-valid and a 21% retry rate: that is a pipeline that pages you at 3am. Constrained decoding (json_schema or grammar) is the only thing that makes nested output boring, which is exactly what you want in prod.

Strict types + 40-value enum + Cyrillic content

Classify and tag localized product feedback: a 40-value category enum, an integer score with bounds, and free-text fields that must preserve Cyrillic without escaping to \uXXXX. The enum is the trap: models love to invent a 41st category.

samples: n=1000model: claude-sonnet-4-6date: 2026-06
Method
Schema-valid
Retry rate
Latency
Notes
  • Naive promptprompt
    70.6%
    parses 91.1%
    29.4%
    baseline
    Parses often, conforms rarely. The gap (91% -> 71%) is almost entirely enum hallucination: the model picks a synonym not in the list. Also escapes Cyrillic to \u codes ~8% of the time.
  • Prompt + enum in systemprompt
    84.2%
    parses 95.7%
    15.8%
    +160ms
    Pasting all 40 values into the system prompt helps a lot but not enough: 16% still drift to near-synonyms under ambiguous input. Overhead is the enum tokens on every request.
  • Forced tool calltool-use
    93.8%
    parses 99.4%
    6.2%
    +60ms
    Enum lives in the tool schema, so structure holds. But the API does not hard-constrain enum membership during sampling, so ~6% still pick an out-of-set value. Good, not airtight.
  • Native json_schemaschema
    99.6%
    parses 100.0%
    0.4%
    +190ms
    Strict enum constraint means an out-of-set category is rejected at decode time. Cyrillic stays native. The 0.4% left are integer bounds violations on adversarial inputs.
  • Constrained grammargrammar
    100.0%
    parses 100.0%
    0.0%
    +520ms
    The enum becomes 40 literal alternatives in the grammar. An invalid category is not a low-probability token, it is an unreachable one. 100%, at the cost of the highest per-token overhead in the test.
Takeaway

Enums are the honest stress test. A model that returns 96% valid JSON can still be 71% schema-valid because it keeps inventing categories. If your enum drives routing or billing, only constrained decoding is safe: json_schema for hosted models, grammar when you self-host and need a hard 100%.