Valid JSON Can Still Be Wrong: Building Trustworthy AI Extraction

A customer asks to reschedule a service visit. An AI extractor returns a tidy object: a customer identifier, a requested date, a location, and an action. Every required field is present. Every type is correct. The downstream system accepts it.

There is only one problem: the date came from a quoted message about an earlier appointment.

Nothing broke at the JSON boundary. The failure happened between the source material and its interpretation. That is the distinction teams need to preserve as structured model outputs become a normal part of application development: schema compliance is an interface guarantee, not evidence that the extracted facts are right.

What structured outputs actually buy you

Structured outputs are a meaningful improvement over asking a model to “please return JSON.” They let applications specify an expected shape and reduce the brittle parsing and repair logic around generated responses.

OpenAI's documentation describes schema adherence and typed SDK helpers. Anthropic's documentation describes constrained decoding for JSON outputs and a complementary strict-tool-use feature. These are useful engineering capabilities, not cosmetic formatting options.

But the boundary matters. Providers support particular schema subsets, and applications must handle unsuccessful outcomes. OpenAI explicitly discusses refusals, incomplete responses, and token-limit cases. It also states plainly that structured outputs can still contain mistakes.

A date can be a valid string and refer to the wrong event. An identifier can have the expected format and belong to the wrong customer. A permitted action can be unsupported by the message being processed.

The practical benefit is a more dependable interface. The dangerous extrapolation is treating that interface as a truth detector.

Separate the checks that answer different questions

A useful extraction pipeline asks several questions rather than collapsing everything into one green “validated” badge.

Structure: Does the candidate satisfy the supported schema? Are required fields present? Are unexpected fields rejected where appropriate? The JSON Schema object reference explains that declaring properties does not, by itself, require them. General JSON Schema also permits additional properties by default. Model APIs may impose stricter requirements; check their supported subset rather than copying a schema blindly.

Internal consistency: Do the fields make sense together? If a record includes a start and end time, is their order valid? If an action requires a destination, is one available? These checks belong in ordinary application code. Pydantic's validators support custom constraints at both field and model levels.

Evidence support: Does the supplied material actually support each important value? An internally consistent object can still be invented or assembled from the wrong passage.

Current authority: Does the referenced record still exist, belong to the correct scope, and permit the proposed operation? Extraction should not silently substitute for authorization or a current-state lookup.

These are different failure classes. Keeping them separate makes repair less arbitrary.

Give missing information somewhere honest to go

A schema can accidentally pressure a model to fill every box even when the source contains no answer.

OpenAI's guide specifically warns that unrelated input can lead to hallucinated content as the model tries to satisfy the schema. Instructions should explain how to handle material incompatible with the task. The data model should make that instruction possible.

For the service-request example, distinguish a missing date from an ambiguous date. A nullable value can represent absence where the provider's schema subset supports it. An explicit review state can represent conflicting evidence. An unrelated message should have a supported non-actionable outcome, rather than being squeezed into a service request.

These are application-design choices, not universal provider fields. The important principle is that a required field must not become an invitation to guess. If the application needs a value before proceeding, missing evidence should block that step, not manufacture certainty.

Make evidence inspectable, then check it

For consequential fields, ask the extractor to return a source reference alongside the proposed value. Depending on the input, that might be a message identifier, document section, or quoted passage.

Do not stop there. A generated reference can be wrong too.

The application should verify that the referenced source exists in the authorized input and that any quoted text matches it. Where rules are deterministic, use code: resolve an identifier against the allowed records, normalize an explicitly stated date, or reject a source reference outside the provided document set.

Other cases remain interpretive. A quoted date may be real but describe a canceled visit. Two messages may disagree about the location. Those cases need contextual evaluation or human review; a substring match cannot settle them.

Evidence pointers are therefore a review aid and a validation input. They are not proof simply because the model supplied them.

Repair the right problem

A common recovery pattern is to send every rejected object back to the model with “try again.” That treats missing information, stale state, and formatting problems as interchangeable.

Use the failure category to select the next step instead.

• An incomplete response needs explicit incomplete-response handling, not a fabricated default.
• A missing source value needs more authorized information or a review path.
• A stale customer record needs a fresh authoritative lookup.
• A cross-field contradiction needs a clear rejection reason and, where useful, a bounded correction attempt.
• An authorization failure needs denial or an approved escalation, not another prompt.

Keep the generated candidate separate from the accepted application record. Only validated, permitted data should cross the write boundary. Where state can change between lookup and write, enforce the relevant preconditions at commit time too.

This architecture does not require an elaborate agent framework. A typed candidate object, a small set of deterministic checks, and an explicit review queue can be enough to start.

Test the objects that look acceptable

Malformed JSON is an obvious test case. The more revealing cases are perfectly shaped objects with the wrong meaning.

Build fixtures containing quoted historical messages, conflicting dates, unfamiliar identifiers, unrelated attachments, missing values, and records that change before submission. Test both the extractor and the application's decision about whether to accept its candidate.

Track schema success separately from evidence-supported acceptance. Track review outcomes and incorrect writes separately from both. A rising parse-success rate should not conceal a growing queue of plausible but unsupported records.

Start with one narrow workflow and a handful of fields whose correctness can be checked against known examples. Expand only after the acceptance logic behaves well on deliberately difficult inputs.

Structured outputs remove a real source of integration pain. That is worth using. The next engineering step is making sure a clean object has earned its place in the system of record.