Imagine a catalog team enriching product descriptions overnight. The model requests are independent, nobody needs an immediate answer, and a lower inference bill sounds like an uncomplicated win. The team submits a batch and moves on.

By morning, some requests have failed. A few products have been edited since submission. One has been withdrawn. The output file contains useful responses, but importing every response would overwrite newer work or revive something that should stay unpublished.

The model did not necessarily make a mistake. The application failed to decide what a delayed answer means.

Batch inference is worth adopting. But the important architectural change is not the endpoint. It is admitting that some AI work belongs in a different product lane, with different deadlines and a separate decision about whether its output is still usable.

Start with the deadline, not the discount

OpenAI's Batch API guide advertises a 50% cost discount compared with synchronous APIs and a 24-hour processing window. It identifies evaluations, dataset classification and repository embeddings as suitable workloads.

Those are compelling candidates because the user is not waiting on each individual response. But “not interactive” does not automatically mean “safe to defer.” A document classification job might be useful tomorrow. A classification that decides whether to block a live transaction might not.

Before moving a workload, define when its result stops being useful, how much source data can change while it runs, and what happens if only part of the job succeeds. A business deadline should include time to retrieve, validate and apply results, not merely time for the provider to generate them.

Independent work fits this lane best. An agent that must inspect one tool response before choosing its next action is not equivalent to a file of unrelated prompts. Batching the wrong dependency structure can turn cheap inference into a complicated waiting room.

A finished batch is not a successful dataset

The provider's job lifecycle is only part of the story.

Anthropic's Message Batches documentation describes independent requests with four possible result types: succeeded, errored, canceled and expired. Processing ending does not mean every request succeeded. OpenAI likewise documents expired batches in which completed responses remain available while unfinished requests are canceled.

Treat those outcomes as records to reconcile, not a single green checkmark on a dashboard. For every submitted item, the application should eventually know whether it received an acceptable response, encountered a retryable failure, requires human review, or deliberately abandoned the work.

Both providers warn that result order may differ from submission order and direct developers to match records using custom_id. Pairing input and output by line number is therefore not a harmless shortcut. It can attach a perfectly reasonable answer to the wrong source record.

An identifier solves correlation. It does not, by itself, prevent the application from applying the same answer twice.

Build an acceptance step between generation and writeback

A useful design is to separate three events: the provider completed a request, the application accepted the answer, and the application committed an update. Those events should not collapse into one import operation.

Start with an immutable submission manifest. Record an opaque item identifier, the source revision, the prompt and output-schema versions, the requested model configuration, and the deadline. Keep enough information to explain what was submitted without scattering sensitive payloads across operational logs.

Then reconcile results against that manifest. Reject unknown identifiers, surface missing ones and investigate duplicates. Check both provider-level success and application-level validity. A response can satisfy a schema while still lacking the evidence or required fields for the intended use.

Next, compare the submitted source revision with current application state. If the product description changed overnight, the old enrichment should not silently replace it. Depending on the workflow, the right outcome could be a fresh request, a review task or an explicitly discarded result.

Finally, make writeback idempotent at the application boundary. A uniqueness rule or conditional update should prevent a repeated import from repeating the same business effect. The exact mechanism depends on the datastore, but the responsibility does not belong to the model.

These are application design recommendations, not promises supplied by a batch API. Their purpose is to preserve the meaning of work across a delay.

Retry the missing work, not the whole night

A partial failure invites a simple but expensive reaction: resubmit everything.

That approach can regenerate accepted results, create competing versions and complicate writeback. Instead, reconcile first. Identify which items genuinely need another attempt, which inputs have changed and which errors require fixing the request rather than trying again.

Cancellation needs the same care. Anthropic documents that cancellation may leave partial results. OpenAI describes a canceling state while in-flight work completes. Neither behavior should be treated as an instantaneous rollback of your application.

If an operator cancels a catalog enrichment job, distinguish “stop generating more answers” from “do not apply any further answers.” The latter is an application decision that the acceptance step must enforce, including for responses already generated but not yet imported.

Provider compatibility is part of the design

Do not assume the synchronous and batch surfaces support identical features.

Amazon Bedrock's batch inference guide describes asynchronous jobs using input and output files in S3. Its current documentation also makes an important invocation-specific distinction: batch jobs using the Converse format do not support tool calling through toolConfig; the documented tool-calling path uses InvokeModel with the model's native request format.

That is not a universal rule about batch inference. It is a reminder to verify the actual model, invocation format, region and feature combination before migrating a workflow. A prompt that works interactively is not sufficient proof that its surrounding request will work unchanged in bulk.

Uploaded inputs and downloadable outputs also need deliberate access and retention policies. Moving work off the live path does not make its data less sensitive.

Measure accepted output, not submitted volume

A useful operating view should distinguish submitted, provider-completed, validated, stale, applied and unresolved items. Otherwise, a high completion count can conceal an unusable backfill.

Track cost per accepted result, elapsed time until usable output, deadline misses and the effort spent reconciling exceptions. A token discount is real, but it does not account for rebuilding expired work or reviewing preventable mismatches.

Start with a bounded, low-risk workload whose source revisions are easy to identify. Run the acceptance step without automatic writeback first. Confirm that reordered results, partial failures, duplicate imports and changed records produce the outcomes you intended.

Then automate the safe cases.

The payoff is bigger than a cheaper API call: interactive features stop carrying work that never needed an immediate answer. But that separation only holds if the delayed lane has its own definition of done. A generated answer is an artifact. A result that is still relevant, validated and safely applied is the product.

Sources