When to Use an AI Batch API Instead of Real-Time Requests

Most AI API examples follow the same pattern: send a request, wait for the response, then move on. That works well when a person or application needs the result immediately.

But many jobs do not need to happen in real time. If you have thousands of reviews to classify, product descriptions to rewrite, documents to summarize, or test prompts to run, waiting for every response individually can add avoidable cost and coordination work.

That is where a Batch API fits. Instead of handling each request as an immediate conversation, you submit a file of independent requests, let them run in the background, and download the results later. On NanoGPT, supported batch routes are priced 50% lower than their normal synchronous equivalent.

What makes a job suitable for batching?

A workload is a good fit when:

• It contains many separate requests
• No request needs another request's result
• All requests can use the same model
• Nobody is waiting on an immediate response
• You can process the results later

Good examples include:

• Classifying customer feedback by topic or sentiment
• Summarizing a collection of documents
• Extracting the same set of details from many records
• Generating descriptions for a product catalog
• Running evaluations against a fixed prompt set
• Producing draft tags, titles, or metadata
• Describing or categorizing images provided as supported image_url content

The important word is independent. One request should not need the answer from another request in the same batch.

When real-time requests are better

Use normal API requests when:

• A user is waiting for the answer
• You want the response to stream as it is generated
• Each step depends on the previous result
• The model needs to call tools or functions
• A single batch would need more than one model
• The job uses an endpoint other than chat completions
• You need guaranteed schema-checked JSON output

A chatbot is the clearest example. Its next request depends on what the user says after seeing the previous answer, so putting an entire conversation into one batch would not make sense.

The same is true for many agents. If an agent must inspect a result, choose a tool, and decide what to do next, it needs an interactive request loop rather than a queue of independent prompts.

A batch is a list of normal requests

NanoGPT uses JSONL files for batch input. JSONL means one JSON object per line. Each line needs a unique custom_id, method: "POST", url: "/v1/chat/completions", and a body containing the model, messages, and an output limit.

For example, a two-request file could look like this:

{"custom_id":"review-001","method":"POST","url":"/v1/chat/completions","body":{"model":"your-supported-model","messages":[{"role":"user","content":"Classify this review as positive, neutral, or negative: The setup was easy and everything works well."}],"max_tokens":12}}
{"custom_id":"review-002","method":"POST","url":"/v1/chat/completions","body":{"model":"your-supported-model","messages":[{"role":"user","content":"Classify this review as positive, neutral, or negative: Delivery was late and the package was damaged."}],"max_tokens":12}}

Replace your-supported-model with a model currently available through the Batch API. Every row must use that same model and include max_tokens or max_completion_tokens, which caps how many tokens each response can use.

You do not combine all your source material into one enormous prompt. Each line remains its own request, with its own messages and result.

The practical NanoGPT workflow

The Batch API page handles the main workflow in one place:

1. Select the API key that should own the job.
2. Upload the JSONL file.
3. Create the batch.
4. Follow its status while it runs.
5. Download the output and error files when it finishes.

The output is also JSONL. Every completed line includes its original custom_id, so your script can reconnect each answer to the correct review, document, product, or test case.

You can cancel an active job, although cancellation may take time if work is already in progress.

Cost and timing

Batch jobs draw from your account balance. When you create one, NanoGPT checks and reserves enough balance for a conservative maximum estimate based on the input and each row's output limit. The final cost is based on actual completed usage, not the maximum estimate.

The only accepted completion_window value is 24h. A job may finish earlier, but batching is the wrong choice if a result is needed in the next few seconds or minutes.

Lower pricing does not automatically make a workload efficient. Keep prompts focused, choose a model appropriate for the task, and avoid requesting far more output than each row needs.

Using the API directly

For automated workflows, use https://api.nano-gpt.com/api/v1 for the calls that upload, create, poll, cancel, and download a batch. Inside the JSONL file, each row's url must remain exactly /v1/chat/completions.

The basic flow is:

1. Upload a JSONL file with purpose=batch.
2. Create a batch for /v1/chat/completions with completion_window set to 24h.
3. Poll the batch status or cancel it if necessary.
4. Download the output and error files.

Batch requests are non-streaming. Current batch rows support chat-completion text and compatible image_url content, but not tools, functions, structured-output modes such as response_format, audio, or arbitrary API endpoints. Check the current model support before building a large file, because not every NanoGPT model is available through the Batch API.

A simple decision rule

Use the normal API when the answer is part of a live interaction. Use the Batch API when you already have a list of independent jobs and can collect the answers later.

If the work can wait, batching runs the same per-item prompts asynchronously at the lower batch rate.

Open the NanoGPT Batch API page to upload and manage a job, or see the API documentation when integrating the workflow into your own application.