For developers
Batch API: half-price model calls, results within 24 hours
Released
Short answer: The Smarty Pants Batch API runs work that can wait, such as evals, tagging or embedding a library, at half the normal per-token price, with results within 24 hours. It uses OpenAI's Batch API format, so the OpenAI SDK works unchanged. Each batch holds up to 10,000 requests for chat, embeddings, reranking or code jobs, and can check answers, stop at a spending cap, notify you when done and run on a schedule.
Plenty of AI work does not need an answer this second: grading a test set, tagging a catalog, summarizing an archive, embedding a document library. Smarty Pants runs that work on Macs that would otherwise sit idle, and charges half the normal per-token price for it.
How do I send a batch?
The same way as OpenAI’s Batch API: upload a JSONL file, create a batch, poll it, download the results.
import json, os, time
from openai import OpenAI
client = OpenAI(base_url="https://smartypants.network/v1", api_key=os.environ["SMARTYPANTS_API_KEY"])
lines = [{"custom_id": f"q{i}", "method": "POST", "url": "/v1/chat/completions",
"body": {"model": "smartypants/arete-27b", "messages": [{"role": "user", "content": q}], "max_tokens": 100}}
for i, q in enumerate(["Name a prime number.", "Name a colour."])]
open("batch.jsonl", "w").write("\n".join(json.dumps(l) for l in lines))
file = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=file.id, endpoint="/v1/chat/completions", completion_window="24h")
while batch.status not in ("completed", "failed", "expired", "cancelled"):
time.sleep(30); batch = client.batches.retrieve(batch.id)
print(client.files.content(batch.output_file_id).text)
No file? Send requests: [{custom_id, body}] inline to POST /v1/batches instead.
What can go in a batch?
| Endpoint | Price in a batch |
|---|---|
/v1/chat/completions |
Half the model’s per-token price |
/v1/embeddings |
Half the model’s per-token price |
/v1/rerank |
Half the model’s per-token price |
/v1/cpu/jobs (code jobs, where your account has them) |
Normal CPU price |
Up to 10,000 requests per batch and 10 unfinished batches per key.
What else can a batch do?
- Check answers:
checkwith required JSON fields, a schema or retries for chat, orexit_codefor code jobs. Each line sayspassedorneeds_review. - Cap spending:
max_spend_usdstops the batch at that cost. - Tell you when it is done:
notify_urlgets a signed POST. - Summarize and rerun:
GET /v1/batches/{id}/summarygives counts, cost and builds used;POST /v1/batches/{id}/rerunreruns only what failed. - Run on a schedule:
POST /v1/batch-scheduleswithevery_minutes(15 or more), or start a run from a webhook.
When is a batch the wrong choice?
When you need answers now, or a promise faster than 24 hours. Use normal requests for those. And as with all Smarty Pants work, keep sensitive data out of batches on the general network, or run them on your Group’s Macs.
See every price on the price page.
Frequently asked questions
How much cheaper is a batch?
Model calls in a batch cost half the normal per-token price. Code jobs in a batch cost the normal CPU price.
How fast do results come back?
Within 24 hours. Batches run on Macs that are otherwise idle, and live requests always come first, so most batches finish sooner but none is promised sooner.
Does the OpenAI SDK work?
Yes. Upload a JSONL file with client.files.create(purpose="batch"), then client.batches.create(...) with base_url set to https://smartypants.network/v1.
Can a batch check the answers?
Yes. Add check, for example required JSON fields or a small schema for chat, or exit_code 0 for code jobs. Each result line says passed or needs_review, and POST /v1/batches/{id}/rerun runs only the ones that failed or need review.
Can I cap what a batch spends?
Yes. max_spend_usd stops the batch once it has cost that much; work already running finishes.