Cost, limits and caching

Prices are per million tokens. Input and output have separate prices.

cost = input tokens × input price + output tokens × output price

Here is an example with made-up prices. Real prices change often. A call sends 3,100 input tokens. The answer is 400 tokens. Input costs $3 per million. Output costs $15 per million. The call costs about 1.5 cents. At 50,000 calls a month, that is about $765.

Priya's bot sent a 6,000-token handbook with every message. It used the biggest model to reset passwords. It kept whole chats forever. A quick test with 3 messages showed none of this.

• Cached input costs a small part of the normal input price.
• Some providers charge extra the first time they save a start. So caching pays only when you reuse it.
• A cache lasts for minutes to hours. Each use usually keeps it alive longer.
• Very short prompts are not cached at all.
• Some providers cache on their own. Others cache only when you turn it on.

A good order is tools, then rules, then documents, then the chat, then the new question. Why a change near the top breaks the match is in Inside one answer.

Some jobs can wait, like sorting 20,000 old support tickets. You can send many requests at once and get the results later. This is a batch API. As of October 2026, big providers charge about half the normal price for batch work. Results come within 24 hours, often sooner. Don't use batch when a person is waiting for the answer.

Limits often count over short time windows. So 60 requests per minute can act like 1 per second. A quick burst can get a 429, even when the minute total is fine.

• Retry only errors that can work later. Most 429s and short server problems can. A 400 Bad Request will not.
• Some 429s say you are out of credit or over your spend limit. Waiting will not fix these. Stop and tell someone.
• If the error has a retry-after header (a note that comes with the reply), wait that long.
• If not, double the wait limit each time, up to a cap. Then wait a random time up to that limit.
• Stop after a set number of tries. Some providers count failed calls, too.

Official SDKs often retry for you. Check what yours does first.

• Use a smaller model for easy work. Test that quality holds (Choosing a model).
• Trim old chat. Drop the wrong part, and answers get worse (Context and memory).
• Cap the answer length with max_tokens. Too low, and answers stop mid-sentence.
• Set a monthly spend limit and alerts. A hard cap stops the service when you reach it.
• Use one API key per project. Log the usage numbers that come back with each answer.

Loops cost the most. An agent with no step limit can make thousands of calls (Agents, subagents and skills). So can a retry loop with no maximum. Give every loop a cap and a budget.