You cut the cost of an LLM application with three techniques. Prompt caching takes up to 90% off input tokens you repeat. Prompt batching takes 50% off work that does not need an instant reply. Intelligent routing sends simple queries to cheap models. Together they lower the bill without a visible drop in quality.
Enterprise spend on LLM infrastructure keeps climbing, and much of it is waste. That waste rarely sits in the model itself. It sits in context reprocessed a hundred times, in background jobs pushed through the expensive real-time endpoint, and in a frontier model handling a simple intent classification. This article covers the three patterns that fix it.
What is LLM cost optimization?
LLM cost optimization is the systematic reduction of the compute cost, token consumption, and latency of a language model application, without compromising output quality. Commercial APIs price per million tokens, with a clear asymmetry: output tokens typically cost five to eight times more than input tokens.
Optimizing comes down to three decisions: how you structure your data, when you process it, and which model handles the request. A hard token cap does not solve that, it just makes your application less useful. The win is in execution, not in the brake pedal.
What is prompt caching and what does it save?
Prompt caching stores previously processed text in the provider’s memory so a later request reads those tokens instead of recomputing them. If you send the same long context repeatedly, you pay full price only once.
How prompt caching works
On each request, the provider checks whether the leading portion of your prompt matches a prefix it processed recently. That prefix usually holds your system instructions, tool definitions, conversation history, and grounding data. On a match, the system reads the stored key-value representations straight from memory. Providers route on a hash of the leading tokens, and a cache hit requires an exact prefix match.
One hard design rule follows from that: static content goes first, variable input goes last. Change a single word near the start and the entire cache behind it is invalidated. Always place the user’s query last.
What OpenAI, Anthropic and Google charge
The three major providers apply different thresholds and pricing models. The figures below come from their documentation, current as of July 2026.
| Provider | Activation | Minimum | Cache read discount | Cache write cost | Lifetime |
|---|---|---|---|---|---|
| OpenAI | Automatic | 1,024 tokens | 90% | 1.25x input rate (GPT-5.6 and later) | 30 minutes (GPT-5.6 and later) |
| Anthropic Claude | Explicit (cache_control) | 512 tokens (Opus 5), 1,024 (Sonnet 5), 4,096 (Haiku 4.5) | 90% | 1.25x at 5 minutes, 2x at 1 hour | 5 minutes or 1 hour |
| Google Gemini | Implicit by default | 2,048 tokens (Gemini 2.5), 4,096 (Gemini 3.5 Flash and 3.1 Pro) | Varies, not guaranteed | No premium, but storage is billed | 1 hour default, configurable |
OpenAI activates caching automatically from 1,024 tokens and bills cached input at one tenth of the standard rate. From GPT-5.6 onward, retention is 30 minutes and cache writes cost 1.25x the input rate.
Anthropic requires an explicit cache_control marker inside the content block array. Cache reads cost 0.1x the input rate. Cache writes cost 1.25x at the five-minute default and 2x if you extend retention to one hour. That write premium only pays back on reuse inside the window.
Google Gemini enables implicit caching by default on Gemini 2.5 and newer, with no guaranteed discount. The threshold is 2,048 tokens on Gemini 2.5 and 4,096 on Gemini 3.5 Flash and 3.1 Pro. For certainty you create an explicit cache object with its own Time-To-Live, where your API surface supports it. That adds a cost line the other two do not have: storage duration.
{
"role": "user",
"content": [
{
"type": "text",
"text": "System instructions or grounding text...",
"cache_control": {"type": "ephemeral"}
}
]
}What is prompt batching and when do you use it?
Prompt batching bundles multiple requests into a single file processed asynchronously, usually within 24 hours. You give up real-time responsiveness and get 50% off in return, on both input and output tokens.
The tradeoff: 50% off against wait time
Batching is unfit for anything a user is waiting on. Chat interfaces, search features, and human-in-the-loop agents are out. What remains is background work, often a bigger share of your consumption than you would guess.
You submit requests as a JSON Lines file (.jsonl) to a dedicated batch endpoint. The OpenAI Batch API takes up to 50,000 requests or 200 MB per batch. The Anthropic Message Batches API goes to 100,000 requests or 256 MB, and most batches there finish in under an hour. Results stay available for 29 days.
A second benefit is often overlooked: batch endpoints run on their own rate limits, separate from your standard quota. A heavy background job no longer eats your tokens per minute, and production does not fall over during a large processing run.
What to use batching for
- Document processing at scale. Contract archives, field extraction from tens of thousands of invoices, entity recognition across a legacy data store.
- Evaluation pipelines. Regression tests, prompt versions benchmarked against thousands of validation samples, historical logs run through quality checks.
- Synthetic data generation. Fine-tuning data, simulated user interactions, content variants for a product catalog.
What is intelligent LLM routing?
Intelligent LLM routing dispatches incoming prompts across different models dynamically, based on an automated assessment of complexity and required reasoning. Instead of sending all traffic to one expensive model, a traffic controller sits between.
How your traffic breaks down
Routing frameworks such as RouteLLM and FrugalGPT show that production traffic splits into clear complexity tiers. A rough but usable distribution:
- Around 70% is simple. Intent classification, text rewriting, keyword extraction, lookups. A small model handles this fine.
- Around 20% is mid-tier. Tracking context across turns, output against a strict JSON schema, synthesis across multiple documents.
- Around 10% is genuinely complex. Agentic loops, code generation, mathematical or legal validation.
Without a routing layer you pay the frontier rate on that first 70%. RouteLLM reports over 85% cost reduction on MT-Bench while retaining 95% of GPT-4’s quality. On MMLU and GSM8K the gain is smaller, 45% and 35% respectively. The lesson: how much routing returns depends heavily on your task mix, so measure it on your own traffic.
Routing methods
Deterministic routing reads metadata: the user’s plan tier, the module making the call, or a regex in the prompt. Rule-based, predictable, and near-zero latency.
Semantic routing converts the prompt into a vector and compares it against predefined cluster centroids representing complexity classes. The nearest class picks the model.
Predictive routing deploys a small, fast classifier that scores the query before execution. Vendors such as Orq.ai report under 40 milliseconds of added latency for this. In practice that delay often disappears against the faster time-to-first-token of the model it selects.
The cost-quality frontier
Routing is a dial, not a switch: you set how aggressively it optimizes for cost. The figures below come from vendor Orq.ai and deserve validation on your own workload, but the shape of the curve holds.
| Premium model share | Quality retained | Cost reduction |
|---|---|---|
| 75% | around 99.5% | around 25% |
| 50% | around 98% | around 50% |
| 25% | around 95% | around 70% |
The middle point is the best trade for most deployments: you halve model cost and give up roughly two percentage points of quality.
Centralized control through an AI gateway
You do not want routing rules hardcoded across ten microservices. That is what AI gateways are for: a central layer intercepting all your LLM traffic. Open source options such as LiteLLM and Helicone add logging, virtual API keys, and hard budget ceilings.
The critical piece is fallback logic. If a cheap model returns a low confidence score or a rate-limit error, the gateway escalates the request to a heavier model. The user never sees a failure.
Prompt caching, batching and routing compared
Each technique targets a different form of waste. They do not conflict.
| Dimension | Prompt caching | Prompt batching | Intelligent routing |
|---|---|---|---|
| Average saving | 50% to 90% on input tokens | 50% on all tokens | 25% to 70% of total spend |
| Latency impact | Up to 80% faster on a cache hit | Up to 24 hours of delay | Around 40 ms added |
| Where you build it | Prompt structure and API call | Asynchronous background scripts | Gateway or infrastructure layer |
| Main constraint | Requires an exact prefix match | Unusable for real-time | Needs continuous quality-drift monitoring |
| Best fit | Long system prompts, chat, RAG | Bulk processing, evals, log analysis | Mixed-complexity endpoints |
How to combine all three in one architecture
Treat them not as a menu but as four filters every request passes in sequence.
The request first hits a semantic cache. This differs from prompt caching: instead of an exact token prefix, it matches on meaning. Each query becomes a vector and is compared against earlier queries, so a differently worded question with the same intent still counts as a hit. Above a set threshold the user gets the stored answer and no model call goes out, removing both input and output cost.
On a miss, the next question is: does this need to be immediate? If not, the request goes to the Batch API queue and earns 50%.
If it does need to be immediate, the router picks the cheapest model capable of the task. And right before dispatch, your prompt construction puts static blocks at the front, so the next identical call still lands the caching discount.
Want to know what this means for your spend? An AI assessment maps your consumption, and an AI demo shows these layers in action. For engineering teams an AI workshop is the fastest route to practice, and AI consultancy helps set up the gateway.
Conclusion
You control LLM cost with architecture, not with a usage cap. Prompt caching removes the cost of repeated context, which adds up fast with RAG and long system prompts. Batching returns 50% on everything that can wait. Routing reserves expensive models for queries that need them. Combine all three and you scale generative AI without the bill scaling with it.
Frequently asked questions (FAQ’s)
Can prompt caching be combined with intelligent LLM routing?
Yes. When combined within an AI gateway architecture, the routing engine first identifies the appropriate model tier for the request. Once the model is chosen, the prompt is structured with its static context at the beginning of the sequence. This ensures that if that specific model tier encounters the same prompt prefix in future calls, it triggers the provider-level prompt caching discount, compounding the savings.
What is the minimum prompt length required to trigger prompt caching discounts?
Minimum requirements vary by model provider. OpenAI automatically activates prompt caching on prompts of 1,024 tokens or more. Anthropic Claude allows caching on smaller blocks, but because writing to the cache requires an initial price premium (1.25x to 2.0x base input costs), the technique becomes financially viable mainly when the cached prefix is large and is reused multiple times within the cache’s retention window.
Does intelligent routing affect user-perceived latency significantly?
Generally no. Advanced classification systems and semantic routers add a small operational latency overhead, reported by vendors at well under 40 milliseconds. In production setups, this minor delay is often neutralized because the router frequently shifts queries away from large, slower frontier models to smaller, more agile model variants that feature faster output token generation speeds.
How does the Batch API handle rate limits compared to standard real-time endpoints?
The Batch API operates outside of standard real-time rate limit quotas. While standard endpoints are restricted by rigid requests-per-minute (RPM) and tokens-per-minute (TPM) ceilings to protect server availability, batch endpoints allow organizations to queue very large volumes of tokens at once. This minimizes rate-limit errors and avoids disrupting concurrent real-time operations.
