Token optimization: How to reduce token usage and costs

Token optimization means reducing the amount of text a language model processes while keeping the information it needs to answer correctly. That includes the instructions, conversation history and documents sent with each question.

The goal is to lower the cost of getting a correct answer. A shorter prompt may cost less, but those savings can disappear if missing information leads to mistakes and retries. To measure the benefit, divide the cost of all attempts by the number of tasks completed correctly.

The hidden cost of unnecessary tokens

Imagine an AI-assistant that answers questions about a company’s internal documents. A user has spent an hour discussing three long reports. Then they ask a short follow-up: “Does the second report reach the same conclusion as the first?”

To answer, the assistant needs to know which reports the user means and find the relevant passages in both. But many applications sends the model everything again: its instructions, the entire conversation, all three reports and earlier search results.

In this hypothetical example, that adds up to about 80,000 input tokens, even though the answer depends on just two passages and enough conversation history to identify the reports.

Sending all that material has a cost. Input tokens are billed, although cached content may receive a lower rate. Large inputs can also take longer to process and leave less room for new information. And irrelevant or conflicting details can make it harder for the model to find the evidence it needs, increasing the risk of an incorrect answer.

Cutting is the obvious fix, but blind cutting creates new problems. Remove the earlier messages and the assistant no longer knows which report is “the second one”. Cut the reports too hard and it answers from the wrong section. Retries can then erase the savings from the cheaper request. The job is to send the right tokens, and only those.

What tokens are

Tokens are the small pieces of text a model reads and writes. A token can be a whole word, part of a word or a punctuation mark. In English, 100 tokens is roughly 75 words. Our explainer on tokens in AI covers the basics.

The exact count depends on the model. Anthropic notes that its Claude 4.7 and later models produce about 30 percent more tokens for the same text than earlier ones. So a count measured on one model does not carry over to the next.

Providers count and bill tokens in a few groups:

• Input tokens are everything you send: instructions, history, documents and tool results.
• Output tokens are what the model writes back. They usually cost more. On Anthropic’s API price list in October 2026, output costs five times as much as input for every current Claude model.
• Cached input tokens are input the provider already processed for an earlier request that started the same way. They cost a fraction of the normal rate.
• Reasoning tokens are the hidden working steps of reasoning models. OpenAI’s documentation says they take up room in the context window and are billed as output. They are already part of the output count, so do not add them twice.

The visual above shows a second point. Most tokens in a request never appear on screen. The user sees one short question and one answer, while the model receives everything behind it.

What prompt engineering is and why it matters

Prompt engineering is the practice of writing and testing the instructions, examples and rules you give a language model, so it does a task reliably. It is where token optimization starts, because the system prompt travels with every call.

A well-designed prompt can cut unnecessary instructions, keep answers appropriately concise and prevent retries. It may also need more examples or detail to work reliably, so measure savings across the whole task, not just the prompt.

Prompt engineering covers the text you write. Context engineering covers everything the application adds while it runs, such as history, documents and tool results. In most applications, that second part holds the most tokens, so this article covers both.

Most system prompts grow over time. Someone adds a line after each problem, and nobody removes the old ones. Here is a typical result for the document assistant:

You are a helpful, friendly and very knowledgeable AI assistant for our company. It is very important that you are always accurate. You will be given documents and you need to answer questions about them. Please read all of the documents very carefully before answering. Try to be thorough but also concise, and cover everything that might be relevant. Always cite your sources. Do not make anything up. If you are not sure about something, say so. Please format your answers nicely. Remember: accuracy is the most important thing, so never make things up and always cite the documents.

And here is a cleaned-up version:

You answer employee questions about the document excerpts provided below.

Rules:
- Use only the excerpts. If they do not answer the question, say what is missing.
- Cite every claim as [document, section].
- Flag any inconsistency between excerpts that affects the answer.

Format: one sentence with the direct answer, then at most three supporting points. Maximum 150 words.

Example:
Q: How much notice does an employee need to give?
A: One month [Employee Handbook, 4.2].
- Managers need to give two months [Employee Handbook, 4.3].

The second prompt is more explicit and less repetitive. It replaces repeated warnings with a word limit, a citation format and one example. It has fewer words, but words are not tokens, so check both versions with your own model’s tokenizer before you count it as a saving. It also tells the model what to do when the excerpts do not contain the answer. Because it changes the task as well as the length, test it before you switch.

Anthropic’s engineering team adds a useful warning: a minimal prompt is not always a short one. The model still needs enough guidance to behave as intended. A few varied examples usually work better than a long list of edge cases.

Measuring before you optimize

You can only tell whether a change helped if you know where you started. Before you edit anything, record a baseline on a set of real tasks. Track these numbers:

• Input and output tokens per call, split into system prompt, history, documents and tool results.
• Cost per successful task, counting every attempt, including failed tasks and retries.
• Response time, both to the first word and to the complete answer.
• Answer quality, scored against a reference answer or a checklist.
• Retries and repair requests, from users or from code that rejects the output.
• The cost of the optimization itself, such as extra summarization calls.

State whether you are measuring model API costs or total workflow costs. For total workflow cost, include retrieval, reranking, paid tools and human review where relevant.

OpenAI and Anthropic both offer token-counting endpoints if you want an estimate before you send a request.

Techniques for minimizing token counts

Six techniques cover the main sources of avoidable tokens. Anthropic describes the goal as the “smallest possible set of high-signal tokens” that makes the outcome you want most likely. The table shows where each technique saves tokens and what to watch.

Technique	Where it saves tokens	What to watch
Write compact, explicit instructions	System prompt and tool definitions	Dropping a rule or example the model needs
Keep only the conversation that still matters	Conversation history	Losing decisions, numbers or exceptions
Retrieve passages instead of whole documents	Retrieved content	Missing the passage that changes the answer
Filter tool results	Tool output	Removing a field a later step needs
Control output length and reasoning	Output and reasoning tokens	Answers that stop early or lose quality
Use code and short hand-offs	Whole calls and multi-step workflows	Tool calls and sub-agents that still use tokens

Write compact, explicit instructions

Apply the prompt cleanup above to every fixed part of the request. Say each rule once, replace vague words with concrete limits and keep only the examples that earn their place. Tool definitions count too. On Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, enabling tools with the default settings adds 286 tokens of built-in instructions. Your own tool definitions come on top, so load only the tools a task needs.

Keep only the conversation that still matters

Most conversations do not need every earlier message word for word. Keep the last few messages in full and replace the rest with a summary.

A summary is a lossy copy, so decide what it must never lose. Keep decisions (“use the 2026 figures”), constraints (“leave out the pilot region”), open questions and the list of documents in play in a separate state block. That list is also what lets the assistant work out which report “the second one” is. Store exact values exactly, because a summary that turns “60 days” into “about two months” has changed the answer.

The research supports focused context. In Chroma’s 2025 study of 18 models, the models tested on a chat-history task scored significantly better with only the relevant parts of the history. That focused input was about 300 tokens, against about 113,000 for the full history. There is a catch, though. The focused versions were built from the dataset’s own labels, so the right passages were handed over in advance. A real application still has to find them.

Retrieve passages instead of whole documents

Sending three full reports to answer one question spends most of the budget on sections nobody asked about. Retrieval sends only the passages that match the question. Tag each passage with its document and section, so the model can cite it. Our guide to retrieval-augmented generation explains how this works.

Follow what a passage depends on, such as a later update, a definition or a cross-reference. And retrieve generously at first. A missing passage costs more than a few extra ones.

Filter tool results

Tool output is easy to overlook, because nobody writes it by hand. Imagine a search tool that returns 40 fields per result, most of them metadata. An assistant that calls it five times carries all of that into every later message. Keep only the fields the task needs, limit the number of results and remove old results once they have been used. Anthropic’s guide calls clearing old tool results one of the safest ways to shrink the context.

Control output length and reasoning

Output usually costs more than input, and it drives most of the waiting time. OpenAI’s latency guide gives a rule of thumb. Halving the output may cut latency by about half, while halving the prompt may gain only 1 to 5 percent. The guide makes an exception for very large contexts, where input length matters more.

Ask for the length you need, and match it to the task. A yes-or-no check needs one sentence. Tell the model to start with the answer and skip the recap at the end. When a program reads the result, choose a compact format, but measure it. JSON, for example, repeats every key for every item, so a long list can cost more tokens as JSON than as plain lines.

Set output limits with care. A limit stops runaway answers, but a tight one cuts answers off, and repairing them means resending the whole context. With reasoning models the limit usually includes the hidden reasoning. OpenAI warns that a response can hit its limit before any visible text appears, so you pay for input and reasoning without getting an answer. Where a model supports a reasoning effort setting, a lower setting uses fewer reasoning tokens and responds faster. OpenAI calls it a tuning knob rather than the main way to recover quality, so test each setting.

Use code and short hand-offs

Date arithmetic, totals, sorting and format checks are jobs for tested code, which handles them reliably without generating any text. If the model calls that code as a tool, the call and its result still use tokens, so return compact results.

In multi-step workflows, pass each step the previous result together with the constraints, source references and unresolved questions it needs. Omit earlier history only when those details have been preserved. Anthropic describes sub-agents that explore with tens of thousands of tokens and hand back a summary of often 1,000 to 2,000 tokens. That keeps the main context small. The sub-agents’ own tokens still count, though, so measure the whole workflow before you call it a saving.

Other ways to reduce LLM costs

Not every saving comes from trimming what the model sees. Some lower the price per token, others avoid calls altogether. Measure them separately from token reduction, so you know which change saved what.

Prompt caching

Prompt caching reuses the provider’s work on an identical opening of the prompt, called the prefix, such as a fixed system prompt or reference document. The model still sees every token. A cache hit can reduce input-processing time and cost. Savings depend on the model’s caching rules, prefix length, cache lifetime and actual reuse. On current OpenAI and Anthropic models, reading from the cache costs a tenth of the normal input price or less.

Writing to the cache costs a little more than a normal request. According to OpenAI’s caching guide, it is billed at 1.25 times the normal input rate on GPT-5.6 and later models. Caching therefore only pays off when the same prefix is reused. Put fixed content first and changing content last. Be aware that summarizing history can change the reusable prefix, which lowers the cache hit rate. OpenAI advises comparing total input cost before and after.

Result caching

You can also store a finished answer and serve it again. This suits answers that change slowly, such as the summary of a document nobody has edited. Base the cache key on everything that shapes the answer: the question, its settings, the document versions, the user’s access rights and the output format.

Batch processing and model choice

Some providers charge less when you can wait. Anthropic, for example, charges half price for batch requests. A cheaper model lowers the price per token, but it does not reduce the number of tokens, and it may need longer prompts or more retries. Our article on LLM cost optimization covers caching, batching and routing in more depth.

Skipping calls

Not every request needs a model. Rules can handle obvious cases, a database lookup can answer a status question and a template can format a confirmation. A skipped call saves its entire cost.

Common optimizations that backfire

Most failed token optimizations fail the same way: the call gets cheaper and the task gets more expensive.

Optimization	What goes wrong	Practical consequence
Compressing prompts into terse shorthand	Abbreviations and missing connectives make instructions ambiguous	The model guesses what you meant, and output varies between runs
Removing examples to save tokens	The model loses its clearest signal about format and tone	Format errors that need post-processing or a repair call
Summarizing history too aggressively	Exceptions, numbers and decisions disappear from the summary	Answers that contradict earlier agreements, with no warning
Retrieving too few passages	The passage that changes the answer is left out	Confident answers based on outdated or partial evidence
Setting output limits too low	Answers stop mid-sentence, or a reasoning model runs out before it answers	A follow-up call that resends the full context
Cutting tokens per call without tracking retries	Each call is cheaper, but the task needs more calls	Higher cost per successful task and slower results for users

The common thread is measuring the wrong unit. Tokens per call are easy to see on a dashboard, but cost per successful task is what you actually pay.

Frequently asked questions

The same principles apply to coding agents like Claude Code, which resends its instructions, your CLAUDE.md and the whole session with every message. DataNorth’s token optimization crash course is a three-hour, hands-on training for engineering teams that use Claude Code daily. Engineers learn context hygiene, a lean CLAUDE.md and how to pick the right model for each task, then practise on a real task from their own codebase. A usage baseline beforehand makes the effect measurable in your own numbers.