Cost Optimization for LLM Applications

Your API bill is too high. Here are the proven strategies for cutting costs without losing quality.

The Cost Breakdown

Input tokens (cheap). Usually $0.50-2 per million tokens.

Output tokens (expensive). Usually $1.50-60 per million tokens. This is where costs blow up.

API calls themselves. Small fixed cost per request, dominates if you're making many small calls.

Quick Wins

1. Reduce output token usage. Set max_tokens limit. Ask for concise responses. Use structured output. Most cost comes from outputs.

2. Cache prompts intelligently. Anthropic and OpenAI both offer prompt caching. A 1000-token system prompt cached saves money per request.

3. Use smaller models for simple tasks. Claude Haiku or GPT-4o mini cost 10x less. Use them for classification, structured extraction, simple Q&A.

4. Batch requests where possible. One API call with 100 items cheaper than 100 calls with 1 item each.

5. Implement retrieval-based answers. Return answers from documents instead of generating. Zero generation cost if the answer exists.

6. Rate limit. Most cost overruns are from runaway loops or abuse. Rate limit per user, per feature.

Structural Changes

Use different models for different tasks. GPT-4 for reasoning, GPT-4o for vision, Haiku for classification. Right-size the model to the task.

Cache everything cacheable. Recent conversations, retrieved documents, computed embeddings.

Implement fallbacks. For time-sensitive questions, try rule-based answer first. Use LLM only if necessary.

Measuring Cost

Track cost per feature, per user, per request. Without measurement you can't optimize.

You should be able to reduce costs 3-5x without losing quality if you optimize systematically.