# LLM Apps in Production — Cost, Latency, Prompt Caching, Rate Limits and Monitoring

Source: https://skills101.in/learn/ai-engineering/llm-production-cost-caching — Skills101 free tutorial (AI Engineering — Building Apps with Large Language Models). Updated 2026-11-09.

A demo that costs ₹2 per answer is fine. The same feature used 50,000 times a day is a ₹1 lakh/day bill — and slow answers lose users. This lesson is about making AI features **cheap, fast and dependable**.

![A production LLM app: the client talks to your backend with auth, limits and logging; the backend uses versioned prompt templates, retrieval, tools, a cache and rate limits, and evals and monitoring, and calls the LLM API](https://skills101.in/learn-img/ai-engineering/architecture.svg)

## 1. Where the money goes

Most requests are dominated by **input tokens**: a long system prompt + retrieved documents + history, resent on every call.

```python
PRICE_IN, PRICE_OUT, PRICE_CACHE_READ = 4.00, 20.00, 0.40   # USD per million tokens (example — check current pricing)
USD_TO_INR = 85

def daily_cost(requests, system_tokens, doc_tokens, question_tokens, answer_tokens, cached_system=False):
    in_uncached = doc_tokens + question_tokens + (0 if cached_system else system_tokens)
    in_cached = system_tokens if cached_system else 0
    per_request = (in_uncached * PRICE_IN + in_cached * PRICE_CACHE_READ + answer_tokens * PRICE_OUT) / 1e6
    return requests * per_request * USD_TO_INR

base = dict(requests=20_000, system_tokens=3_000, doc_tokens=1_500, question_tokens=60, answer_tokens=250)
print(f"no caching:         ₹{daily_cost(**base):,.0f} per day")
print(f"system prompt cached: ₹{daily_cost(**base, cached_system=True):,.0f} per day")
print(f"…and 2 docs instead of 5: ₹{daily_cost(**{**base, 'doc_tokens': 600}, cached_system=True):,.0f} per day")
```

```text
no caching:         ₹39,508 per day
system prompt cached: ₹21,148 per day
…and 2 docs instead of 5: ₹15,028 per day
```

(Ignoring the small one-time cost of writing the cache.) The levers, roughly in order:

1. **Prompt caching** — a long, unchanging prefix (system prompt, tool definitions, a fixed document) is cached by the provider and re-read at a fraction of the price.
2. **Send less** — fewer, better retrieved chunks; trimmed history; no copy-pasted manuals.
3. **Shorter outputs** — ask for the length you need.
4. **Right model per task** — the most capable model for hard reasoning; test whether a smaller, cheaper one is good enough for simple, high-volume tasks like classification.
5. **Batch processing** — non-urgent jobs (grading 5,000 essays overnight) can use the provider's batch API at a discount.

## 2. Prompt caching in code

```python
import anthropic

LONG_RULES = "You are the help-desk assistant for Example Engineering College. " + "Rule text. " * 400

client = anthropic.Anthropic()
for question in ["What is the fee deadline?", "Where is the exam cell?"]:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=300,
        system=[{"type": "text", "text": LONG_RULES, "cache_control": {"type": "ephemeral"}}],   # cache this prefix
        messages=[{"role": "user", "content": question}],
    )
    u = response.usage
    print(f"{question:28s} cache_read={u.cache_read_input_tokens:5d}  uncached_input={u.input_tokens:5d}")
```

```text
What is the fee deadline?    cache_read= 1116  uncached_input=   50
Where is the exam cell?      cache_read= 1116  uncached_input=   50
```

*(Token numbers from our local stand-in API, which imitates caching; on the real API check `cache_read_input_tokens` the same way.)*

Caching is a **prefix match**: put stable content first (rules, tool definitions) and changing content (the question, timestamps) after it. A date or user name inside the system prompt breaks the cache on every request.

## 3. Speed

- **Stream** answers (lesson 5) — time to first word matters most.
- Keep prompts short; caching also cuts latency.
- Run independent calls **in parallel** (async), not one after another.
- Use simpler models or lower reasoning effort where quality allows; reserve deeper reasoning for hard tasks.

## 4. Protect yourself: limits, timeouts, fallbacks

```python
import time

class TokenBucket:
    """Allow `rate` requests per second per user, with bursts up to `capacity`."""
    def __init__(self, rate: float, capacity: int):
        self.rate, self.capacity = rate, capacity
        self.tokens, self.updated = float(capacity), 0.0

    def allow(self, now: float) -> bool:
        self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(rate=0.5, capacity=3)          # 1 request every 2 s, bursts of 3
times = [0, 0.1, 0.2, 0.3, 1.0, 2.5, 6.0]
print([("ok" if bucket.allow(t) else "wait") for t in times])
```

```text
['ok', 'ok', 'ok', 'wait', 'wait', 'ok', 'ok']
```

- **Per-user limits** (like above, or daily quotas) stop one user — or a bug in a loop — from draining your budget.
- **Timeouts and retries** with backoff for 429/5xx errors (the SDK does this).
- **Fallbacks**: if the model is unavailable, show a helpful message or a non-AI path (FAQ search).
- **Refusals**: check `stop_reason == "refusal"` and show a polite message instead of an empty box.
- A **spending cap** in the provider console as the last safety net.

## 5. What to log (and monitor)

Request id, prompt template version, model, input/output/cached tokens, cost, latency, stop reason, retrieved sources, tool calls, user feedback — **not** raw personal data you don't need. Dashboards on daily cost per feature and p95 latency catch problems before the bill does.

## Common mistakes

- Timestamps or user names at the start of the system prompt (cache never hits).
- No per-user limits on a public feature.
- One model for everything without measuring.
- Not logging token usage, so cost spikes are a surprise.

## Quick recap

- Input tokens dominate cost: cache stable prefixes, send less, cap outputs, batch offline work.
- Caching is a prefix match — stable first, changing last.
- Stream, parallelise and right-size models for speed.
- Rate-limit users, set timeouts, handle refusals and errors, cap spending, and log usage.


Practice quiz and free certificate: https://skills101.in/learn/ai-engineering/llm-production-cost-caching