# Chatbots — Conversation History, Memory Limits and Streaming Responses

Source: https://skills101.in/learn/ai-engineering/llm-chatbot-history-streaming — Skills101 free tutorial (AI Engineering — Building Apps with Large Language Models). Updated 2026-11-09.

The API is stateless: every call starts fresh. A "conversation" is something **your code** keeps and resends.

![Bars showing input tokens growing with every turn because each call resends the whole conversation](https://skills101.in/learn-img/ai-engineering/history-growth.svg)

## 1. A chatbot class that remembers

```python
import anthropic

class Chat:
    def __init__(self, system: str):
        self.client = anthropic.Anthropic()
        self.system = system
        self.history: list[dict] = []          # alternating user / assistant turns

    def ask(self, text: str) -> str:
        self.history.append({"role": "user", "content": text})
        response = self.client.messages.create(
            model="claude-opus-5-5",
            max_tokens=500,
            system=self.system,
            messages=self.history,              # the WHOLE conversation, every time
        )
        answer = "".join(b.text for b in response.content if b.type == "text")
        self.history.append({"role": "assistant", "content": answer})
        print(f"[turn {len(self.history) // 2}] input tokens: {response.usage.input_tokens}")
        return answer

bot = Chat(system="You are a patient programming tutor for college students.")
bot.ask("My while loop never ends.")
bot.ask("I am using i = 0 and while i < 10.")
bot.ask("Oh, I forgot i += 1. Thanks!")
```

```text
[turn 1] input tokens: 47
[turn 2] input tokens: 98
[turn 3] input tokens: 148
```

*(Token counts from our local stand-in API, which estimates them roughly; real counts differ but grow the same way.)*

Notice **input tokens grow every turn**: turn 10 resends turns 1–9. Long chats get slower and more expensive, and eventually hit the context limit.

## 2. Keeping history under control

```python
def trim_history(history: list[dict], max_turns: int) -> list[dict]:
    """Keep only the last max_turns user+assistant pairs. Always start with a user turn."""
    keep = history[-max_turns * 2:]
    while keep and keep[0]["role"] != "user":
        keep = keep[1:]
    return keep

history = []
for i in range(1, 7):
    history += [{"role": "user", "content": f"question {i}"}, {"role": "assistant", "content": f"answer {i}"}]
print([m["content"] for m in trim_history(history, max_turns=2)])
```

```text
['question 5', 'answer 5', 'question 6', 'answer 6']
```

| Strategy | When |
|---|---|
| **Keep the last N turns** | casual chat; old turns rarely matter |
| **Summarise old turns** | long tutoring sessions: ask the model to summarise turns 1–20 into a short note and keep that note + recent turns |
| **Store facts separately** | user profile (name, course, language preference) kept in your database and added to the system prompt |
| **Retrieve old messages** | very long histories: search them like documents (lesson 7) |

Real assistants use a mix: a summary, key facts, and the last few turns.

## 3. Streaming: show the answer as it's written

A long answer can take several seconds. **Streaming** sends tokens as they're generated, so users start reading immediately:

```python
import anthropic

client = anthropic.Anthropic()
with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=500,
    messages=[{"role": "user", "content": "Explain recursion to a first-year student."}],
) as stream:
    for piece in stream.text_stream:          # arrives in small chunks
        print(piece, end="", flush=True)
    final = stream.get_final_message()        # the complete message, with usage, at the end
print()
print("stop reason:", final.stop_reason)
```

```text
Recursion is when a function solves a problem by calling itself on a smaller version of the same problem, until it reaches a simple case it can answer directly (the base case).
stop reason: end_turn
```

In a web app, your server streams these chunks to the browser (server-sent events or a WebSocket). Streaming doesn't make the total time shorter, but **time to first word** drops to under a second — and that's what users feel.

## Case study: a WhatsApp doubt-solving bot

A coaching institute's bot answered doubts on WhatsApp. Problems in week one: replies got slower in long chats, and costs tripled. Fixes: keep the last 6 turns plus a running summary; store each student's course and language in the database instead of in the chat; cap answers at 150 words; and stream to the in-app chat (WhatsApp itself needs complete messages, so there they sent one reply when ready). Costs fell by more than half with no drop in ratings.

## Common mistakes

- Expecting the model to remember previous API calls.
- Letting history grow forever.
- Cutting history so it starts with an assistant turn (the first message must be from the user).
- Not streaming long answers in chat interfaces.

## Quick recap

- Your code keeps the conversation and resends it each call.
- Input tokens grow every turn; trim, summarise and store facts separately.
- `client.messages.stream(...)` + `text_stream` for word-by-word output; `get_final_message()` at the end.


Practice quiz and free certificate: https://skills101.in/learn/ai-engineering/llm-chatbot-history-streaming