Chatbots — Conversation History, Memory Limits and Streaming Responses

The API is stateless: every call starts fresh. A "conversation" is something your code keeps and resends.

1. A chatbot class that remembers

import anthropic

class Chat:
    def __init__(self, system: str):
        self.client = anthropic.Anthropic()
        self.system = system
        self.history: list[dict] = []          # alternating user / assistant turns

    def ask(self, text: str) -> str:
        self.history.append({"role": "user", "content": text})
        response = self.client.messages.create(
            model="claude-opus-5-5",
            max_tokens=500,
            system=self.system,
            messages=self.history,              # the WHOLE conversation, every time
        )
        answer = "".join(b.text for b in response.content if b.type == "text")
        self.history.append({"role": "assistant", "content": answer})
        print(f"[turn {len(self.history) // 2}] input tokens: {response.usage.input_tokens}")
        return answer

bot = Chat(system="You are a patient programming tutor for college students.")
bot.ask("My while loop never ends.")
bot.ask("I am using i = 0 and while i < 10.")
bot.ask("Oh, I forgot i += 1. Thanks!")

[turn 1] input tokens: 47
[turn 2] input tokens: 98
[turn 3] input tokens: 148

(Token counts from our local stand-in API, which estimates them roughly; real counts differ but grow the same way.)

Notice input tokens grow every turn: turn 10 resends turns 1–9. Long chats get slower and more expensive, and eventually hit the context limit.

2. Keeping history under control

def trim_history(history: list[dict], max_turns: int) -> list[dict]:
    """Keep only the last max_turns user+assistant pairs. Always start with a user turn."""
    keep = history[-max_turns * 2:]
    while keep and keep[0]["role"] != "user":
        keep = keep[1:]
    return keep

history = []
for i in range(1, 7):
    history += [{"role": "user", "content": f"question {i}"}, {"role": "assistant", "content": f"answer {i}"}]
print([m["content"] for m in trim_history(history, max_turns=2)])

['question 5', 'answer 5', 'question 6', 'answer 6']

Real assistants use a mix: a summary, key facts, and the last few turns.

3. Streaming: show the answer as it's written

A long answer can take several seconds. Streaming sends tokens as they're generated, so users start reading immediately:

import anthropic

client = anthropic.Anthropic()
with client.messages.stream(
    model="claude-opus-5-5",
    max_tokens=500,
    messages=[{"role": "user", "content": "Explain recursion to a first-year student."}],
) as stream:
    for piece in stream.text_stream:          # arrives in small chunks
        print(piece, end="", flush=True)
    final = stream.get_final_message()        # the complete message, with usage, at the end
print()
print("stop reason:", final.stop_reason)

Recursion is when a function solves a problem by calling itself on a smaller version of the same problem, until it reaches a simple case it can answer directly (the base case).
stop reason: end_turn

In a web app, your server streams these chunks to the browser (server-sent events or a WebSocket). Streaming doesn't make the total time shorter, but time to first word drops to under a second — and that's what users feel.

Case study: a WhatsApp doubt-solving bot

A coaching institute's bot answered doubts on WhatsApp. Problems in week one: replies got slower in long chats, and costs tripled. Fixes: keep the last 6 turns plus a running summary; store each student's course and language in the database instead of in the chat; cap answers at 150 words; and stream to the in-app chat (WhatsApp itself needs complete messages, so there they sent one reply when ready). Costs fell by more than half with no drop in ratings.

Common mistakes

• Expecting the model to remember previous API calls.
• Letting history grow forever.
• Cutting history so it starts with an assistant turn (the first message must be from the user).
• Not streaming long answers in chat interfaces.

Quick recap

• Your code keeps the conversation and resends it each call.
• Input tokens grow every turn; trim, summarise and store facts separately.
• client.messages.stream(...) + text_stream for word-by-word output; get_final_message() at the end.