How Large Language Models Work — Tokens, Next-Token Prediction, Context Windows and Limits

ChatGPT, Claude and Gemini feel like magic. Underneath, a large language model (LLM) does one simple thing extremely well: given some text, predict what comes next. Everything else — answering questions, writing code, summarising — is built from that.

1. Next-token prediction

Text is split into tokens — word pieces (roughly ¾ of an English word on average; Hindi and code often take more tokens per word). The model outputs a probability for every possible next token, one is chosen, appended, and the process repeats until the answer ends.

You can build a (very!) tiny version of the idea yourself — count which word follows which:

from collections import Counter, defaultdict

corpus = """the exam is on monday . the exam is easy . the result is out .
the exam hall is open . the result is good ."""
words = corpus.split()

following = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    following[current][nxt] += 1

def predict(word):
    counts = following[word]
    total = sum(counts.values())
    return [(w, round(c / total, 2)) for w, c in counts.most_common(3)]

print("after 'the':", predict("the"))
print("after 'is':", predict("is"))

text = ["the"]
for _ in range(4):
    text.append(following[text[-1]].most_common(1)[0][0])   # always take the most likely word
print(" ".join(text))

after 'the': [('exam', 0.6), ('result', 0.4)]
after 'is': [('on', 0.2), ('easy', 0.2), ('out', 0.2)]
the exam is on monday

A real LLM is this idea scaled up enormously: instead of counting word pairs, a transformer neural network with billions of parameters looks at the whole text so far, trained on a huge amount of text and then refined with human feedback to be helpful and safe.

2. Training vs using

As an AI engineer you almost never train a model. You use one well: good prompts, the right data in the context, tools, evaluation and safety. That's this track.

3. The context window

• The model has no memory between calls. A chat app resends the conversation every time.
• Everything — instructions, history, documents — must fit in the context window (current large models accept hundreds of thousands to a million tokens).
• You pay per input token and per output token. Long prompts cost more and run slower.

4. Why LLMs make things up (hallucinations)

The model produces the most plausible continuation, not a looked-up fact. If the answer isn't in its training data or your prompt, it may produce a confident, fluent, wrong answer — a fake citation, a wrong exam date, a function that doesn't exist.

Defences you'll learn: give the model the facts (RAG, lesson 7), let it call tools for live data (lesson 8), ask for sources, allow "I don't know", and evaluate (lesson 9).

5. Good at / careful with

6. Case study: the college help-desk bot

A college wants a bot that answers students' questions about fees, attendance and exams.

• Plain chatbot: answers fluently — but invents the attendance rule and the fee deadline. Dangerous.
• AI-engineered version: searches the official handbook first (RAG), answers only from it with the section number, says "please contact the office" when the handbook doesn't cover the question, and is tested on 50 real student questions before launch.

Same model; completely different reliability. The difference is engineering.

Common mistakes

• Treating the model as a database of facts.
• Assuming it remembers earlier chats.
• Ignoring cost: a 50-page document in every request adds up quickly.

Quick recap

• An LLM predicts the next token, repeatedly; tokens are word pieces.
• The context window holds everything the model sees; there's no memory between calls.
• Hallucinations come from plausibility, not lookup — ground answers in real data.
• AI engineers build prompts, retrieval, tools, evaluation and safety around a fixed model.