# How Large Language Models Work — Tokens, Next-Token Prediction, Context Windows and Limits

Source: https://skills101.in/learn/ai-engineering/how-llms-work — Skills101 free tutorial (AI Engineering — Building Apps with Large Language Models). Updated 2026-11-09.

ChatGPT, Claude and Gemini feel like magic. Underneath, a **large language model (LLM)** does one simple thing extremely well: **given some text, predict what comes next**. Everything else — answering questions, writing code, summarising — is built from that.

## 1. Next-token prediction

![The model reads "The capital of India is" and gives probabilities for the next token: New 0.91, Delhi 0.04, a 0.02, the 0.01; it picks one, appends it, and repeats](https://skills101.in/learn-img/ai-engineering/next-token.svg)

Text is split into **tokens** — word pieces (roughly ¾ of an English word on average; Hindi and code often take more tokens per word). The model outputs a probability for every possible next token, one is chosen, appended, and the process repeats until the answer ends.

You can build a (very!) tiny version of the idea yourself — count which word follows which:

```python
from collections import Counter, defaultdict

corpus = """the exam is on monday . the exam is easy . the result is out .
the exam hall is open . the result is good ."""
words = corpus.split()

following = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    following[current][nxt] += 1

def predict(word):
    counts = following[word]
    total = sum(counts.values())
    return [(w, round(c / total, 2)) for w, c in counts.most_common(3)]

print("after 'the':", predict("the"))
print("after 'is':", predict("is"))

text = ["the"]
for _ in range(4):
    text.append(following[text[-1]].most_common(1)[0][0])   # always take the most likely word
print(" ".join(text))
```

```text
after 'the': [('exam', 0.6), ('result', 0.4)]
after 'is': [('on', 0.2), ('easy', 0.2), ('out', 0.2)]
the exam is on monday
```

A real LLM is this idea scaled up enormously: instead of counting word pairs, a **transformer** neural network with billions of parameters looks at the **whole** text so far, trained on a huge amount of text and then refined with human feedback to be helpful and safe.

## 2. Training vs using

| | Training (done by the AI lab) | Using (what you do) |
|---|---|---|
| Cost | crores of rupees, thousands of GPUs, months | rupees per thousand requests |
| What changes | the model's parameters | nothing — the model is fixed |
| You control | — | the **prompt**, the data you send, the tools you allow |

As an AI engineer you almost never train a model. You **use** one well: good prompts, the right data in the context, tools, evaluation and safety. That's this track.

## 3. The context window

![Everything the model sees — system prompt, conversation so far, retrieved documents and the new question — must fit in one context window; the output is generated after it](https://skills101.in/learn-img/ai-engineering/context-window.svg)

- The model has **no memory between calls**. A chat app resends the conversation every time.
- Everything — instructions, history, documents — must fit in the **context window** (current large models accept hundreds of thousands to a million tokens).
- You pay per **input token** and per **output token**. Long prompts cost more and run slower.

## 4. Why LLMs make things up (hallucinations)

The model produces the most *plausible* continuation, not a looked-up fact. If the answer isn't in its training data or your prompt, it may produce a confident, fluent, wrong answer — a fake citation, a wrong exam date, a function that doesn't exist.

Defences you'll learn: give the model the facts (**RAG**, lesson 7), let it call **tools** for live data (lesson 8), ask for **sources**, allow "I don't know", and **evaluate** (lesson 9).

## 5. Good at / careful with

| Good at | Be careful with |
|---|---|
| writing, rewriting, summarising, translating (incl. Hindi & English) | exact arithmetic on long numbers (use code/tools) |
| extracting structure from messy text | recent events after its training data |
| explaining concepts, tutoring | facts about small or local things (your college's rules) |
| writing and reviewing code | anything where a wrong answer is costly without a human check |

## 6. Case study: the college help-desk bot

A college wants a bot that answers students' questions about fees, attendance and exams.

- **Plain chatbot:** answers fluently — but invents the attendance rule and the fee deadline. Dangerous.
- **AI-engineered version:** searches the official handbook first (RAG), answers **only** from it with the section number, says "please contact the office" when the handbook doesn't cover the question, and is tested on 50 real student questions before launch.

Same model; completely different reliability. The difference is engineering.

## Common mistakes

- Treating the model as a database of facts.
- Assuming it remembers earlier chats.
- Ignoring cost: a 50-page document in every request adds up quickly.

## Quick recap

- An LLM predicts the next token, repeatedly; tokens are word pieces.
- The context window holds everything the model sees; there's no memory between calls.
- Hallucinations come from plausibility, not lookup — ground answers in real data.
- AI engineers build prompts, retrieval, tools, evaluation and safety around a fixed model.


Practice quiz and free certificate: https://skills101.in/learn/ai-engineering/how-llms-work