How an LLM works, step by step: from tokens to a generated answer.

Before we start: what this walkthrough does and does not describe

A large language model is a trained numerical function that maps a sequence of inputs to scores for possible continuations. In this guide, we follow a text-generating decoder-only transformer: the family of architecture behind many well-known language models. ‘Decoder-only’ means we use a causal generation stack rather than the original translation Transformer's separate encoder and decoder.

You need only basic arithmetic to follow the narrative. The formulas and matrix dimensions are there for readers who want the machinery. Read sections 1–10 for one forward pass, 11–14 for training and adaptation, and 15–20 for serving, tools and practical consequences. Skip a formula and return to it after the example if necessary.

Real systems vary: tokenisers, normalisation, activation functions, positional schemes, attention patterns and training recipes are not universal. A product's public name does not reveal all its internals. Multimodal encoders and image generators introduce further machinery that this text-only walkthrough does not attempt to specify.

1. Follow the complete route from a prompt to one new token

Take ‘The capital of France is’ as a running example. The model does not open a geography database simply because this looks like a factual question. The application assembles the input; a tokeniser converts it into IDs; the neural network transforms those IDs into contextual vectors; an output layer scores the vocabulary; a decoding rule selects the next token. The selected token becomes part of the next input.

The diagram shows inference: using already-trained weights. The arrows are stages of computation, not a claim that each stage is a separate online service. One displayed word may require more than one token, and the model need not select the correct answer.

messages → formatted input → token IDs
        → embeddings + position information
        → repeated transformer blocks
        → vocabulary logits → decoding rule
        → one new token → repeat

2. Tokenisation turns text into IDs, not meanings

A token is an entry in a vocabulary. Depending on the tokeniser, it can represent a word, a fragment, punctuation, whitespace or a sequence of bytes. Text is encoded as integer IDs. Those IDs are lookup keys: token 42 is not twice the meaning of token 21.

Byte-pair encoding starts with small units and learns merge rules from a training corpus, repeatedly combining frequent adjacent units. At use time, the already-learned rules determine how a new input is split. Byte-level variants can represent unfamiliar characters through bytes. Other tokenisers use different algorithms; do not assume a particular app uses the exact BPE variant demonstrated here.

Our toy vocabulary might split ‘unhelpful’ into [un, help, ful]. That is a made-up split, not an observed encoding from ChatGPT. To obtain real IDs, run the matching tokeniser and record its version. Capitalisation, leading spaces and language can change the encoding. This helps explain why character counting and spelling manipulations can be awkward, but tokenisation alone does not explain every failure.

• Inspect a real BPE implementation ↗

3. Embeddings give every token a trainable vector

Suppose the vocabulary has V entries and the model width is d. The embedding table E has shape V × d. Looking up n token IDs produces a matrix X with shape n × d, ignoring the batch dimension for now. Each row is the numerical starting representation of one token position.

The coordinates are learned parameters, not human-labelled slots such as ‘French’, ‘positive’ and ‘place’. Features can be distributed across many coordinates. The initial vector for a token is normally a lookup; its representation later in the network depends on the surrounding prefix. Two occurrences of ‘bank’ can therefore acquire different contextual representations.

Do not confuse these internal token embeddings with a separate embedding model used to index documents for search. Both use vectors, but their training, output shape and intended similarity behaviour can differ.

Vocabulary size: V       Model width: d
Embedding table E:       [V, d]
Input IDs:               [n]
X = E[input IDs]:        [n, d]
With batch size B:       [B, n, d]

4. Position information tells the network about order

‘Dog bites person’ and ‘Person bites dog’ contain the same word types but mean different things. An attention operation needs information about positions to distinguish ordering reliably. The original Transformer adds sinusoidal position vectors to token embeddings; learned position embeddings are another approach. Modern architectures can use different position-dependent transformations within attention.

Position information is not a clock or a memory of a previous conversation. It concerns where tokens sit in this model input. A model's supported context length also depends on training and implementation, not just whether you can allocate a longer array. Extending position indices is not proof of equally reliable comprehension.

• Original architecture: embeddings and positional encoding ↗

5. Queries, keys and values prepare the attention calculation

Within one attention head, the incoming representations are projected through three learned matrices. Q = XWQ gives queries; K = XWK gives keys; V = XWV gives values. Queries and keys determine mixing weights; values supply the vectors being mixed. ‘Query’ is a mathematical role here, not necessarily your typed question.

For one head, Q and K have n rows and dk columns. V has n rows and dv columns. Multiplying Q by the transpose of K creates n × n scores: a query at each position is compared with keys at other positions. Divide scores by √dk before applying softmax. This scaling helps keep dot products from becoming excessively large as the head dimension grows.

The attention formula is a learned compatibility-and-mixing operation. It does not perform a Boolean truth check, and a large weight is not proof that a source is correct. The Q/K/V projections are recalculated from each layer's incoming representations, not permanently attached labels on the original words.

Q = XWQ          K = XWK          V = XWV
S = QKᵀ / √dk + causal_mask
A = softmax(S), applied separately to each row
head_output = AV

X: [n, d]      WQ, WK: [d, dk]      WV: [d, dv]
Q, K: [n, dk]  A: [n, n]           output: [n, dv]
Here V means the value matrix, not vocabulary size.

6. The causal mask prevents looking at future tokens

During next-token training, the full example is available to the training program. Without a restriction, a position could simply inspect the later token it is supposed to predict. Causal attention allows a position to use itself and earlier positions, but not later ones.

With rows as query positions and columns as key positions, the allowed region is the diagonal and lower-left triangle. We add zero to allowed scores and negative infinity to forbidden scores before softmax. A forbidden position then receives weight zero. Multiplying a score by zero is not equivalent: softmax of zero can still give a positive weight.

This mask lets a training program compute many next-token predictions together while preventing future-token leakage within the attention operation. During ordinary generation, future answer tokens do not exist yet. Padding may require a separate mask as well.

Allowed key positions for each query:
          key 1  key 2  key 3  key 4
query 1     ✓      ×      ×      ×
query 2     ✓      ✓      ×      ×
query 3     ✓      ✓      ✓      ×
query 4     ✓      ✓      ✓      ✓

7. Work through one attention head with actual numbers

This is a deliberately tiny, invented example for the third position, which can attend to three available positions. Its query is q = [1, 0]. Let the keys be k1 = [1, 0], k2 = [0, 1] and k3 = [1, 1]. The dot products are [1, 0, 1]. With dk = 2, divide by √2 to obtain approximately [0.7071, 0, 0.7071].

Softmax takes the exponential of each score and divides by their sum. The resulting weights are approximately [0.4011, 0.1978, 0.4011], summing to one. Choose values v1 = [2, 0], v2 = [0, 2] and v3 = [2, 2]. Their weighted sum is approximately [1.6044, 1.1978]. That vector is this head's output for this query—not a generated word and not a confidence score.

Changing the query or keys changes the weights. Changing only a value changes the mixed output without changing those weights. A later layer can transform the result again. In a real model, these vectors have many more dimensions and come from learned projections rather than our hand-picked integers.

softmax(s)i = exp(si) / Σj exp(sj)

output = 0.4011 × [2, 0]
       + 0.1978 × [0, 2]
       + 0.4011 × [2, 2]
       ≈ [1.6044, 1.1978]

8. Multiple heads, residuals and feed-forward layers build a block

Multi-head attention runs several sets of projections, then combines the head outputs through an output projection. Different heads can represent different patterns. Do not assign every head a tidy permanent job such as ‘grammar head’: the computation is learned and interpretation is more complicated than the diagram suggests.

A transformer block also has a position-wise feed-forward network: a learned nonlinear transformation applied to each position. A simple version expands the vector, applies a nonlinearity, then projects back to the model width. Nonlinearity matters: stacking linear transformations alone would collapse into another linear transformation. Implementations may use gated activations or other variants.

Residual connections add a sublayer's output to the continuing representation. Normalisation controls activation scale and helps training. The original paper normalises after the addition; many later architectures use pre-normalisation. Below is a simplified pre-normalised block, not the original paper's exact layout or any current commercial model's blueprint. Stacking blocks repeatedly mixes contextual information and transforms features.

# Simplified pre-normalised block; not runnable code
h = x + attention(norm(x))
y = h + feed_forward(norm(h))

# Repeat with separately learned parameters per block.
# Attention mixes positions; feed-forward transforms each position.

9. The output head turns a vector into vocabulary scores

After the final block and any final normalisation, an output projection maps the last position's d-dimensional representation to V logits—one score per vocabulary entry. A logit is an unnormalised number. It can be negative and is not yet a probability. Some architectures share output and input embedding weights; others do not.

Softmax converts the logits into a probability distribution over the next token. These probabilities describe the model's continuation distribution under this input and decoding setup. They are not calibrated probabilities that a factual statement is true. A highly predictable citation can still be fabricated.

The model's mathematical factorisation is p(x1,…,xT) = ∏t p(xt | x1,…,x(t−1)). Next-token prediction is the local interface; the learned representations supporting it can encode complex patterns and support multi-step problem solving. Calling the system ‘just autocomplete’ does not explain those capabilities, while calling its output proof of understanding or consciousness goes beyond what this calculation establishes.

last_hidden: [d]
output_projection: [d, V]
logits: [V]
next_token_probabilities = softmax(logits)

10. Sampling chooses a token; it does not verify the answer

Greedy decoding chooses the highest-scoring token at each step. Sampling draws from a distribution. Temperature T > 0 rescales logits before softmax: lower temperatures concentrate the distribution, while higher temperatures flatten it. The T→0 limit favours a maximum; practical implementations may expose a separate greedy setting.

For invented logits [2, 1, 0], softmax gives approximately [0.6652, 0.2447, 0.0900]. At T = 0.5, it becomes [0.8668, 0.1173, 0.0159]. At T = 2, it becomes [0.5065, 0.3072, 0.1863]. Top-k restricts candidates to a fixed number; top-p retains a cumulative-probability nucleus, then samples from the renormalised set. Implementations and supported controls differ.

Greedy selection is not a guarantee of the most probable complete answer, because each choice changes later probabilities. Lower randomness also cannot turn false learned associations into reliable evidence. Even nominally deterministic serving can vary with numerical precision, hardware, batching, ties or application changes. Do not promise identical replies from a consumer app.

temperature_distribution = softmax(logits / T)

# Stable softmax subtracts the largest input first:
p_i = exp(z_i - max(z)) / Σj exp(z_j - max(z))

11. Pretraining teaches prediction across many examples

Pretraining repeatedly presents sequences from a prepared dataset. For a causal language model, a position predicts the following token using only the permitted prefix. The input [A, B, C] can supply targets [B, C, D]. This is called teacher forcing: training uses actual dataset tokens as the prefix, rather than feeding back the model's own sampled mistakes at every position.

A causal mask allows many positions to be processed in parallel within a training batch. That is different from generating an unknown answer, where the next input token depends on the token just selected. Training examples may be packed, padded or separated with special tokens; the data pipeline must handle boundaries and loss masks correctly.

Dataset composition matters: coverage, duplication, quality, conflicting claims, languages, code, unwanted content and contamination of evaluation data affect what can be learned and what a test really measures. Parameters encode learned associations and transformations; they are not a tidy document store. Models can nevertheless memorise some training material, so ‘not a database’ must not become a claim that memorisation or privacy risk is impossible.

Training input:   A   B   C
Prediction target: B   C   D

Predict B from A; C from A,B; D from A,B,C.
The causal mask prevents using the target as future input.

12. Loss and backpropagation change the weights

Cross-entropy for an observed next token is −ln(p(target)). If the model assigns the target probability 0.1, the loss is about 2.3026; at 0.5 it is about 0.6931. Lower loss means the observed token was assigned more probability. It does not mean the generated response was verified against the real world. The training target can itself contain an error.

Backpropagation applies the chain rule through the output head, blocks and embeddings to calculate gradients. For softmax cross-entropy at one position, the derivative with respect to logit j is p(j) minus one if j is the target, otherwise p(j). An optimiser uses gradients and its own state to update trainable parameters. The simple gradient-descent equation below is a teaching approximation; adaptive optimisers do more.

Gradients are usually aggregated over many token positions and examples. Learning rates, regularisation, numerical precision and distributed training affect the update. Validation on held-out material helps detect poor generalisation, but data leakage and task mismatch can still make evaluation misleading. Training loss and user usefulness are not the same objective.

loss_at_t = -ln p(target_t | prefix_t)
mean_loss = sum(loss_at_t) / number_of_scored_tokens

For one position: gradient(logit_j) = p_j - 1[j is target]
Simple gradient descent: θ_new = θ_old - learning_rate × gradient

Illustrative losses: -ln(0.1) ≈ 2.3026; -ln(0.5) ≈ 0.6931

13. Instruction tuning changes what kind of continuation is preferred

A model trained to continue arbitrary text is not automatically an assistant that follows your request. Supervised instruction tuning trains on examples of prompts and desired responses, encouraging a helpful conversational pattern. A chat template supplies the roles and turn boundaries in the format expected by that model.

The InstructGPT research describes supervised demonstrations followed by comparisons of model outputs and reinforcement learning from human feedback. In that recipe, a learned preference signal guides further training. Other post-training recipes exist; do not assume every current assistant uses an identical sequence or objective.

Human or model preferences are not a perfect truth oracle. A fluent, agreeable response can be preferred without every claim being checked. Post-training can improve instruction following and some safety behaviours, yet important errors remain possible. Role markers and training help distinguish instructions from source text, but do not by themselves create a secure boundary against prompt injection.

• Original instruction-following and human-feedback study ↗

14. In-context learning, fine-tuning and LoRA are different operations

If you supply three examples in a prompt, you alter the computation's input. The model's activations respond to those examples, but an ordinary inference pass does not update weights. This is in-context adaptation. The GPT-3 few-shot study explicitly applied tasks without gradient updates. A useful pattern can disappear when the relevant examples are absent from a later request.

Fine-tuning runs additional training on a chosen dataset and changes trainable parameters. LoRA freezes the base weights and learns a low-rank update through smaller matrices. For a weight W with shape m × n, write an update as BA where B has shape m × r and A has shape r × n. This uses r(m+n) trainable entries rather than mn when r is small, ignoring other trainable components and scaling.

For an illustrative 1000 × 1000 matrix and rank 8, that is 16,000 update entries rather than 1,000,000 full-matrix entries. This is an arithmetic example, not a speed or quality guarantee. Fine-tuning may teach a format or behaviour; it is not automatically the best way to supply changing business facts, and it does not provide document provenance on its own.

Normal prompt:       same weights, different input
Full fine-tuning:    training updates base parameters
LoRA:                W_effective = W_frozen + scale × BA

Toy count: 8 × (1000 + 1000) = 16,000 trainable update entries

• Few-shot adaptation without gradient updates ↗
• Original LoRA method and limitations of its experiments ↗

15. Prefill and the KV cache explain how generation is served

During prefill, the model processes the available prompt positions and builds per-layer keys and values. The final position supplies logits for the first answer token. During decode, each selected token is passed through the model to predict the following one. A KV cache reuses earlier keys and values instead of recomputing them for every step.

Why can this work? In a causal transformer, a previous position's representation does not depend on future tokens. Its stored keys and values can therefore be reused while the new query attends to the permitted cached positions. New keys and values are appended at each layer. Position indices, masks and model state must be handled consistently.

A KV cache is temporary computational state, not evidence that the model has permanently learned the conversation. Reuse across requests is a serving feature with its own correctness and isolation requirements. Some cache designs evict older entries or use restricted attention; the simplest full-context cache is not the only implementation.

# Conceptual inference loop, not a library API
logits, cache = prefill(prompt_ids)
while not finished:
    token = choose_next_token(logits)
    emit(token)
    logits, cache = decode_one(token, cache)

# Weights stay fixed throughout this ordinary inference loop.

• Official implementation explanation of key/value caching ↗

16. Memory, long contexts and FlashAttention have different costs

For ordinary dense attention during prefill, n positions compare with n positions: attention arithmetic scales quadratically with sequence length, holding dimensions fixed. Doubling n roughly quadruples that part of the work—not necessarily the entire service cost or latency. During cached, single-token decode, that token's dense attention work grows roughly linearly with the available cached length, again holding other dimensions fixed.

A simple full-context KV-cache estimate is 2 × L × B × n × Hkv × dh × bytes per stored value. The 2 covers keys and values. For 32 layers, batch 1, 8192 tokens, eight KV heads, head dimension 128 and two bytes per value, it gives 1,073,741,824 bytes: 1 GiB. This excludes weights, activations, buffers, allocator overhead and any different cache representation.

Weights have a separate cost. Seven billion parameters at two bytes each require about 14 GB in decimal units for the weights alone. Four-bit storage would be roughly 3.5 GB before scales, metadata and other costs; it does not establish total runtime memory or equivalent accuracy. Quantisation trades representation precision for storage/computation properties, with results depending on the method and workload.

FlashAttention reorganises exact attention computation to reduce transfers between GPU memory levels rather than magically removing all quadratic arithmetic. Dense, sliding-window, grouped-query, sparse and other designs have different trade-offs. A bigger advertised context window is capacity, not proof that every passage will be used correctly. The 2023 Lost in the Middle study found position-sensitive failures on its tested tasks and models; that is a reason to test your workflow, not a universal accuracy rate for today's products.

Illustrative full KV cache, bytes:
2 × 32 × 1 × 8192 × 8 × 128 × 2
= 1,073,741,824 bytes = 1 GiB

L = layers; B = batch; n = cached tokens
Hkv = key/value heads; dh = head dimension

• Exact IO-aware attention: the FlashAttention paper ↗
• Historical evidence on information position in long inputs ↗

17. Retrieval, conversation history and persistent memory sit around the model

An application can fetch documents before asking the model to answer. A common retrieval-augmented generation workflow splits documents into chunks, indexes representations, retrieves candidates for the question and adds selected passages to the model input. The original RAG research used a particular retriever/generator setup; not every modern retrieval pipeline is that exact architecture.

Retrieval changes the available evidence, not necessarily the base model's weights. A search embedding is used to find candidates; the answer generator then processes the supplied text. Bad chunk boundaries, stale documents, poor retrieval, missing passages and unsupported synthesis can all cause errors. A real URL next to a sentence does not prove the sentence is supported.

Conversation history can be replayed or summarised into later inputs. A persistent-memory feature can store selected information outside the model and inject it when needed. These are product behaviours, not the same thing as the KV cache or fine-tuning. Their availability, retention and privacy implications depend on the service. Do not assume a fresh chat has every old detail, or that text inside an uploaded document should be obeyed as an instruction.

• Apply this to a real document-analysis task

• Original retrieval-augmented generation research ↗

18. Reasoning and tool use add a workflow, not a truth switch

A system can generate intermediate working tokens, use a calculator, search, run code or revise a draft before returning an answer. More computation may help on a demanding task, but missing evidence remains missing evidence. A reasoning model's private internal process is not established by a polished explanation it produces for the user.

In a tool-using loop, the model proposes a structured call; the application validates and executes it; the resulting observation returns as input; the model continues. The weights do not directly browse the web or send an email. Tool permissions, validation, sandboxing, approval and error handling belong to the surrounding system. A well-formed tool request can still be the wrong action.

Ask for checkable outputs: source locations, reproduced calculations, tested code and explicit assumptions. Do not treat ‘show your reasoning’ as proof that a result is correct. An assistant's answer must still be compared with the original evidence and the actual effects of any action.

• Brief a difficult comparison with evidence you can inspect
• Scope a tool-using code change and its checks

19. Why fluent answers can still be wrong

The model is producing a continuation, not conducting an automatic audit of every claim. It may have learned inconsistent information, lack the relevant fact, misread supplied material, follow a misleading pattern or generate a plausible but nonexistent source. Sampling can add variability, but removing sampling does not remove these causes.

Errors can also enter through the application: the wrong PDF passage was retrieved, an image was unreadable, an outdated summary was replayed, or a failed tool call was treated as success. Diagnose the layer before changing the prompt. The neural model, context selection and external tools should not be treated as one undifferentiated black box.

Neither a high next-token probability nor a convincing attention picture establishes factual confidence. Nor does a self-reported percentage necessarily reflect calibration. To estimate reliability for a task, evaluate representative held-out examples, record failures and repeat under the conditions you intend to use. A single impressive answer cannot establish a universal model ranking.

20. Use the machinery to write and test better prompts

The practical consequence is not ‘write a gigantic technical prompt’. Supply the facts that change the result, label their roles, request an inspectable output and define how you will check it. Examples can clarify a subtle pattern without retraining. Retrieval can supply current evidence without pretending it has become model knowledge. Code can check arithmetic instead of relying on a fluent number.

Try three bounded exercises. First, compare the toy attention calculation with an independently run calculator. Second, inspect the same sentence with two explicitly identified tokenisers: record their actual splits rather than inventing IDs. Third, give an assistant a short source document with one missing answer and verify that it says ‘not found’ instead of filling the gap.

For a document-position experiment, use fictional numbered facts and move one target fact between the beginning, middle and end. Keep the question fixed, repeat runs and record model, tools, date and source locations. It is a small workflow test, not a reproduction of a research benchmark. Keep factual correctness separate from style and do not publish a measured reliability rate from one run.

• Keep a fair record of your prompt experiments
• Return to the practical prompting learning path

Keep these terms separate when reading AI claims

A product description may use ‘memory’, ‘learning’ or ‘thinking’ loosely. Ask which concrete mechanism is meant before inferring a capability or privacy property.