How LLMs generate text
A model scores every possible next token, and the sampling settings decide how it picks one. Temperature, top-p, top-k and max tokens, and what each can and cannot fix.
By the end
Explain how temperature, top-p and max tokens change output, why none of them adds knowledge, and why a length cap truncates instead of tightening.
Lab 1
Try it
Temperature and top-p, on one next token
The model has scored every possible next token for “The sky is ___”. Drag the settings and watch the pool of choices change. Faint bars are the raw probabilities; solid bars are what the sampler can actually pick.
- ·blue62.0%
- ·clear14.0%
- ·falling7.0%
- ·dark6.0%
- ·on4.0%
- ·purple3.0%
- ·made2.0%
- ·the2.0%
8 of 8 tokens can be picked. Raising temperature moves weight down the list; lowering top-p removes the bottom of it.
Illustrative distribution, not measured from a model.
An LLM does one thing repeatedly: it predicts a probability for every possible next token, then picks one. The sampling settings decide how it picks. None of them add knowledge. They only change how adventurous the pick is.
Tokens and the context window
Models read and count tokens, roughly four characters of English each. That is why "under 10 words" is a soft target the model often misses: it is counting something else.
The context window is one shared budget. The system prompt, the conversation history, any documents and the answer all draw from it. Facts buried in the middle of a long context get used less reliably, so put the thing that matters near the start or the end.
Sampling settings
| Setting | What it does | Raise it for | Cost of raising it |
|---|---|---|---|
| Temperature | Flattens or sharpens the token probabilities | Brainstorming, variety, creative copy | More drift, broken constraints, odd or abstract output |
| Top-p (nucleus) | Samples only from the top tokens whose probabilities sum to p | Same as temperature | Tune one, not both |
| Top-k | Samples only from the k most likely tokens | Rarely tuned by hand | Same as temperature |
| Max tokens | Hard cap on output length | Long answers | Lowering it truncates mid-sentence; it does not make answers concise |
| Frequency / presence penalty | Discourages reusing tokens already used | Repetitive output | Awkward word choices |
Learn
- Temperature at 0 always picks the most likely token. Raising it spreads probability onto less likely tokens.
- Top-p cuts off the long tail before sampling. At a low p, the model only chooses among its strongest candidates.
- Max tokens is a truncation rule. Lowering it cuts the reply off; it does not make the model write less.
Drill
- Ask for five eco-friendly slogans of exactly 7 words at temperature 0, 0.5 and 1.0.
- Count how many break the length rule, and how similar the slogans are at each setting.
- Note which setting gives the best mix of variety and rule-following.
Check yourself
- Why does raising temperature make repetitive output less likely but constraint violations more likely?
- A chatbot rambles. Why is lowering max tokens the wrong first fix?
Test yourself
6 multiple-choice questions and 9 flashcards on this lesson, shuffled every time, with the reason behind every answer.