How LLMs work: tokens, context windows and sampling

1.Tokens: the unit the model reads and writes

A model never sees characters or words directly. It sees tokens. Before Claude reads a prompt, the text is broken into a sequence of tokens, and every answer Claude writes is produced as tokens too. A token can be a whole word, part of a word, a single character or even a byte. Large tokens make processing efficient, and small ones let the model handle rare words it has never seen whole.

For English, a rough rule is that one Claude token covers about 3.5 characters. The ratio changes with language, so the same sentence in another language can produce a different token count. This matters well beyond trivia. Tokens are what a context window is measured in, what max_tokens caps, and what a request is billed by. When an API response reports usage, it reports tokens, not words or characters.

Sources1

2.Next-token generation and why it hallucinates

Generative AI has more in common with a very sophisticated autocomplete than with a search engine. It writes an answer token by token, each time choosing what tends to follow the text so far. Nothing in that loop looks up a fact or checks one. The model keeps extending a sequence in the way its training makes most natural.

That one mechanism explains both what the model is good at and how it fails. When a task resembles patterns the model has seen many times, such as summarizing, reformatting or explaining a common concept, the likely continuation is usually also the correct one. In sparse or novel territory, or wherever the task needs "true" told apart from "sounds true", the likely continuation can be fluent, confident and invented. A made-up citation reads exactly like a real one because it was produced the same way.

Use this to decide where to spend scrutiny. The more precise a claim is (a statistic, a URL, a direct quote, a job title), the more it needs checking. Citations, uncertainty signalling, constrained generation and generator-verifier loops all exist to push this limitation further out.

Sources2

3.The context window is the model's working memory

The context window is all the text the model can refer to while generating a response, including the response itself. It is separate from the huge body of data the model was trained on. Think of it as working memory for one request. If something is not in the window, the model cannot attend to it in that request.

Everything in the request takes up space: the system prompt, every message in messages (tool results, images and documents included) and your tool definitions. The output Claude generates for the turn also counts, including any extended thinking. In a multi-turn conversation over the API, each response becomes part of the input for the next turn and earlier turns are kept in full, so the window fills as the conversation goes on. Every response reports what it consumed in its usage field.

Window size is fixed per model. Claude Opus 5.5, Opus 4.8, Sonnet 5.5, Sonnet 4.6 and the other current flagship models have a 1M-token window, and 1M is the default with no beta header. Claude Sonnet 4.5, Claude Haiku 4.5 and other older models have 200k. A 700,000-token contract therefore fits in one request to Opus 4.8 but not to Sonnet 4.5. Bigger is not automatically better, though. As the token count grows, accuracy and recall degrade, which is called context rot. What you put in the window matters as much as how much room it has.

Sources3

4.Why examples steer the model without retraining it

A prompt can include no examples, one, or several: zero-shot, single-shot and multi-shot prompting. How to write each one is covered in [Zero-, one- and few-shot prompting](lesson:6.2). The question here is why they work at all.

Examples work because they sit in the context window. The model picks up the pattern there during the request, the same way it picks up any other text it attends to. Nothing about the model changes. Changing the model itself is fine-tuning, which means further training a pretrained model on additional data, and the Claude API does not currently offer it. Whatever a few-shot prompt teaches lasts only for that request. The next request has to carry the examples again, and they take up window space every time.

Next-token generation also explains why example choice matters. The model continues whatever patterns the context shows, including ones you did not intend. If every example has the same length or opening phrase, the output will copy that as well. Diverse examples that cover edge cases teach the pattern you want and not an accident of how the examples were written.

Sources14

5.Sampling: why the same prompt gives different answers

Run the same request twice in fresh conversations and you will usually get two different answers. This is how generation works, not a bug. At each step the model has many possible next tokens, and it samples one of them instead of always taking the most likely. The variation between runs is that sampling at work.

Three parameters shape the sampling:

- temperature controls how random the choices are. It defaults to 1.0 and ranges from 0.0 to 1.0. Lower values keep to the most probable phrasing; higher values allow rarer, more surprising words. Use values near 0.0 for analytical or multiple-choice work and near 1.0 for creative work. - top_k samples only from the K most likely options for each next token, which cuts off the long tail of unlikely choices. - top_p (nucleus sampling) cuts the options off by cumulative probability instead of by count.

Two cautions apply. First, temperature 0 does not make Claude deterministic: identical inputs can still give different outputs across API calls. Design for variation by validating outputs and not relying on byte-identical responses. Second, on current models these knobs are mostly locked. On Claude Opus 5.5, Opus 4.8, Opus 4.7, Sonnet 5.5, Sonnet 5 and the other Claude 5 models, any non-default temperature, top_p or top_k returns a 400 error. On older models the restriction applies only while thinking is on.

Sources256

6.Non-determinism: building for outputs that vary between runs

Non-determinism is the property that identical inputs may produce different outputs across API calls. It breaks anything that compares two responses: a regression test that diffs today's answer against a stored one, a cache keyed on output text, an audit that expects a rerun to reproduce a decision. The cause is the per-token sampling described in [the sampling section above], and it holds on Anthropic's first-party inference service and through third-party cloud providers alike. Moving a workload to a cloud provider does not buy repeatable output.

No setting has ever switched it off. Temperature 0 never guaranteed identical outputs, even on older models that accepted it, so code migrating to a newer model loses nothing real when it drops temperature=0. Drop the assumption of repeatability along with the parameter.

The variation reaches token counts, not just wording. With adaptive thinking, Claude decides its thinking allocation per request, so thinking token usage, billed as output, changes from one run of the same prompt to the next. A cost estimate taken from a single run can be off in either direction; size budgets and max_tokens from the range seen over many runs.

A pipeline that expects non-deterministic output checks it the way a lab handles a noisy measurement:

- Validate structure and meaning (a parseable JSON object, the right label, the required fields) instead of byte equality. - Run each eval case more than once and note which parts hold steady and which shift between runs. - Store the response you acted on when a decision must be auditable, because a rerun is a new sample, not a replay.

Sources173

Exam traps

Each one states something that sounds right. Open it to see what is actually true.

1. 1.Setting temperature to 0 makes Claude's output fully deterministic, so identical prompts always return identical text.Why is that wrong?

   Temperature 0 favours the most probable choices, but identical inputs can still produce different outputs across API calls.

   Covered in Sampling: why the same prompt gives different answers

2. 2.Prompt caching frees context window space, because cached tokens are not re-sent.Why is that wrong?

   Caching changes the price of the cached prefix, but those tokens still occupy the context window.

   Covered in The context window is the model's working memory

3. 3.Lowering temperature is always a valid way to make answers more consistent on any Claude model.Why is that wrong?

   On Claude 5 models and Opus 4.7 or later, any non-default temperature, top_p or top_k returns a 400 error, whether or not thinking is used.

   Covered in Sampling: why the same prompt gives different answers

4. 4.Non-determinism is a quirk of Anthropic's own API; running Claude through a third-party cloud provider returns identical outputs for identical inputs.Why is that wrong?

   Identical inputs can produce different outputs on Anthropic's first-party inference service and on third-party cloud providers alike.

   Covered in Non-determinism: building for outputs that vary between runs