Taming the Dice

1000 identical requests. 80 different answers.

In September 2025, researchers at Thinking Machines Lab did something almost nobody bothers to do: they sent the exact same prompt to an LLM one thousand times, with the randomness dial turned all the way to zero, on the same hardware, changing nothing. In a perfect world you would get the same answer a thousand times.

They got 80 different answers. The most common one showed up 78 times out of 1000. And the very first word where the runs started to disagree was the 103rd token in. Same prompt. Same settings. Same machine. Eighty different completions. [1]

If you have ever asked a chatbot the same thing twice and gotten two different answers and quietly wondered whether you were losing it: you were not. This is the default behaviour of the technology, and it is stranger and more fixable than the usual "well, it is random" shrug suggests. By the end of this you will know exactly why it happens, at every layer, and what you can actually do about each one.

• 01How it works
• 02The dice: sampling
• 03Why it varies
• 04The knobs: seed & friends
• 05Prompting that travels
• 06Forcing structure
• 07How to test it
• 08The bestiary of pain
• 09The playbook
• 10Opinion

It is autocomplete. Enormous, uncanny, expensive autocomplete.

Strip away everything and a language model does exactly one thing: it reads the text so far and predicts the next chunk of text. That is the entire job. It does not "look up" an answer, it does not have a database of facts it queries, it does not decide what it thinks and then phrase it. It guesses the next piece, adds that piece to what it has read, and guesses again. Over and over, one piece at a time, until it decides to stop.

The phone keyboard that suggests "morning" after you type "good" is doing a tiny, dumb version of the same thing. Scale that up by a few hundred billion knobs, train it on a large fraction of everything humans have written, and the guessing gets good enough to write code, explain calculus, and pass the bar exam. But it is still, mechanically, guessing the next piece. Hold onto that. It explains almost everything weird that follows.

The five words you need: token, embedding, transformer, attention, softmax

• Token. Models do not see letters or even whole words. Text is chopped into tokens, sub-word chunks. "Tokenization" might become token + ization. Each token is just a number to the model.
• Embedding. Each token number is turned into a long list of numbers, a vector, that places it in a giant space where "king" and "queen" or "Paris" and "France" sit in geometrically meaningful spots. This is the model's sense of meaning, learned, not programmed.
• Transformer. The engine. A stack of identical layers introduced in the 2017 paper literally titled Attention Is All You Need, which threw out the older step-by-step approach so every word could be processed at once and any word could directly influence any other. [2]
• Attention. The trick that makes it work. At each layer, every token asks a question and every token offers an answer, and the ones that match get to influence each other (more on this in a second).
• Softmax. The final step that turns the model's raw scores into probabilities that add up to 100%, so one token can be picked. This little function is the secret star of this entire article.

Attention, in one honest sentence

Picture every word in a sentence at a meeting. Each one raises a hand with a question ("who here is relevant to me?"), wears a name-tag key ("here is what I am about"), and holds some content to share. Each word compares its question against everyone's name-tag, and blends in the content of whoever matches best. In "the animal did not cross the street because it was too tired," attention is the mechanism that lets "it" figure out it means the animal and not the street. That soft, learned lookup, repeated at every layer for every token, is the whole magic trick. [3]

Weighting the dice, then deciding how hard to shake

Back to step 4 of Fig 1. The model has produced a raw score, a logit, for every token in its vocabulary. Logits are just unbounded numbers, not probabilities, so we need to turn them into a clean distribution that sums to 1. That is softmax. It exponentiates each score and divides by the total, which does two things at once: kills negatives and makes the big scores dominate.

See that T tucked into the exponent? That is temperature, and it is the single most misunderstood dial in AI. Before softmax runs, every logit gets divided by T. That one division is the whole knob. Watch what it does with three candidate words whose logits are 2.0, 1.0, and 0.1.

top_p and top_k: throwing away the tail before you roll

Temperature reshapes the dice. The other two knobs do something different: they chop off the unlikely tail entirely before the roll, so the model can never blurt out something from deep left field. They just chop it two different ways. Take one distribution over five tokens and watch.

The rest of the dice bag, quickly

• frequency_penalty and presence_penalty (OpenAI-family, range -2 to 2). Frequency scales with how many times a token already appeared (kills verbatim repetition). Presence is a flat one-time nudge the moment a token has appeared at all (pushes toward new topics). Handy when the model loops. [6]
• repetition_penalty (open-source / Hugging Face). Same idea, multiplicative. 1.0 means no penalty.
• min_p (open-source). Keep tokens whose probability is at least min_p × (probability of the top token). The cutoff floats with the model's confidence: strict when it is sure, lenient when it is not. [7]

Two layers of randomness, and only one of them has an off switch

Layer one is the dice. Above temperature 0, the model draws from the distribution instead of always taking the top token. That is deliberate, and you switch it off by setting temperature to 0 (greedy decoding). Easy. If that were the whole story, temperature 0 would give you the same answer forever, and this article would be four sections shorter.

Layer two is the one that ruins your week. Set temperature to 0, change nothing, run the same prompt again, and you can still get a different answer. That is the 80-out-of-1000 result from the opening. The dice are off. So where does the wobble come from? Two places, and the second one surprised even the people who build these systems.

Culprit A: computers cannot add up

Floating-point addition is not associative. On paper (a + b) + c equals a + (b + c). On a computer, with finite precision, it does not always. The Thinking Machines write-up gives the clean example: (0.1 + 1e20) - 1e20 = 0, but 0.1 + (1e20 - 1e20) = 0.1. Same three numbers, different grouping, completely different result. [1]

A GPU running a model adds up millions of these numbers across thousands of cores in parallel. If the order of those additions changes between runs, the result drifts by a few bits. A few bits changes a logit by a hair. A hair is enough to flip which token is the argmax on some step. One flipped token cascades into a totally different sentence. That is the story most people know, and it is true. It is also, it turns out, not the main thing.

Culprit B: your request has roommates

Here is the genuinely counter-intuitive finding. On a fixed batch, the same matrix multiply on the same GPU gives bit-identical results every time. The forward pass has no random atomic operations, it is run-to-run deterministic. So why the variance? Because inference servers batch your request with other people's requests, and how many roommates you get depends on how busy the server is at that instant, which changes constantly. The kernels are not "batch-invariant," so the exact arithmetic done on your tokens depends on the size of the batch you happened to land in. Same request, different crowd, different math. As the researchers put it, "the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies." [1]

Culprit C: the model changed under you and did not send a memo

The third one is mundane and brutal. Providers update infrastructure, quantization, and routing behind a stable-looking model name. Your prompt that ran on one configuration last month runs on a subtly different one today, same name, different brain. There is even a measured version of this: one study found GPT-4's accuracy at telling prime numbers from composite ones fell from 84% in March 2023 to 51% in June 2023, on the same task, same model name, three months apart. [8]

You cannot buy determinism. You can buy a receipt.

First, three words people use interchangeably and absolutely should not, because knowing which one you actually need saves you from chasing an impossible one:

For reproducibility, the OpenAI family (and Azure OpenAI, which mirrors it) gives you two things that work together:

And here is the honesty, straight from Microsoft's own page, which to their credit does not oversell it: "Determinism isn't guaranteed... it's currently not uncommon to still observe a degree of variability in responses. Identical API calls with larger max_tokens values will generally result in less deterministic responses even when the seed parameter is set." Translation: shorter answers are more stable than long ones, and even with everything pinned, you have narrowed the variance, not deleted it. [9]

The knobs, by provider (as of Aug 2026)

Good prompting is just narrowing the dice before you roll

Here is the reframe that makes prompt engineering click. Every instruction you add reshapes the probability distribution before sampling ever happens. A vague prompt leaves the model a huge spread of plausible next words, so tiny wobbles send it anywhere. A specific, well-structured prompt collapses that spread to a narrow peak, and now the same wobbles barely matter. You are not "asking nicely," you are pre-loading the dice.

The anatomy of a prompt that survives a model swap

The three big vendors publish separate prompt guides, and the striking thing is how much they agree. That agreement is the model-agnostic core, the stuff that works whether you are on GPT, Claude, or Gemini:

• Be clear, specific, and complete. Do not assume the model knows your context, spell it out. Anthropic leads its list with "be clear and direct," Google frames the whole game as "clear and specific instructions." [13] [14]
• Use delimiters and structure. Markdown headers and XML tags to fence off sections. OpenAI recommends an Identity / Instructions / Examples / Context skeleton, Anthropic explicitly recommends XML tags. [15]
• Set the role. A system message that fixes tone, goal, and constraints, separate from the user's query.
• Show, do not just tell (few-shot). A handful of input/output examples pins the format harder than any adjective. Google is blunt: "prompts without few-shot examples are likely to be less effective." Watch the ceiling though, too many examples and the model overfits to them. [14]
• Ask for the reasoning when accuracy matters. Let the model think in steps. And if you can afford it, sampling several reasoning paths and taking the majority answer (self-consistency) measurably lifts accuracy, +17.9 points on the GSM8K maths benchmark in the original paper. [16]
• Specify the exact output shape. "Return a JSON object with keys x, y, z" beats "give me the answer" every single time. This is the bridge to section 06.

Stop asking for JSON. Make JSON the only thing it can say.

Prompting nudges the format. It does not enforce it. You write "return JSON," and 96 times out of 100 you get clean JSON, and the other 4 times you get a chatty "Sure! Here's your JSON:" wrapped in a markdown fence, and your parser explodes at 2am. For anything a machine reads next, "usually valid" is a bug with a delay on it.

The real fix is not a better prompt, it is constrained decoding. Remember step 4, where the model has a probability for every token? A grammar sits on top of that step and sets the probability of every schema-invalid token to zero before the roll. The model literally cannot emit a token that would break the structure, because those tokens are not on the dice anymore.

How much does it matter? OpenAI's own numbers: with Structured Outputs, gpt-4o-2024-08-06 hit a "perfect 100%" on a hard JSON-schema-following eval. The same model with prompting alone managed 93% ("insufficient for production"), and older gpt-4-0613 scored under 40%. That is the gap between "hope" and "guarantee." [17]

Run it a hundred times and measure the disagreement

Everything so far narrows the variance. Testing is how you find out whether you narrowed it enough. And the method is exactly the experiment from the opening, turned into a tool: take a prompt, run it many times over a set of known cases, and measure how much the outputs disagree. High disagreement on a task that should have one answer is not noise to average away, it is a red flag telling you where the model is unsure and where a human needs to look.

Non-determinism is one monster in a whole zoo

If you are shipping LLMs to real users, the wobble from section 03 is just the one you noticed first. Here is the rest of the menagerie, each with a number attached, so you know which ones to actually lose sleep over. None of these are reasons not to build. They are the things you build around.

Ten moves, in order, from "just guessing" to "governed"

• 01Default to temperature 0 for anything with a right answer. Turn the dice off before you do anything clever. Save the heat for brainstorming.
• 02Move one knob, not two. Temperature or top_p. Every vendor says it, everyone ignores it, do not be everyone.
• 03Pin the exact model snapshot (gpt-4o-2024-08-06, not "latest"). A model change should be a dated decision you made, not a Tuesday you survived.
• 04Set seed, log system_fingerprint where they exist. You are not buying determinism, you are buying a receipt that explains drift after the fact.
• 05Structure the prompt: role, delimiters, one or two examples, and the exact output shape. Narrow the distribution before sampling ever runs.
• 06For machine-read output, use structured outputs / constrained decoding. Make invalid output impossible, not merely unlikely.
• 07Let it think, then make it format. Free-text reasoning first, constrained JSON second, so the cage does not crush the thinking.
• 08Test by running N times and measuring disagreement. Exact match, then semantic similarity, then a judge, with a human on the tail. Gate every prompt change on it.
• 09Treat "the answer changed" as an expected event, like a flaky network call. Log everything that feeds a decision. High variance is a routing signal, not noise.
• 10Scope agent permissions as if the model will occasionally do the dumbest allowed thing. Second signature on anything irreversible.

"It's just predicting the next word" is true, and it is not the dunk people think

Mechanically, everything in this article comes back to one fact: the model is sampling from a probability distribution over the next token. Calling that "guessing" or "autocomplete" is not an insult, it is an accurate description of the algorithm. People say it like a debunking. It is not one. The genuinely startling thing is that a machine built entirely on "predict the likely next piece of text" writes working code, explains ideas it was never explicitly taught, and is right often enough to have changed how a lot of us work. The mechanism being simple does not make the behaviour un-remarkable. It makes it more so.

Here is where I land. The two loud camps are both, in my view, dodging the actual problem. "It is just autocomplete, ignore it" and "it reasons like a person, trust it" are the same laziness pointed in opposite directions. Both let you skip the uncomfortable truth: this is a genuinely new kind of tool that is probabilistically excellent and individually unverifiable, and that combination does not map cleanly onto anything we had before. A model right 97% of the time is a phenomenal drafting partner and a terrible sole decision-maker for the 3% where being wrong is expensive, precisely because you cannot tell, from a single confident answer, which bucket you are in.

So the skill that actually matters is not "prompt hacking" and it is not deciding whether to be a believer or a cynic. It is knowing where your 3% lives and building the second signature exactly there. Determinism tooling, structured outputs, evals, logging, none of it removes the underlying fact that the thing is rolling dice. It just makes the dice honest: shaped, bounded, watched, and written down. For anything you would have to explain later to a user, a client, or a court, that might be the whole game.

Everything factual above traces to these

Vendor behaviour and benchmarks change monthly. Before you build on any specific claim here, go read the source. That is not a disclaimer, it is the single most useful habit in this whole field.

• [1] Thinking Machines Lab - Defeating Nondeterminism in LLM InferenceThe 80-of-1000 result, the batch-invariance thesis, and the fix. The single most important recent source here.
• [2] Vaswani et al. - Attention Is All You Need (2017)The transformer paper.
• [3] Jay Alammar - The Illustrated TransformerThe best beginner-friendly visual walkthrough of attention and Q/K/V.
• [4] Holtzman et al. - The Curious Case of Neural Text Degeneration (2019)Origin of nucleus (top_p) sampling.
• [5] AWS Bedrock - Anthropic Claude message parameterstop_k range and defaults; the temperature-or-top_p rule.
• [6] Microsoft Learn - Azure AI Foundry chat completionsfrequency_penalty / presence_penalty ranges.
• [7] Hugging Face - Text generation configmin_p, repetition_penalty, top_k/top_p defaults.
• [8] Chen, Zaharia, Zou - How Is ChatGPT's Behavior Changing over Time? (2023)The 84% → 51% prime-number drift.
• [9] Microsoft Learn - Reproducible output with Azure OpenAIseed / system_fingerprint mechanics and the "not guaranteed" caveat.
• [10] OpenAI Cookbook - Reproducible outputs with the seed parameterThe mechanism Azure mirrors, and the "small chance responses differ" line.
• [11] Anthropic - Glossary (temperature / determinism)"Even with temperature set to 0... identical inputs may produce different outputs."
• [12] Google - GenAI SDK GenerateContentConfigGemini's best-effort seed wording.
• [13] Anthropic - Prompt engineering best practices
• [14] Google - Gemini prompting strategiesfew-shot, decomposition, output format.
• [15] OpenAI - Prompt engineering guide
• [16] Wang et al. - Self-Consistency Improves Chain of Thought Reasoning (2022)The +17.9 point GSM8K result.
• [17] OpenAI - Introducing Structured Outputs in the API100% vs 93% vs under-40% schema adherence; constrained decoding.
• [18] OpenAI - Structured Outputs guideStructured Outputs vs JSON mode.
• [19] Anthropic - Structured outputs
• [20] Outlines - Structured generationOpen-source constrained decoding; see also llama.cpp GBNF grammars.
• [21] Tam et al. - Let Me Speak Freely? (2024)Format restrictions can degrade reasoning.
• [22] Zheng et al. - Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)80%+ agreement, plus position/verbosity/self-enhancement biases.
• [23] promptfoo - Assertions & metricsDeterministic, similarity, and model-graded assertions.
• [24] Vectara - Hallucination LeaderboardLive rates (1.8%-24.2% at time of writing).
• [25] Menlo Ventures - 2024 State of Generative AI in the EnterprisePilot-failure causes; RAG at 51% of deployments.
• [26] OWASP - LLM01:2025 Prompt Injection
• [27] Liu et al. - Lost in the Middle (2023)The U-shaped long-context accuracy curve.
• [28] Barnett et al. - Seven Failure Points When Engineering a RAG System (2024)
• [29] Stanford HAI - AI Index 2025The $20.00 → $0.07 per-million-token cost drop.
• [30] OWASP - Top 10 for LLM Applications 2025
• [31] OpenAI - Rate limits