How does a chatbot pick the next word?

A chatbot doesn't write its answer and then show it to you. It writes one small piece of text, reads everything so far, and writes the next piece. For each piece, the model gives every possible option a score, and a step called softmax turns those scores into odds. Then a few lines of ordinary code roll dice weighted by those odds, and whatever comes up goes on the end. A setting called temperature reshapes the odds before each roll. That's why the same question can get two different answers.

It writes one piece, then loops

The reply that streams onto your screen isn't a finished thought being typed out for show. It's being decided as you watch.

Here is the loop. The model reads your message plus everything it has written so far. It scores every possible next piece of text. One piece gets picked and stuck on the end. Then the whole thing runs again with that piece included. A Hugging Face explainer by Patrick von Platen describes this kind of generation as breaking a sentence's probability into a chain of next-word probabilities, one after another. A long answer is hundreds of trips around the same loop.

"Word" is a simplification. Models work in tokens, which can be a whole word, part of a word or a bit of punctuation. GPT-2's paper puts its vocabulary at 50,257 tokens, so each trip scores tens of thousands of candidates.

Scores become odds

The raw scores aren't probabilities yet. A softmax step fixes that. In a 2015 paper, Geoffrey Hinton and his co-authors describe softmax as converting each raw score into a probability "by comparing" it with all the others. The results are all positive and add up to one.

After "The cat sat on the", a model might give "mat" the best odds, "floor" and "sofa" a decent chance, and "moon" almost none. I made those odds up for this page. Real ones depend on the model.

The model doesn't choose. The sampler does

Once the odds exist, the model's job for that step is done. Picking one piece is a separate step, usually called the sampler.

The simplest sampler always takes the top choice. Hugging Face's documentation calls that greedy search and warns that on longer text "it begins to repeat itself". A 2020 paper by Ari Holtzman and colleagues, The Curious Case of Neural Text Degeneration, found the same problem more broadly. Aiming for the most likely text "leads to text that is bland and strangely repetitive."

The fix is to roll dice instead. Sampling, in Hugging Face's words, "randomly selects a token based on the probability distribution". The dice are loaded. "Mat" comes up most often, "sofa" sometimes, and "moon" very rarely, but never quite never.

Temperature reshapes the odds

Before the roll, the scores can be divided by a number called temperature. Hinton's paper explains that the usual setting is 1 and that a higher temperature "produces a softer probability distribution". The odds even out, so long shots get a real chance. Below 1, the gaps between scores grow and the favourite pulls further ahead. Von Platen's explainer notes that as temperature heads to zero, sampling "becomes equal to greedy decoding", which means no dice at all.

Try it below. Slide the temperature, then build a sentence one piece at a time. The odds are illustrative, but the softmax and the dice are the real maths.

Set the temperature to 0 and start over a few times. You get the cat on the mat, purring, every time. Push it to 2 and the cat ends up on the keyboard or the moon, and the next pieces bend to fit.

It never goes back

Look at what happens after a long shot. Once "moon" is on the page, the model reads it like everything else, and the next odds are about cats on moons. In the basic loop there's no step that deletes a word. A model that picks an odd word early has to make sense of it for the rest of the answer, which is one way it can talk itself into a corner. When a chatbot seems to correct itself, it does that by writing more pieces, like "Actually, that's wrong", after the old ones.

Why the same question gets two answers

Rolling dice is the usual reason. Two runs of the same question can take different pieces near the start, and every later piece builds on those.

Turning the temperature to zero should end that, and in theory it does. In practice it doesn't always. A September 2025 post from Thinking Machines Lab, by Horace He, notes that LLM services "are still not deterministic in practice" even at temperature 0. The main cause it found is that a server batches your request with other people's, and the size of the batch can change the arithmetic slightly.

More dice tricks

Temperature is one knob among several. Top-k sampling, von Platen explains, keeps only the k most likely next pieces and rolls among those. Top-p, the nucleus sampling from Holtzman's paper, keeps the smallest set of pieces whose odds add up past a threshold, so the shortlist grows when the model is unsure and shrinks when it's confident.

None of this makes the answer true. The dice pick a piece that fits what came before, which is why it's worth knowing how to check an AI's explanation. And the loop has gaps of its own. How Qwenputer works shows what happens when a model has to do exact arithmetic one token at a time.

Play with the dice yourself

Learnfold's course How LLMs actually work starts with this exact loop. Its first lesson, One word at a time, has a sampler you can drag from temperature 0 to 2, and the next lesson, on tokens, shows what models read instead of words. Every lesson ends with follow-up questions, so a curiosity rabbit hole keeps going. Building a course from your own question is part of Learnfold's paid subscription. It's an AI tutor, built on the very loop this page describes, so it can be wrong too.