How Generative AI Answers — A Sentence Built One Piece at a Time

When You Ask the Same Question Twice

You open a chatbot window at lunchtime and type, "Recommend a story to listen to on my commute home." A few lines of recommendations come back right away. If you put the exact same sentence into a new window one more time, this time you get an answer with slightly different recommended stories and a different tone. Both are completely fine sentences, but they are not the same sentence.

You see something similar every day on your phone keyboard. Once you type "Tonight's dinner," a few likely follow-up words appear above the keyboard. If you tap one of them, that word is attached, and the next recommendation appears immediately. The keyboard guesses the word that should come next by looking at the words you typed in front.

A chatbot's answers are also created by repeating this process in the same direction. It looks at the text placed in front, picks one next piece, attaches that piece, and then picks again. The differences from a keyboard are three. A keyboard usually guesses by looking at just the few words typed right before it, but a chatbot examines all the preceding text in the chat window at once. A keyboard shows candidates and a human picks one, but a chatbot even does the picking itself. And a chatbot commonly mixes in a bit of a lottery when choosing. This lottery is the main reason why the two lunchtime answers turned out differently.

The sections that follow trace this process step by step. We cut the text into pieces, assign probabilities to the next piece, and draw one among them. After that, we look at the practice leading up to being able to do that, and the structure that decides which parts of the preceding text to look at. Finally, we trace how a single lunchtime question becomes an answer from beginning to end.

Cut the text into pieces

When the sentence "Recommend a story I can listen to on my way home from work" goes into a chatbot, it is first chopped up into small pieces. Let us first name what is doing the cutting.

Artificial intelligence is a term that broadly refers to technology aimed at having computers perform tasks that humans used to do using intelligence. Among them, generative AI is artificial intelligence that creates new content, such as text or pictures. For creating pictures, methods from a different lineage than this article are also used, and one of those lineages achieved record-breaking performance in making pictures and videos (DOI: 10.1145/3626235). Here, we will look only at the text-creating side.

A model is a calculation device that receives an input and produces an output, and the numbers inside it can be adjusted to make it closer to the desired answer. What creates answers inside a chatbot is a text-handling model. A language model is a model that receives text and outputs the text that should follow it.

The text entering a language model is first cut into tokens. A token is the unit by which a language model cuts and handles text, sometimes being a single word, and other times being a fragment of a word or a punctuation mark. Frequently used words tend to become a single token entirely, while rare words are split into a few smaller pieces. Cutting it this way allows even unfamiliar words or rare names to be written by connecting known pieces (arXiv:1508.07909, a preprint version published on arXiv — a version uploaded before undergoing peer review). Whether "퇴근길" becomes a single token, splits into "퇴근" and "길", or is chopped up finer than a single Korean letter depends on the cutting rules predefined by each model.

The cut tokens are converted into numbers that can be calculated inside the model before going in (DOI: 10.17721/2706-9699.2025.1.06). Therefore, to the model, a lunch-hour request is just a line of tokens lined up. This line is the starting point of the next section.

Calculating the Probability of the Next Token

If you think of what to say after "It's raining, so I", "brought" or "used" comes to mind first, and "ate" almost never comes to mind. Language models do the same thing with numbers.

Next-token prediction is the task of looking at the tokens placed in front and assigning a probability to each candidate for what token will come right next. The candidates are all the tokens the model knows, and at every step, a probability is attached to every single one of them (DOI: 10.48550/arxiv.2407.01082, preprint version released on arXiv). It works in a way where a high probability is attached to "brought", and a probability close to 0 is attached to "ate".

The response is created by repeating this process. One next token is determined and attached to the end of the line, and the entire lengthened line is used again as the preceding text to calculate the probability of the following token. This method of using what it just made as input again and stringing it together piece by piece is the basic framework by which large language models generate text (DOI: 10.17721/2706-9699.2025.1.06). The language model inside the chatbot repeats the task of choosing and attaching one next token until the response is finished, thereby creating a response.

It is the same direction of work as the keyboard recommendations in the opening scene. However, in this framework, the front piece of the response immediately becomes the preceding text for the back piece. If one piece changes in the front, the probabilities assigned to everything after it change as well.

Why the Answer Changes for the Same Question

Once probabilities are assigned to each candidate after "the umbrella," one of them must now be chosen. The easiest method is always to pick the top candidate with the highest probability.

A 2019 study revealed that when allowing free-form generation of long texts, always picking only the top choice tends to make the writing flat and prone to repeating the same words strangely, and subsequent studies also address the same issue in open-ended text (arXiv:1904.09751, a preprint version made public on arXiv; DOI: 10.18653/v1/2023.findings-acl.262). For short answers with a single clear correct answer, the method of picking only the top choice is also commonly used. Even with the same model, just changing the method for selecting the next token greatly alters the quality of the generated text (arXiv:1904.09751). Assigning probabilities and choosing one are separate tasks, and the latter drastically changes the shape of the text.

Therefore, language models often sample according to probabilities (DOI: 10.48550/arxiv.2407.01082). Sampling is the task of picking one among the candidates according to their assigned probabilities. Candidates with high probabilities are picked frequently, and candidates with low probabilities are also picked occasionally. The main reason the answer changes for the same question lies in the fact that this drawing is included within the process of generating the response. Because the piece picked earlier becomes the preceding text for what follows, a difference of a single step can snowball into a different sentence further down.

It is also possible to decide how boldly to perform the sampling. Temperature is a setting that determines how daringly the language model will pick even low-probability candidates when selecting the next token. The lower the temperature, the more the share is skewed toward higher-probability candidates so the top choice is picked more often and the answer wavers less. Lowering it all the way is virtually the same as picking only the top choice. The higher the temperature, the more evenly the share is spread so candidates with lower probabilities are picked more often. Compared to typing on a keyboard, a low temperature is someone who almost always taps the first suggestion, while a high temperature is someone who frequently taps suggestions further down. Temperature is a setting rewritten right before the model's assigned probability table is sampled, so the numbers inside the model remain unchanged. Usually, the service provider sets this in advance.

Neither extreme is entirely good. When using the common method of keeping only up to a certain share starting from the highest-probability candidates, raising the temperature too much easily results in incoherent text or text that repeats the same words (DOI: 10.48550/arxiv.2407.01082). The approach of picking only the top choice tends to become flat in long texts, as seen earlier. Reducing this trade-off—where broadening the sampling increases diversity but tends to degrade quality—is the goal of various selection methods (DOI: 10.18653/v1/2023.findings-acl.262).

A commonly used method is to adjust the temperature while cutting off candidates on the very low-probability tail and sampling only from the remaining candidates (DOI: 10.18653/v1/2025.acl-long.1278). The side that cut off the implausible tail increased diversity while losing neither fluency nor coherence (arXiv:1904.09751). The results of these methods depend heavily on the chosen settings, so the method of figuring out which method and settings to select is itself being researched (DOI: 10.18653/v1/2025.acl-long.1278).

Massive Autocomplete Practice

In the probability table of the previous section, the reason "took care of" is high and "ate" is low is because the model has gone through a vast amount of text beforehand. The LLaMA models released in 2023 used trillions of tokens as raw material, using only publicly available data (DOI: 10.48550/arxiv.2302.13971, a preprint published on arXiv).

The name given to what is done with that material is learning. Learning is the process of slightly adjusting the numbers inside a model to fit example data or experience, also called training. The numbers that get adjusted also have a name. Parameters are the numbers in a model determined through learning, and how the model calculates inputs to produce answers is determined by these numbers. GPT-3, released in 2020, had 175 billion such numbers (DOI: 10.48550/arxiv.2005.14165, a preprint published on arXiv). How these numbers are shifted slightly and what changes when they are made larger are covered in another piece in this series, "Neural Networks and Parameters."

Pre-training is the first stage of learning where parameters are adjusted by repeating the task of guessing the next token across a tremendous amount of text. This practice does not require humans to attach correct answers separately. If you show the beginning of a sentence and have the model guess the next token, the correct answer is simply the very next token already written in that sentence. Since the text provides its own problems and answers, there is plenty of text to practice with, and this single practice serves as the foundation used broadly for various applications (DOI: 10.1145/3605943). The basic capabilities of large language models are built through this large-scale learning using text without human-attached answers (DOI: 10.1007/s11704-026-60308-3). Compared to keyboard autocomplete, this is the stage of practicing guessing the next word using a massive amount of text.

This practice alone does not make a partner that always answers questions appropriately. Making the model larger does not automatically make it follow user intent better, and even large models can output text that is untruthful, harmful, or unhelpful (arXiv:2203.02155, a preprint published on arXiv). That is why a refinement stage is attached after pre-training (DOI: 10.1007/s11704-026-60308-3).

Fine-tuning is the task of taking a pre-trained model and training it one more time with a curated collection of examples to refine it to answer in the desired way. Once refined, the probability table for the next token itself changes. This is done by having humans write and show desirable answers for various requests, and training the model to follow those answers (arXiv:2203.02155). Next, humans compare and rank the multiple answers generated by the model. Reinforcement learning from human feedback is a method of using the ranks given by humans as a signal to adjust the model one more time toward generating answers that humans choose as better. In a common procedure, a scoring model that mimics those rankings to give a score to each answer is trained first, and the language model is adjusted in the direction where that score increases. The learning method of reinforcement learning itself is covered in another piece in this series, "AI Models."

In a 2022 study, the answers of a 1.3 billion parameter model refined in this way were preferred in human evaluations over the answers of the 175 billion parameter GPT-3 on the requests collected by that study. Truthfulness improved and harmful outputs decreased. Still, the authors noted that this model still makes simple mistakes, and viewed human feedback refinement as a promising direction for aligning with human intent (arXiv:2203.02155). The rankings given by humans only point to the answers humans consider better, and mistakes remained even after truthfulness improved.

There is also research comparing the contributions of the two stages: pre-training and refinement. A 2023 study fine-tuned a model pre-trained with 65 billion parameters using only 1,000 curated pairs of requests and responses, without using reinforcement learning or human rankings. In human comparative evaluations, this model was judged to be equal to or better than GPT-4 43% of the time, with GPT-4 being better the rest of the time. Based on this, the authors interpreted that language models learn almost everything they know during pre-training, and learning how to give good answers can be taught with even a small number of examples (DOI: 10.48550/arxiv.2305.11206, a preprint published on arXiv). This is the interpretation of one study. In this study as well, the model refined with a few examples had its probability table changed, altering how it answered.

The structure that decides where to look in the preceding text

"Minsu left his wallet on the dining table, and only after getting on the bus did he realize it was missing." In this sentence, what "it" refers to is determined by "wallet," which is far ahead in the text. All the tokens of the preceding text are not equally important for guessing the next token, and sometimes a single distant spot holds the answer.

Attention is a mechanism that calculates on the fly which parts of the input to pay attention to and how much. The Transformer is a model architecture built using attention as its backbone. When first proposed, it was an architecture for translation, stripping away the complex structures used by previous translation models and leaving the job of connecting tokens entirely to attention. Because translation quality was better and it was easy to distribute computations across multiple places, training time was also greatly reduced (arXiv:1706.03762, a preprint version published on arXiv). Large language models build on this architecture, repeatedly calculating across multiple layers how much each token should refer to the tokens placed before it, and then assigning probabilities for the next token. There is a limit to the length of the preceding text that can be viewed at once (DOI: 10.17721/2706-9699.2025.1.06).

Where attention actually points has been analyzed by looking inside a large model called BERT. BERT is a model with a pre-training method different from chatbots that continue writing text, but it shares the point of using attention. Among the various attention mechanisms inside it, some pointed to verb objects or words pointing to the same target with an accuracy that the authors described as surprisingly high. Some attention looked at punctuation marks or positions at fixed distances, and some looked broadly across the entire sentence. The authors concluded that a considerable amount of grammatical information is contained in BERT's attention (DOI: 10.18653/v1/w19-4828). Whether to view this as the same thing as human understanding is outside the question this analysis answers. The details of the internal calculations are passed on to another piece in this series, "Neural Networks and Parameters."

From a Single Question to an Answer

Let us follow a lunchtime request from beginning to end.

When "Recommend a story I can listen to on my way home" comes in, the sentence first becomes a line of tokens. The model that receives this line has finished pre-training on a massive amount of writing and has been refined once more to answer. One example is the method of using model answers written by people and rankings assigned by people, and the refinement method differs for each model (DOI: 10.1007/s11704-026-60308-3).

The model looks at the entire line, goes through attention calculations across multiple layers, and assigns probabilities to all candidates that could become the first token of the answer. According to the set temperature, one of them is picked and attached to the end of the line. With the lengthened line, it assigns probabilities again, picks again, and attaches again. A mark indicating that the answer is finished is also included among the tokens, and when that token is picked or a set length is reached, the answer stops. The answer is the result of repeating these steps up to that point.

If you put in the same request in a new window, even if the probability table for the first step calculated by the model is the same, the token picked can be different as long as sampling is turned on. Once a different token is picked, the preceding text for the subsequent steps changes, and the answer that continues after that can also change. In a procedure with sampling turned on, you cannot always expect the same answer to the same question. Conversely, if you turn off sampling and always make it choose the top-ranked one (commonly referred to as setting the temperature to 0), almost the same answer comes out for the same request.

There is no step of searching the internet anywhere in these steps. What the model uses in the basic procedure are the parameters adjusted during training and the text placed in front of the model right now. The text placed in front of the model can include text invisibly added by the service in addition to the chat window text. Search is a separately attached device, and chatbot services with that device attached are common. Sometimes relevant documents are found in advance and placed in front of the model, and sometimes when the model asks for a search while answering, an external program puts the found text into the line. Either way, the retrieved text enters as part of the text placed in front of the model. Limitations that remain in preventing factual inaccuracies are being studied even in models with search attached (DOI: 10.1145/3703155).

The token that gets picked is usually the one with higher probability. In a model that has only finished pre-training, high probability means it is likely to come in that spot in light of the learned text, and in a post-refined model, it has also shifted toward what humans would choose as better. Either way, text made this way can be very plausible. GPT-3 in 2020 produced news articles that human evaluators had a hard time distinguishing from articles written by humans (DOI: 10.48550/arxiv.2005.14165). Plausibility and truth are different questions. Language models producing plausible but untrue content is actually reported (DOI: 10.1145/3703155), and why that happens and what kinds exist are covered in another part of this series, "AI Errors."

There is one last question remaining. The question of whether this procedure is "understanding." What we followed here is the procedure by which an answer is made, and whether to call that procedure understanding is not decided by this explanation alone.

Where to Look Further

The story one level above this article is in "How Training to Guess the Next Word Becomes a Conversation: The Minimum Foundation for Reading Language Models". It covers the traps created by the method of cutting into tokens, what and how things improve when you scale up the model, and the phenomenon where things change just by giving examples in the chat window without retraining.

What search-attaching mechanisms fix and what they cannot fix is continued in "Does Adding Search Make Hallucinations Disappear?", and why plausible-sounding yet wrong answers come out is covered in "Why Do LLMs 'Lie'?". The debate surrounding whether this procedure can be called understanding or thinking is handled by "Does AI Really 'Think'?". Whether and how what you write in the chat window changes the answer can be seen in "Do Prompts Really Work?".

Other parts of the same series continue as well. What models and training are is in "AI Models", how parameters are adjusted is in "Neural Networks and Parameters", and the various kinds of being wrong are in "AI Errors".

Source Papers

Writings cited by arXiv number or arXiv DOI are preprint versions published on arXiv (versions uploaded before peer review), and were indicated when first cited in the text.

• "Diffusion Models: A Comprehensive Survey of Methods and Applications" (2023) — DOI: 10.1145/3626235 — Diffusion models are a family of deep generative models that achieve record-breaking performance in image synthesis, video generation, and molecular design. The research is organized into three branches: efficient sampling, likelihood estimation, and specialized structural data.
• "Neural Machine Translation of Rare Words with Subword Units" (2015, arXiv preprint version) — arXiv:1508.07909 — Enables open-vocabulary translation by writing rare or unseen words as a string of subword pieces. Includes byte-pair-encoding-based segmentation, improving translation scores over dictionary-replacement methods.
• "Principles of Large Language Models (LLM)" (2025) — DOI: 10.17721/2706-9699.2025.1.06 — Surveys autoregressive next-token generation, self-attention transformers, tokenization and embedding, next-token selection methods (greedy, top-k, top-p, and temperature) and output randomness, the trade-off between consistency and creativity, and context length limits.
• "Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs" (2024, arXiv preprint version) — DOI: 10.48550/arxiv.2407.01082 — Language models draw the next token from the probability distribution of the entire vocabulary at each decoding step. Top-p yields incoherent or repetitive outputs especially at high temperatures, while min-p adjusts the threshold based on the top-1 token probability to improve quality and diversity together.
• "The Curious Case of Neural Text Degeneration" (2019, arXiv preprint version) — arXiv:1904.09751 — Using likelihood as the decoding objective makes text bland and strangely repetitive. Quality varies greatly by decoding strategy alone even in the same model, and nucleus sampling, which cuts off the unbelievable tails of the probability distribution, increases diversity without losing fluency and consistency.
• "Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding" (2023) — DOI: 10.18653/v1/2023.findings-acl.262 — Greedy and beam search suffer from degeneration and diversity issues, while temperature, top-k, and nucleus sampling yield diverse but low-quality outputs. Proposes minimum Bayes risk decoding to solve this diversity-quality trade-off.
• "Balancing Diversity and Risk in LLM Sampling" (2025) — DOI: 10.18653/v1/2025.acl-long.1278 — Sampling decoding balances diversity and quality through temperature adjustment and tail cutting. Because results depend heavily on configuration values, it systematically estimates the capabilities of tail-cutting methods and provides guidelines for setting value selection.
• "LLaMA: Open and Efficient Foundation Language Models" (2023, arXiv preprint version) — DOI: 10.48550/arxiv.2302.13971 — Models based on 7 billion to 65 billion parameters trained on trillions of tokens using only public data. The 13 billion model outperforms GPT-3 (175 billion) on most benchmarks.
• "Language Models are Few-Shot Learners" (2020, arXiv preprint version) — DOI: 10.48550/arxiv.2005.14165 — At a time when task-specific fine-tuning after large-scale pre-training was common, the 175-billion-parameter autoregressive language model GPT-3 was evaluated with a few examples without fine-tuning. It remains weak on some datasets and generates news articles that human evaluators have difficulty distinguishing from human-written ones.
• "Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey" (2023) — DOI: 10.1145/3605943 — The core idea is reusing representations learned once on general tasks across multiple tasks, and self-supervised text-rich language modeling is that general task. Organizes approaches and limitations in post-pre-training fine-tuning, prompting, and text generation.
• "A Survey of Large Language Models" (2026) — DOI: 10.1007/s11704-026-60308-3 — Large-scale self-supervised pre-training establishes core capabilities, while post-training including supervised fine-tuning and reinforcement learning enhances downstream task adaptation, alignment, and safety. Organized into four axes: utilization, evaluation, and open problems.
• "Training language models to follow instructions with human feedback" (2022, arXiv preprint version) — arXiv:2203.02155 — Scaling up does not make models follow user intent better. Supervised fine-tuning with demonstrations followed by reinforcement learning from human feedback on output rankings. The 1.3-billion InstructGPT is preferred over the 175-billion GPT-3 with improved truthfulness, reduced toxic outputs, and small performance degradation on public datasets. Still makes simple mistakes but is a promising direction.
• "LIMA: Less Is More for Alignment" (2023, arXiv preprint version) — DOI: 10.48550/arxiv.2305.11206 — Fine-tuned a 65-billion LLaMA using only 1,000 curated examples (no reinforcement learning or preference models). 43% of human comparisons judged it equal to or better than GPT-4. Authors' interpretation: knowledge is mostly learned during pre-training, and a small amount of instruction data is sufficient to teach good outputs.
• "Attention Is All You Need" (2017, arXiv preprint version) — arXiv:1706.03762 — Transformers consisting solely of attention without recurrence or convolution. Better quality on two translation tasks, easier parallelization, and significantly shorter training time. Generalizes to syntactic parsing as well.
• "What Does BERT Look at? An Analysis of BERT's Attention" (2019) — DOI: 10.18653/v1/w19-4828 — BERT's attention shows patterns focusing on delimiter tokens, fixed positional intervals, and the entire sentence, with some heads matching verb objects, noun determiners, and coreference with high accuracy. Syntactic information is substantially contained in attention.
• "A Survey on Hallucination in Large Language Models" (2024) — DOI: 10.1145/3703155 — Language models tend to generate plausible-sounding but fact-untrue content. Classifies, causes, detects, and mitigates the problem, and discusses current limitations when retrieval-augmented models tackle this issue.