Large Language Models

Large language models generate responses by predicting one token at a time, based on patterns they’ve learned from massive datasets. This explainer covers their core components: data, architecture, training, and how transformer models use context to simulate conversation.

Tokens are words or subwords (e.g., prefixes) that a large language model processes to generate a response. During training, the model is taught that if it ever generates a special token called end-of-sequence, its response is complete.

Large language models identify patterns in massive datasets of books, code, and unlabeled text from which to generate coherent responses. GPT-3 was trained on about 45 terabytes of data and uses 175 billion parameters—each parameter being a value the model adjusts as it learns.

Instead of understanding the meaning of words, large language models break up sentences into words and subwords, assigning each a set of numbers called tokens. Grammar rules are then "learned" by finding associations between these sets of numbers.

It changed how text was translated from a sequential, "one word at a time" method to one where every word in a text is processed in parallel. Advances in positional encoding and self-attention also helped models recognize context and word order better.

Transformer architecture is the neural network framework upon which large language models predict and generate text based on user inputs. This interactive lets you explore the "thinking" a GPT does to generate responses based on your input.

Training large language models begins with pretraining the model to predict the next word in a sequence based on finding patterns in massive amounts of text. Patterns are then fine-tuned to model human-written dialogue and to align responses with user preferences.

In large language models, temperature adjusts the randomness in text generation by modifying how often words with lower probabilities are selected next in the sequence. Lower temperatures produce more obvious outputs while higher temperatures produce more creative responses.

Unlike large language models, which do not take follow-up actions after generating their output, agentic AI uses an initial prompt to take multiple steps, learn from its environment and outcomes, and continue working without follow-up human prompting.

AI models use this parameter to prevent repeated outputs by sometimes choosing less likely next words during sequential generation. However, this randomness—alongside insufficient data and training—may cause hallucinations of incorrect results.