Large Language Models

Large language models generate responses by predicting one token at a time, based on patterns they’ve learned from massive datasets. This explainer covers their core components: data, architecture, training, and how transformer models use context to simulate conversation.

Tokens are words or subwords (e.g., prefixes) that a large language model processes to generate a response. During training, the model is taught that if it ever generates a special token called end-of-sequence, its response is complete.

Founded in 1996, the Archive promotes open access and digital preservation of online information and, as of 2024, contains more than 145 petabytes of web pages, books, audio recordings, software programs, and other digital content.

Large language models identify patterns in massive datasets of books, code, and unlabeled text from which to generate coherent responses. GPT-3 was trained on about 45 terabytes of data and uses 175 billion parameters—each parameter being a value the model adjusts as it learns.

Instead of understanding the meaning of words, large language models break up sentences into words and subwords, assigning each a set of numbers called tokens. Grammar rules are then "learned" by finding associations between these sets of numbers.

Transformer architecture is the neural network framework upon which large language models predict and generate text based on user inputs. This interactive lets you explore the "thinking" a GPT does to generate responses based on your input.

Training large language models begins with pretraining the model to predict the next word in a sequence based on finding patterns in massive amounts of text. Patterns are then fine-tuned to model human-written dialogue and to align responses with user preferences.

In large language models, temperature adjusts the randomness in text generation by modifying how often words with lower probabilities are selected next in the sequence. Lower temperatures produce more obvious outputs while higher temperatures produce more creative responses.

Unlike large language models, which do not take follow-up actions after generating their output, agentic AI uses an initial prompt to take multiple steps, learn from its environment and outcomes, and continue working without follow-up human prompting.

AI models use this parameter to prevent repeated outputs by sometimes choosing less likely next words during sequential generation. However, this randomness—alongside insufficient data and training—may cause hallucinations of incorrect results.