An LLM is a next-word predictor that got very good
A large language model (LLM) is a neural network trained on billions of words of text to do one simple thing: predict the most likely next token. A “token” is a chunk of text — roughly three-quarters of a word. Given “The sky is”, the model assigns probabilities to the next token and picks one (“blue”, “gray”, “falling”…). Repeat that millions of times and you get fluent paragraphs.
The surprising part isn’t the trick — it’s that next-token prediction, at enough scale, produces models that can summarize, translate, write code, and reason. No one fully understands why scale unlocks these abilities; the field’s best answer is “emergence.”
Tokens, parameters, and context — the three numbers that matter
- Parameters — the model’s learned weights, roughly “how much it knows.” Modern models run from a few billion to over a trillion.
- Tokens — the unit of text the model reads and writes. Models are measured in “context window”: how many tokens of input they can hold at once.
- Context window — the model’s working memory. 128k-token windows are common now, letting a model read an entire book in one prompt.
Why do they hallucinate?
An LLM doesn’t look anything up; it samples from a probability distribution over the next token. When the training data doesn’t cover a fact — or when the model is pushed past what it knows — it confidently generates something plausible but wrong. “Hallucination” is the polite word for “made it up.” This is a feature of the architecture, not a bug that’s about to be patched.
How they’re trained, in three stages
- Pretraining — read a huge slice of the internet and learn to predict text. This is the expensive, GPU-hungry part.
- Fine-tuning — train on curated examples (instructions, dialogue) to make the model helpful and follow directions.
- Reinforcement learning (RLHF) — humans rank outputs; the model learns to prefer the ones we like.
What they’re good at — and not
Good: drafting, summarizing, translating, coding, explaining, brainstorming, and transforming text between formats. Weak: exact math, current facts (no real-time knowledge), true planning, and anything requiring a source of truth. That’s why the best systems pair an LLM with tools, search, and databases.
New to the vocabulary? See the AI glossary. Want to run one yourself? Read how to run LLMs locally.
Guides & explainers
Filed under: AI & Machine Learning, AI, LLM