Glossary · AI Engineering Course

Foundation model: A large model trained on a broad sweep of data that can be adapted to many tasks without being retrained. LLMs are the text kind; there are also image/audio/multimodal ones. As an FDE you build on these, you don't train them.
LLM — large language model: A foundation model for text. Under the hood it is a next-token predictor: given some text, it outputs a probability distribution over what comes next.
Token: The unit an LLM actually reads and writes — usually a chunk of a word, not a whole word. "tokenization" might be 3 tokens: token, iz, ation. Prices, context limits, and speed are all measured in tokens, not words.
Tokenization: The step that chops input text into tokens before the model sees it (and stitches output tokens back into text). Why the model can be weird about spelling, rare words, and counting characters.
Next-token prediction — autoregression: The core loop: predict the next token, append it to the input, predict again, repeat. Everything an LLM does is this loop running fast. The single most important mental model in the course.
Parameters — weights: The billions of numbers, fixed after training, that encode the patterns the model learned. They don't change when you use the model — they are the "compressed" version of its training text.
Pretraining: The expensive phase where the model learns to predict next tokens over a huge corpus of text. This is where its "knowledge" comes from — and why it's frozen at a training cutoff.
Inference: Actually running the trained model to generate output — the part you do as a builder, via an API call. Costs money and time per token.
Context window: The maximum amount of text (in tokens) the model can consider at once — your prompt plus its answer. It has no memory outside this window; anything the model should "know" for a call must be inside it.
Prompt: The input text you give the model. Because output is conditioned entirely on the input, the prompt is your main control surface — small wording changes shift the output.
System prompt: Standing instructions the app developer sends with every API call, framing the whole conversation ("You are a support assistant for Acme…"). Mechanically it's just more tokens in the context window, but models are trained to weight it heavily — it's where an FDE encodes the customer's rules.
Few-shot prompting: Putting two or three worked input→output examples in the prompt so the model imitates their pattern. The most reliable prompt-level way to pin down output format. "Zero-shot" = instructions only, no examples.
Sampling — temperature: Because the model outputs a distribution, it picks the next token by sampling from it. "Temperature" tunes how random that pick is: low = more deterministic/repetitive, high = more varied/creative. Why the same prompt can give different answers.
Input & output tokens: The two sides of the billing meter. Input tokens are everything you send (prompt, history, documents); output tokens are what the model generates. Output tokens cost ~5× more per token and dominate latency, because each one is a pass of the next-token loop.
Fine-tuning: Continuing a model's training on your own input→output examples so your kind of output becomes the most probable. Changes behavior (style, format, task specialization) — unreliable for adding facts (that's RAG). Last rung of the ladder: prompt → few-shot → RAG → tune.
Distillation: Using a large model to generate or grade training examples, then fine-tuning a small model on them — inheriting the narrow skill at the small model's cost and latency.
Eval — evaluation: Unit tests for model behavior: a test set of real inputs (plus edge cases), a runner, a grader, and a tracked score. The evidence that answers "how do you know it works?" — and the regression net for every prompt or model change.
LLM-as-judge: Using a second model to grade outputs against a written rubric — for qualities code can't check and humans can't scale (tone, groundedness). Only trustworthy after its verdicts are validated against human labels.
RAG — retrieval-augmented generation: Retrieve the passages relevant to a question from the customer's documents, put them in the context window, and instruct the model to answer only from them, with citations. The standard cure for hallucination on private facts.
Embedding: A vector (list of numbers) representing a text's meaning, positioned so that similar meanings land near each other. Turns "find related text" into geometry — the engine behind semantic search in RAG.
Chunking: Splitting documents into retrieval-sized pieces (typically a few hundred tokens) before indexing. Retrieval returns chunks, not whole documents — chunk boundaries and size materially affect answer quality.
Agent: An LLM that runs tools in a loop to achieve a goal: it requests an action, your code executes it, the result goes back into context, repeat until done. The model chooses the steps; your code keeps the authority (and the guardrails).
Workflow: A system where your code decides the steps and models fill in the smarts — prompt chaining, routing, parallelization, orchestrator–workers, evaluator–optimizer. Predictable and testable; the right default when the steps are known in advance (contrast: agent).
Structured outputs: Attaching a JSON schema to an API call so the platform guarantees the reply conforms — via constrained decoding (tokens that would break the schema are masked out before sampling). Guarantees shape, not correctness.
Tool use — function calling: You describe functions your code offers (name, purpose, JSON-schema parameters); the model can reply with a structured request to call one. The model never executes anything — your code runs the function and sends the result back.
Streaming: Receiving the response token-by-token as it's generated, instead of waiting for the whole answer. Changes nothing about cost — only about when the user starts seeing words. The default for anything user-facing.
Time to first token — TTFT: How long before the first piece of the answer appears. Roughly the time the model spends reading your input. What a user feels as responsiveness — streaming optimizes for it.
Hallucination: When the model produces fluent, confident text that is false. Not a bug bolted on — a direct consequence of next-token prediction: it generates plausible continuations, with no built-in check for truth. Managing this is core FDE work.

This is a living document. New terms get added as lessons introduce them. If a term here is unclear, paste it (with its definition) into any AI assistant and ask for a different angle or a concrete example.