Structured Output & Function Calling

⏱ ~7 min read · 🎧 5 min listen · ✎ 3 quizzes · 🧪 ~30 min lab

In the Lesson 03 lab you hit a wall that every AI engineer hits in week one: the model's answer was text, and your program needed data. You resorted to fishing JSON out of the reply with a regex — and even your best prompt sometimes returned chatty extras or a malformed object. This lesson removes the duct tape. Two API features — structured outputs and function calling — turn the model from a text generator into a component you can program against.

The problem: prose isn't parseable

Software integration runs on contracts: your code expects a severity field that is exactly one of three strings, every single time. A next-token sampler makes no such promise. Prompting "return only JSON" plus few-shot examples gets you to mostly — but "mostly" at 10,000 calls a day means a daily pile of parse failures, and a 2% silent-corruption rate is worse than a crash at a customer site.

You could catch failures and retry — and production systems do — but retries cost latency and tokens, and they only paper over format errors, not schema drift like a severity of "URGENT!!". The right fix moves the contract from polite request to enforced guarantee.

Structured outputs: the schema becomes a guarantee

Modern APIs let you attach a JSON schema to the call, and the platform guarantees the reply conforms — valid JSON, right fields, right types, allowed values only.¹ In practice you rarely write raw schemas: you define a Pydantic model (Python) or Zod schema (TypeScript) and the SDK converts it. Your parsing code stops being defensive; a whole failure class disappears.

The mechanism is worth knowing, because it's Lesson 01 again: constrained decoding. At each step of the next-token loop, tokens that would break the schema are masked out of the distribution before sampling. The model literally cannot emit an invalid character. Nothing about its "understanding" changed — the sampler's menu did.

Function calling: the model asks your code to act

Structured output makes the model's answers machine-readable. Function calling (Anthropic calls it tool use) goes further: you describe functions your code offers — name, purpose, parameters as a JSON schema — and the model can respond with a request to call one: {"name": "lookup_order", "input": {"order_id": "A-1729"}}.²

Hold on to the key reframe: the model never executes anything. It emits a structured request — the API even sets stop_reason: "tool_use" and waits. Your code decides whether to run the function, runs it, and sends the result back as a new message. The model then continues with that result in context. Control, permissions, and side effects stay entirely in your code — a sentence you will repeat in every security review a customer runs on your design.

One craft point does most of the work in practice: tool definitions are prompts. The model chooses tools by reading their names and descriptions, so Anthropic's docs push you to write them like documentation for a new colleague — what it does, when to use it, what each parameter means, what it returns.² Most "the model picked the wrong tool" bugs are description bugs.

What this unlocks

With these two features, the model becomes a programmable component: an extraction pipeline that feeds a database, a router that classifies and dispatches tickets, an assistant that can look up live order status instead of hallucinating it. And one question should be nagging you: if the model can request a tool call, and get the result, and request another… what happens when you put that in a loop? That's the next lesson — the agent loop.

🧪 Lab: kill the regex, then hand the model a phone

The lab file is labs/0004-structured-extraction.py — same setup as before. Part 1 re-runs Lesson 03's ticket extraction, but with the schema enforced by the API instead of begged for in the prompt: the regex fishing is gone, the parse step cannot fail, and you'll see the checker go clean on format errors.

Part 2 defines a lookup_order tool, asks a question that requires it, and walks the wire format: the tool_use stop reason, the structured arguments the model produced, your code returning a result, and the model's final grounded answer. You will watch the moment of hand-off — model asks, your code acts — happen in plain sight.

You are a senior AI engineer reviewing my work: the output of my structured-output and function-calling lab (labs/0004-structured-extraction.py), plus my written answers to two design questions: (1) what schema enforcement does and does not guarantee, and (2) who executes a tool call and why that matters for security.

Grade each item as Strong / Adequate / Missing, with one sentence of evidence:
- My Part 1 output shows schema-conforming extractions with no parse/regex step, and I can explain constrained decoding in one sentence (invalid tokens are masked from the sampling distribution).
- I clearly distinguish shape guarantees from correctness — and name evals as the tool for the latter.
- My Part 2 trace shows the tool_use stop reason and structured arguments, and my explanation puts execution and permissions in MY code, not the model.
- My tool description would let a new colleague use the function correctly without asking questions.

Be skeptical — challenge my weakest explanation first. Then ask me 2–3 follow-up questions a customer's security engineer would ask. Finish with the single highest-leverage improvement.

My work follows below.

Check yourself — design the integration

Three integration decisions. Don't scroll up — reason from the mechanism. Wrong picks stay live.