Capstone: Customer Problem → Demoable Solution

⏱ ~7 min read · 🎧 6 min listen · ✎ 3 quizzes · 🧪 ~3–5 hr capstone build

Eight lessons ago an LLM was a mysterious oracle; now it's a next-token sampler you can prompt, program, ground, and measure. This final lesson assembles those skills into the shape you'll actually be paid for: a full mini-engagement. You get a realistic customer brief; you deliver a working, evaluated solution and the memo that sells it. No new machinery today — the lesson is the sequence.

The FDE engagement loop

Every engagement, whatever the domain, runs the same five steps: scope → architect → build → prove → demo. The failure pattern in the field is doing them in the wrong order — building first, scoping never, proving on demo day with three cherry-picked examples. Walk them in order and the project mostly runs itself.

Scope (Lessons 02, 07): before code, two artifacts. The cost-and-latency envelope — tokens per call, calls per month, model tier, the arithmetic a CFO can audit. And the success criteria — specific, measurable, on a named test set. Writing the eval's acceptance threshold before building keeps demo day honest.

Architect (Lessons 04–06): now the two questions from the agent lesson — are the steps predictable? is success verifiable? — pick your pattern. Then place the seams: where structured outputs make prose into data, where a workflow routes, where RAG grounds claims in the customer's documents. Most support-and-operations problems come out as a routing workflow with retrieval, not an autonomous agent — boring is a feature at a customer site.

Build (Lessons 03–04): start at the bottom of the ladder — clear prompts, few-shot examples, fenced data — and add machinery only where the eval says the simple version falls short. You have working reference code for every component: your five labs are the parts bin.

Prove (Lesson 07): the eval isn't a checkbox at the end; it's the engine of the build loop. Run it on every change; keep the score history. The deliverable isn't "it works" — it's "94% on the 30-case held-out set, up from 61% for the naive prompt, at 1.1¢ per ticket."

Demo (Lesson 08's discipline): show the live system on inputs the audience supplies, then show the eval dashboard, then the roadmap — including where fine-tuning would and wouldn't earn a place later. Leave behind the memo: scope math, architecture sketch, eval results, next steps. That document is what gets the contract extended.

Your brief: Harbourline Freight

The full brief lives in labs/0009-capstone-brief.md. The short version: Harbourline, a mid-size freight company, gets ~30,000 support emails a month. They want each email classified (category + urgency + structured fields), answered with a draft reply grounded in their policy handbook when it's a standard query, and escalated to a human when it isn't. They've asked, of course, whether you'll "train the AI on our data."

Deliverables — each one maps to a lab you've already done: a scoping memo with the cost model at two tiers (Lab 02); a working pipeline — structured extraction → route → grounded draft or escalation (Labs 03–06 assembled); an eval suite with a held-out test set, code grading for the classification and a validated judge for draft groundedness (Lab 07); and a one-page recommendation memo answering the fine-tuning question (Lab 08's format).

How to work it

Budget three to five hours, in engagement order — scope memo first, and don't write pipeline code until your success criteria and test set exist. Reuse your lab code shamelessly; assembling proven parts is the skill being tested. When the eval plateaus, diagnose like Lesson 06 taught you: is the failure in retrieval, extraction, routing, or generation? Fix the failing component, not the vibe.

"Done" looks like this: a stranger could run your eval and see your number; your memo would survive a skeptical CTO; and you can answer, with course-grounded reasoning, why there's no autonomous agent and no fine-tuning in v1. Then take the review prompt below — it grades the whole engagement, not just the code.

You are a senior AI engineer reviewing my capstone engagement for Harbourline Freight (labs/0009-capstone-brief.md): a scoping memo, a pipeline (structured extraction → routing → RAG-grounded drafts or escalation), an eval suite with scores, and a fine-tuning recommendation memo.

Grade each deliverable as Strong / Adequate / Missing, with one sentence of evidence:
- Scoping memo: token math and monthly cost at two model tiers, plus SMART success criteria written BEFORE the build.
- Pipeline: structured outputs at the code seams, a workflow (not an agent) with a justified routing step, and drafts grounded in retrieved handbook passages with citations.
- Eval suite: a held-out test set, code-graded classification, a rubric-driven judge for groundedness validated against human labels, and a score history showing at least one measured improvement.
- Recommendation memo: the fine-tuning question answered with the ladder, an eval-gated criterion for revisiting, and honest costs.

Be skeptical — challenge the weakest deliverable first. Then ask me 2–3 questions a customer CTO would ask before signing off. Finish with the single highest-leverage improvement.

My work follows below.

Check yourself — run the engagement

Three moments from a capstone-shaped project. Don't scroll up — sequence and architecture from the loop. Wrong picks stay live.