Fine-tuning: When It's Worth It

⏱ ~8 min read · 🎧 7 min listen · ✎ 3 quizzes · 🧪 ~25 min exercise

Somewhere in your first month as an FDE, a customer will lean forward and say it: "Can't we just train the model on our data?" It sounds like the obvious move — and it's usually the wrong one. This lesson is deliberately conceptual: what fine-tuning actually changes, what it costs, the short list of cases where it genuinely wins, and how to redirect the conversation when it doesn't. No training runs today; the skill is the decision.

What fine-tuning actually is

Fine-tuning takes a trained model and continues training it on your examples — hundreds to thousands of input→output pairs — nudging the weights so your kind of output becomes the most probable continuation.¹ You met the idea in Lesson 01: pretraining built the base model, and a smaller tuning stage made it an assistant. Customer fine-tuning is a third, far smaller pass of the same mechanism, usually via a provider's API or parameter-efficient methods rather than touching all the weights.

Now the sentence that saves engagements: fine-tuning changes behavior, not knowledge. It's excellent at style, format, tone, and task-specialization — the how of the output. It is unreliable for adding facts, because Lesson 01's compression story still holds: training distills patterns from examples; it doesn't file documents for lookup. A model tuned on your policy docs learns to sound like your policy docs — and will still hallucinate clause numbers. Facts belong in the context window: that's RAG.¹

What it costs — the part the pitch decks skip

Four bills arrive with a tuning project. Data: hundreds to thousands of high-quality labeled examples — someone has to produce and maintain them. Compute and hosting: training runs plus, often, a premium to serve your custom model. Maintenance: base models deprecate; every upgrade means re-tuning and re-validating, forever. And the quiet one, the eval dependency: without Lesson 07's harness you literally cannot tell whether the tuned model beats the prompted one — so evals come first, always.²

The decision ladder

The field has converged on an order of operations: prompt → few-shot → RAG → fine-tune, and stop at the first rung that passes your eval.³ Each rung is cheaper, faster to iterate, and easier to undo than the next. You've spent five lessons on the first three rungs precisely because most engagements end there. Fine-tuning is the rung you climb to when you've measured the others falling short.

So when does tuning win? Three honest cases. Behavior a prompt can't hold: a strict brand voice or exotic output format that drifts even with few-shot examples stuffed in every call. Making a small model punch up: tune Haiku on a narrow task until it matches Opus on that task — at Lesson 02's prices, a 5× per-call saving that compounds at volume, with faster tokens too. A task genuinely unlike pretraining: proprietary DSLs, niche medical coding — distributions the base model never saw.¹

Note what all three winning cases share: a stable, narrow, high-volume task with an eval to prove the gain. Hamel Husain's Rechat case study is the model citizen — they fine-tuned only after their eval system exposed failures prompting couldn't fix, and the same eval data became the training set.² That's the flywheel: production logs → eval cases → training data.

Translating the customer's request

Back to the meeting. "Train it on our data" is almost never a request for gradient descent — it's a requirement wearing a solution costume. Translate it: "answers should use our documents" → grounding → RAG (Lesson 06). "it should always respond in our format" → structured outputs (Lesson 04) or few-shot (Lesson 03). "it gets our jargon wrong" → usually a glossary in the system prompt before it's a tuning dataset. The FDE move is extracting the requirement, proposing the cheapest rung, and defining the eval that would justify climbing higher.³

And the scoping math seals it: a tuning project is weeks (data, training, evals, hosting) before the first improved answer, while a RAG index or prompt revision ships the same week, reversibly. Even when tuning is right, you'll usually deploy the prompted/RAG version first and tune against its production logs later — the ladder isn't just cheaper, it's the natural sequencing.

🧪 Exercise: four verdicts, on paper

No API calls today — this lab is the meeting you'll actually be in. In labs/0008-tuning-decisions.md you'll find four realistic customer requests: an insurer who wants the model "trained on 20 years of claims manuals," a fintech whose brand voice keeps drifting, a logistics firm running 2 million classification calls a month on an expensive model, and a hospital wanting discharge summaries in a rigid format.

For each, write a short decision memo: the requirement behind the request, your recommended rung on the ladder (with the reasoning), what eval would prove success, and — where you do recommend tuning — the data plan and the maintenance cost you'd disclose. That document is your artifact; the review prompt below grades it like a skeptical solutions architect.

You are a senior AI engineer reviewing my work: four decision memos (from labs/0008-tuning-decisions.md) on whether to fine-tune, each covering the real requirement behind the customer's request, my recommended technique, the eval that would prove success, and costs where tuning is proposed.

Grade each memo as Strong / Adequate / Missing, with one sentence of evidence:
- I translated each request into the underlying requirement (grounding, format, cost, voice) instead of taking "train it" literally.
- My recommendations follow the ladder — prompt, few-shot, RAG, then tuning — and stop at the first rung that could pass an eval, with reasoning.
- Where I recommend fine-tuning, I identify the narrow stable high-volume task, a data plan, AND the maintenance/re-tuning cost I'd disclose.
- Every memo names a concrete eval (test set + grader + threshold) — no recommendation rests on "it should be better."

Be skeptical — challenge my weakest memo first. Then ask me 2–3 follow-up questions the customer's CTO would ask. Finish with the single highest-leverage improvement.

My work follows below.

Check yourself — hold the ladder

Three requests from the field. Don't scroll up — find the requirement behind each, and the right rung. Wrong picks stay live.