Choosing between prompting, RAG and fine-tuning

Step 1 Understand it

The problem it solves. Teams often jump to fine-tuning when a better prompt or retrieval would do, wasting weeks, or try to prompt their way around missing knowledge. Choosing the right tool — or the right combination — saves cost and time and gives better results.

Watch it happen

The idea

Try in this order. Prompting first, because it is cheapest. RAG when the model needs knowledge it does not have. Fine-tuning when it needs a new behaviour, style or format that prompts cannot give reliably.

How it works, step by step

1. Start with prompting: clear instructions and examples. Measure on your eval set.
2. If the model lacks knowledge (your documents, recent or per-user facts), add RAG or a lookup tool.
3. If it has the knowledge but won't reliably follow a behaviour, style or format despite good prompts and examples, consider fine-tuning.
4. Fine-tuning can also make a small model do one narrow task as well as a large one, cutting cost and latency.
5. Combine them when needed: a fine-tuned model that follows your format can still use RAG for facts.
6. Re-measure after each step, and stop as soon as quality is good enough.

Picture it

flowchart TD
  P["Try prompting first"] --> Q1{"Good enough?"}
  Q1 -->|"yes"| DONE["done"]
  Q1 -->|"no"| Q2{"What's missing?"}
  Q2 -->|"knowledge or facts"| RAG["add RAG"]
  Q2 -->|"a behaviour, style or format"| FT["fine-tune"]

Analogy

A new employee. First give instructions (the prompt). Give them the company handbook for facts (RAG). Send them on a training course to change how they work (fine-tuning).

Where the analogy breaks

Training courses are a poor way to learn facts that change often, and fine-tuning is too. Use RAG for changing knowledge.

Worked example

'Answer from our 2026 price list' → RAG. 'Always reply in our strict legal-letter format' → prompt first, and fine-tune only if it keeps failing.

Must-know points

• Order: prompt → RAG (knowledge) → fine-tune (behaviour).
• RAG for facts that change or are private; fine-tuning is poor at storing facts.
• Fine-tuning for format, style, tone and narrow skills.
• Fine-tuning needs data, evals and maintenance; prompts are instant to change.
• They are not exclusive.

Common mistakes

• Fine-tuning to add knowledge that changes weekly.
• Fine-tuning before trying a good prompt with examples.
• Using RAG to fix a formatting problem.
• Not measuring, so the "improvement" is a guess.

Step 2 Go further (optional)

Optional. The lesson above is written to be enough on its own. If you want a second explanation, start with the one marked Worth it.

• Read

  AI Engineering, Ch 7 (finetuning: when to finetune)

  Chip Huyen · book you own

• Read

  Patterns for Building LLM-based Systems & Products

  Eugene Yan

Step 3 Practise

Try each exercise yourself first: on paper, out loud or in code. Then open the answer and compare it with yours.

1. ExplainSay it in plain words, with no notes.

   Explain the order of choices with the new-employee picture.

   Show answer

   Think of how you would get a new employee working well. First, give clear instructions: that is prompting, and it is cheap, quick and often enough. If they keep lacking information, give them the company handbook to look things up: that is RAG, for facts that live in documents and change. Only if they need to work differently, in a new style or a new skill that instructions can't produce, do you send them on a training course: that is fine-tuning, which is slower, costs more and needs good training examples.

   You go in that order, and only move to the next step when you can show the earlier one isn't enough, with the same test used each time.

2. DeriveWrite the maths, or the steps, from memory.

   Choose prompting, RAG or fine-tuning for each case, and say why: answering from company policies; always replying in strict JSON; writing in a company's tone; answering about yesterday's news.

   Show answer

   Answering from company policies: RAG. The facts live in documents that change, answers must be up to date and should cite the policy. Fine-tuning would bake in a snapshot and can't reliably store facts.

   Always replying in strict JSON: structured outputs, which is a prompting-level feature, not fine-tuning. The API can enforce the schema directly. (Fine-tuning only for a model or provider that can't do this.)

   Writing in a company's tone: prompting first (a style guide and a few examples of real writing). If that isn't consistent enough at scale, fine-tuning, because tone is behaviour, which is what fine-tuning is good at.

   Answering about yesterday's news: RAG, or a search tool. The model's training stopped long before yesterday, and the news changes daily; you need fresh retrieval, never fine-tuning.

   The pattern: knowledge that changes → retrieval; format → structured outputs; behaviour and style → prompting, then fine-tuning.

3. BuildCode it from scratch in NumPy, with no ML library.

   Take one task. Try prompting first, then RAG. Measure both with your eval harness and write a one-paragraph decision.

   Show answer

   Use the same eval harness for both, so the comparison is fair. "Prompting only" sends the question with no documents; "RAG" retrieves three chunks first. Same model, same questions, same judge.

   On this task RAG wins clearly: without documents the model can only answer what it happens to remember (it knows famous facts, like the 178 wine samples, but not the details only the documents contain), so it guesses or declines on those; with RAG it answers and cites. A one-paragraph decision might read: "Prompting alone scored 4/6 (measured on the local qwen2.5:7b): it knew the famous figures but missed the two that only the documents contain. RAG scored 6/6 with citations, for about 1,500 extra input tokens per question. We choose RAG. Fine-tuning isn't needed: the failures were missing knowledge, not behaviour."

   The habit matters more than this result: decide with numbers from the same eval, and write the reason down.

   pythonCopy

   from llm import ask, text_of
   from evals import run_eval, summary
   from rag import CHUNKS, VECTORS, answer
   
   def prompting_only(q):
       return text_of(ask(f"{q}\nIf you don't know, say \"I don't know\".")), []
   
   def with_rag(q):
       return answer(q, CHUNKS, VECTORS)[:2]
   
   for name, fn in [("prompting only", prompting_only), ("RAG", with_rag)]:
       print(f"--- {name}")
       summary(run_eval(fn))

4. BreakShow when it fails, and why.

   Try to teach the model new facts by fine-tuning on a few examples, then ask about them in different words. How well did it learn them?

   Show answer

   Fine-tune on a handful of examples that state a new fact, such as "Q: Who founded Northcode? A: Priya Shah, in 2019", then ask in different words: "Which person started Northcode?", "When was Northcode set up?", "Priya Shah founded which company?".

   The usual result: the model repeats the fact well when asked in almost the same words as the training examples, and much less reliably when the question is rephrased, asked in reverse (from the name to the company), or combined with another fact. It may also become more confident about related facts it was never taught, and invent them. Teaching facts reliably by fine-tuning needs many varied phrasings of each fact, and even then it is fragile and hard to update.

   That is the core reason for the advice: fine-tune for behaviour (format, tone, a narrow skill), and use retrieval for facts. A fact in a document can be changed in a second and cited; a fact in the weights can't.

Step 4 Prove it

Check you understood

Tick each check when you can do it without notes