Vice President · TheGreyMatter.ai
In brief
Choose retrieval for missing evidence and consider fine-tuning for repeatable behavior failures after establishing a baseline.
Key takeaways
- Classify failures before choosing an architecture.
- Evaluate retrieval and answer quality separately.
- Compare improvements against the same held-out cases and operating costs.
Retrieval augmented generation, usually called RAG, supplies relevant information when a model receives a request. Fine-tuning changes a model through additional training. The useful choice starts with a failure you can observe: missing evidence, inconsistent behavior, or a problem outside the model. This guide proposes a small experiment that makes those differences visible before an architecture becomes a commitment.
1. Separate knowledge from behavior
Microsoft's RAG guidance distinguishes grounding answers in private or changing information from fine-tuning for behavior, style, or task performance. That distinction is a starting point, not a rule that every knowledge problem needs a search index. A short, stable policy may fit directly into a prompt. A deterministic calculation may belong in ordinary software. Choose the simplest baseline that performs the task reliably.
Take a sample of failed outputs and assign a primary cause. Did the system lack the current source? Retrieve the wrong passage? Misread the right passage? Break the required format? Make an unsupported inference? Record the evidence for the label. If reviewers disagree, resolve the task definition first. Otherwise an expensive experiment may optimize a target nobody has actually agreed to accept.
2. Define a retrieval experiment
Consider a hypothetical support assistant that cites an expired returns policy. Supply the current policy directly and repeat the question. If the answer improves, investigate document freshness and retrieval before training. Build a small authorized collection with version dates, document owners, and stable source links. Include the expired document as a deliberate test case. The retrieval layer should help distinguish it from the current policy.
Score the retrieved evidence before scoring the prose. Check whether the needed passage appears, whether access is permitted, and whether the date matches the question. Then assess whether the answer is supported by that passage. A relevant citation is insufficient if it does not support the actual claim. Record missing evidence separately from incorrect interpretation so the next change targets the right component.
3. Define a behavior experiment
Now imagine that the assistant consistently has the correct policy but fails a stable classification task. Review the instructions, examples, and output validation before considering training. If those inexpensive changes plateau, a fine-tuning experiment may be reasonable. Assemble examples with consistent labels, explain ambiguous cases, and exclude information the organization lacks permission to use. A larger collection of contradictory examples will not create a clear target.
Keep a separate evaluation collection that the training process never sees. Group related cases so near duplicates do not cross that boundary. Compare the tuned candidate with the strongest untuned baseline using identical inputs and acceptance criteria. Include unfamiliar phrasing and cases outside the supported scope. The goal is a repeatable improvement on new work, not excellent recall of examples prepared for the experiment.
4. Compare the whole workflow
Use a shared scorecard: accepted outputs, important errors, correct escalations, reviewer minutes, response time, and total operating cost. Include index maintenance for retrieval and dataset maintenance for tuning. Both approaches can introduce work that a model benchmark omits. Compare typical and difficult cases separately; an average can conceal a workflow that is useful only when the answer was already easy to find.
RAG and fine-tuning can also be combined. A tuned model may follow a specialized output convention while retrieval supplies current evidence. Introduce that combination only when separate experiments establish what each part improves. Change one major component at a time, retain its configuration, and preserve a route back to the baseline. Otherwise an improvement becomes difficult to explain and a regression becomes difficult to repair.
5. Make the choice reversible
Write a short decision record containing the failure category, candidate configurations, evaluation cases, results, cost assumptions, and unresolved limitations. Name the person responsible for updating sources or training examples. Specify what triggers another comparison: a new document collection, a material task change, a model replacement, or a recurring error pattern. Treat the result as valid for the tested workflow rather than an enterprise-wide verdict.
Choose retrieval when better evidence demonstrably repairs the failure. Consider tuning when consistent examples improve the required behavior beyond a strong baseline. Repair the workflow when neither addresses the problem. For example, no model can settle two conflicting policies without an authorized owner deciding which governs. The architecture should make that missing decision visible instead of turning organizational ambiguity into apparently confident text.
Common questions
Does RAG eliminate hallucinations?
No. Retrieval can supply useful evidence, but it can return irrelevant or outdated material, and a model can misinterpret a correct source. Evaluate retrieval, support for claims, and appropriate abstention separately.
Should a company use both RAG and fine-tuning?
Only when testing shows a separate benefit from each. A combined system can use retrieval for current evidence and a tuned model for stable behavior, but it also adds maintenance and evaluation work.
Sources and evidence
AI assisted editorial analysis. Primary sources checked October 10, 2026. The procedures are editorial recommendations, not vendor instructions, reported customer results, or claims of personal experience. Numerical examples are hypothetical.
One useful next step: Choose one idea from this note and test it at the smallest scale that could teach you something this week.
