Fine-tuning versus retrieval for a changing knowledge base

Fine-tuning can change how a model behaves, but it is a poor database update mechanism. When policy, inventory or account state changes, the application needs a source that can be updated, authorized and cited without retraining weights.

Classify the requirement

The adaptation lab routes three synthetic needs.

from adaptation_cases import adaptation_choice_case

result = adaptation_choice_case()
assert result["retrieval_choices"] == ["current refund policy"]
assert result["needs"][1]["choice"] == "prompt_or_tune"
assert result["needs"][2]["choice"] == "tool"
assert result["changing_facts_put_in_weights"] is False

Current policy goes to retrieval, live balance to a tool and stable output behaviour to a prompt-or-tune investigation. This is a design fixture, not a model experiment.

Diagnose before adapting

Build an error set and separate missing knowledge, stale evidence, instruction-following failure, output-format failure and tool error. Improve the prompt and workflow baseline first. Use retrieval for governed documents that must be refreshed, deleted, permission-filtered or cited. Use a tool for transactional state. Consider adaptation when repeated examples show a stable behaviour or domain representation that the baseline cannot meet economically.

Evaluate the whole system. A fine-tuned model can still need retrieval, and retrieval can fail before generation. Compare paired tasks for correctness, support, abstention, latency and cost. Include future-dated policy changes, deleted records and unauthorized sources.

Document update frequency, owner, source of truth and rollback for each knowledge class. Never claim that tuning makes a checkpoint current after its data cutoff.

The Generative & Agentic AI course connects adaptation decisions to RAG, tools and evals.

Exercise

Label thirty failures by cause. Implement the smallest prompt, retrieval or tool fix for each class, then justify any remaining tuning proposal with a paired baseline.

Continue learning

This article is part of the Model adaptation and multimodal tasks sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Continue with Prepare instruction examples without duplicating evaluation data.
• Then apply it in LoRA explained through trainable parameter scope.

Reference: NIST AI RMF Generative AI Profile.