"Should we fine-tune?" This question comes up in every enterprise AI engagement. The answer is almost always "not yet" but rarely "never." The decision depends on specific, measurable criteria, not intuition. Here's the framework we use.
What You Need to Know
- Prompting (including RAG) should be the default approach. Fine-tuning is an optimisation, not a starting point
- Fine-tuning wins when: you need consistent output format, domain-specific language patterns, or latency reduction
- Prompting wins when: the task is flexible, the domain knowledge changes frequently, or you need model-agnostic portability
- The decision should be based on empirical comparison, not theoretical arguments
The Decision Framework
Start With Prompting
Always. Prompting with a well-designed RAG pipeline can handle 80-90% of enterprise use cases without fine-tuning. It's faster to implement, cheaper to maintain, and easier to update when your domain knowledge changes.
Prompting is model-agnostic. You can switch from GPT-4 to Claude to an open-source model without retraining. Fine-tuning locks you to a specific model and version.
Consider Fine-Tuning When:
1. Output format consistency matters.
If your use case requires structured, consistent output (always JSON, always a specific schema, always a particular classification taxonomy), fine-tuning can enforce format compliance more reliably than prompting.
Prompting can achieve format compliance, but it degrades under edge cases. Fine-tuned models maintain format consistency even on unusual inputs because the format is learned, not instructed.
2. Domain-specific language patterns are critical.
If your domain uses specialised terminology, abbreviations, or conventions that the base model handles poorly, fine-tuning on domain data improves performance measurably.
Example: legal document analysis where specific clause structures, Latin terms, and jurisdictional conventions are standard. A fine-tuned model recognises these patterns natively rather than relying on prompt instructions.
3. Latency matters.
Fine-tuned models can be smaller and faster than their base counterparts because the task-specific knowledge is baked in. If your use case requires low-latency responses (sub-second for user-facing applications), fine-tuning a smaller model often outperforms prompting a larger one.
4. Cost at scale matters.
Long prompts with detailed instructions and RAG context are expensive at high volume. Fine-tuning reduces prompt length because the model already knows the task. At thousands of queries per day, the per-query cost difference adds up.
The mathematical case for fine-tuning is strongest when the loss function you're optimising is well-defined and stable. If your task has a clear correct answer and that answer doesn't change frequently, fine-tuning can converge on a solution that prompting can only approximate. If the task is ambiguous or the correct answer shifts with new information, prompting's flexibility is the better choice.
Dr Vincent Russell
Machine Learning (AI) Engineer
Don't Fine-Tune When:
The domain knowledge changes frequently. Fine-tuning bakes knowledge into model weights. If your knowledge base updates weekly, you'd need to retrain weekly. RAG retrieves current knowledge at query time.
You need model portability. Fine-tuned models are version-locked. When GPT-5 launches, your fine-tuned GPT-4 doesn't benefit. With prompting, you upgrade the model and keep your prompts.
The task requires reasoning over new information. Fine-tuning improves pattern matching on known patterns. It doesn't improve reasoning on novel inputs. For tasks that require genuine inference over unfamiliar content, a larger base model with good prompting outperforms a fine-tuned smaller model.
You don't have enough training data. Fine-tuning with fewer than 500 high-quality examples risks overfitting. If your domain doesn't have this volume of labelled data, prompting is safer.
The Empirical Test
Before committing to fine-tuning, run a head-to-head comparison:
- Build the best possible prompting solution (optimised prompts, RAG, few-shot examples)
- Fine-tune a model on your domain data
- Evaluate both on a held-out test set using the same metrics
- Compare performance, latency, and cost
If the fine-tuned model doesn't improve on the prompting solution by at least 5% on your primary metric, the maintenance overhead of fine-tuning isn't justified.
The Hybrid Approach
The best enterprise deployments often use both. RAG retrieves current knowledge. Fine-tuning handles output format, domain language, and latency optimisation. The fine-tuned model is the engine. RAG is the fuel.
This hybrid approach gives you the consistency of fine-tuning with the flexibility of retrieval. It's more complex to maintain, but for high-volume, mission-critical enterprise use cases, the performance justifies the complexity.
The fine-tuning decision should be empirical, not philosophical. Start with prompting. Measure. If prompting hits a ceiling on a specific metric that matters to your use case, try fine-tuning and compare. Let the data decide.