Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.
Hook
Build Log. I'm Nick.
If you're using a fine-tuned model to answer questions from your company docs, you're probably wasting a staggering amount of money and getting worse results. I tested both approaches in production for three months, and the winner wasn't even close.
It's the classic build-versus-buy decision, but for your AI's brain: RAG versus fine-tuning. And this isn't just an academic debate. It's a weekly invoice and an accuracy report that determines whether your AI project gets budget next quarter or gets killed.
Context
Here's why this matters right now in Q3 2024. GPU costs are plummeting, but fine-tuning API costs haven't budged proportionally. OpenAI still charges the same premium for custom models they did six months ago.
But there's a bigger problem. Knowledge cutoffs.
[BED: DUCK]
Last month, I was running a fine-tuned model on our product documentation from 2023. Beautiful responses. Perfectly formatted. Completely wrong when someone asked about our Q2 product update. The model had no idea we'd shipped new features because it was trained on old data.
RAG solves this inherently. New document hits the vector store, and thirty seconds later, your AI knows about it. No retraining. No additional costs. No deployment cycle.
[BED: SWELL]
Every engineering lead I talk to is facing this exact decision right now. Do we fine-tune for our specific use case, or do we build a retrieval system? The conventional wisdom says fine-tuning for customization, RAG for fresh data.
Here's what actually happens when you run both in production with real users asking real questions.
The Brutal Economics of Fine-Tuning
Let me break down the real costs, because the fine-tuning sales pitch always focuses on that initial training fee. Fifty to a hundred dollars, depending on your dataset size. Seems reasonable.
But that's not where they get you.
A fine-tuned model is always “on.” Every single API call burns tokens through your custom weights, even for simple questions. “What's our vacation policy?” costs the same as “Explain our entire technical architecture in detail.”
Here's the exact numbers from my HR documentation system. I fine-tuned a GPT-3.5 model on a fifty-page employee handbook. Beautiful results. Perfect tone. Cost per query: twelve cents.
Twelve cents doesn't sound like much until you multiply it by three hundred queries per week.
So I built the RAG version. Same documents, same quality responses. I'm using text-embedding-3-small for the embeddings, storing them in a local Chroma instance, and hitting GPT-4o-mini for the final generation.
Cost per query: four cents.
That's a 3x savings, shipped and running for two months. The kicker? The RAG version gives better answers because it can cite exactly which page and section it pulled from. Fine-tuning gave me confident-sounding nonsense when it didn't know something.
[BED: DUCK]
And this is where it gets interesting from an operations standpoint. Fine-tuning has a hidden tax on every single API call you make. You're not just paying for compute. You're paying for the privilege of using your own data.
With RAG, cost scales linearly with actual usage. Embedding storage is essentially free. Vector searches are microseconds. You only pay the big inference cost when someone actually needs an answer.
The Practical RAG Stack That Just Works
Forget the theory. Here's the pipeline I deployed ninety days ago and haven't touched since.
Document comes in through a webhook. Could be a Slack upload, could be someone dropping a PDF in our shared folder. The webhook fires, catches the file, and dumps it into an S3 bucket.
Lambda function picks it up within seconds. Chunks the document using LangChain's recursive character splitter. I learned this the hard way – chunk size matters more than anyone tells you. Too small, you lose context. Too big, you hit token limits.
Sweet spot for our docs: 1000 characters with 200-character overlap.
Each chunk gets embedded using OpenAI's text-embedding-3-small model. Costs about two cents per document, regardless of size.
Embeddings go into the vector store. I use Chroma because it runs locally and I don't need another service to manage. The whole pipeline, from upload to searchable, takes under sixty seconds.
When someone asks a question, here's what happens. Claude Haiku does initial classification to route the query. Takes fifty milliseconds, costs almost nothing. Then I retrieve the top five relevant chunks from the vector store.
Those chunks become context for Claude Opus, which writes the final response. Haiku for speed, Opus for reasoning. It's the cheapest way to get high-quality answers without burning budget on overkill.
[BED: SWELL]
The magic happens in the prompt design. I tell Opus exactly how to use the retrieved context, how to admit when it doesn't know something, and how to cite sources. Simple instructions, consistent results.
Total infrastructure cost: about fifteen dollars per month for a system that handles 200 queries per week across thirteen different document sets. Compare that to the fine-tuning approach, which was hitting 180 dollars monthly just for the HR docs.
Mid-Roll CTA
[BED: DUCK]
Want the exact LangChain script we use to chunk documents and avoid context fragmentation? It's the one thing that made our RAG reliability jump from 80% to 99%.
I'm putting together a simple cheat sheet with the chunking strategy, the prompt templates, and the error handling that actually works in production.
Get the exact script that fixed our biggest headache. Link's in the show notes.
[BED: SWELL]
When Fine-Tuning Actually Wins
Everyone in the space will tell you RAG is always the answer. They're mostly right, but here's where they're wrong.
RAG is for knowledge. Fine-tuning is for personality.
I learned this when we tried to automate customer support responses. RAG could pull the right product information, cite the correct policies, find relevant troubleshooting steps. But it sounded like a robot reading from a manual.
So I fine-tuned a smaller model on every customer support ticket we've ever sent. Not for the knowledge – for the style. How our best support agent phrases apologies. The specific way we escalate technical issues. Our brand voice when someone's frustrated.
The fine-tuned model learned empathy patterns I couldn't encode in a prompt.
This is expensive. Inference costs 40% more than base models. But it's worth it for customer-facing interactions where tone matters as much as accuracy.
The hybrid approach works best. RAG retrieves the facts, fine-tuned model shapes the response. Knowledge plus personality.
I'm running this in production now for our highest-tier support tickets. The fine-tuned model doesn't know product information – that would go stale. It knows how to sound human when delivering RAG-retrieved answers.
[BED: DUCK]
This is high-cost, high-value territory. You're not fine-tuning for information storage. You're fine-tuning for behavior that can't be prompted.
For everything else? RAG wins on cost, freshness, and explainability.
Your Action for Today
Don't overthink this. Your action for today: pick one internal document and run a single test.
Grab a wiki page, a PDF manual, anything with actual information your team uses. Use the OpenAI Playground or follow a basic LlamaIndex tutorial. Feed the document as context and ask it five real questions.
See the magic of RAG for yourself in under ten minutes.
You'll immediately understand why this approach beats fine-tuning for 90% of use cases.
Start simple. One document, five questions, ten minutes. Then you can decide if you want to build the full pipeline.
But I guarantee you'll be impressed by how well it works with zero training and zero custom infrastructure.
The hardest part about RAG isn't the technology. It's admitting that the simple approach might actually be better than the complex one.
Cross-Promo
Signal Notes this week goes deeper into the architecture behind what I just described. Same voice, more technical depth.
If you're thinking about the long-term architecture of this, we break down the entire system diagram on our sister show, Production Ready.
Outro
That's the build log for this week.
Ship something. Measure it. Tell me what happened.
