How To Build Rag Pipeline From Scratch

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

HOOK

Build Log. I'm Nick.

Every AI tutorial will tell you to use a vector database for your RAG pipeline. They're mostly wrong. I've deployed and tested a pipeline from scratch in production, and today I'm showing you the one critical component you're probably missing that cuts hallucinations by more than half.

CONTEXT

Here's the thing about RAG that nobody wants to admit. It's not just some cool demo feature anymore. It's becoming the default way to ground LLMs in actual truth instead of letting them make things up and cost you real money.

Last month, one of my content sites was using a basic GPT-4 setup to answer reader questions. No retrieval. Just raw model knowledge. The thing was confidently telling people that a WordPress plugin I knew had been discontinued in 2021 was still actively maintained. That's not a harmless mistake when you're running a business.

So I built a RAG system. But here's what I learned after processing twelve thousand real user queries: everyone focuses on the “Retrieval” part — the sexy vector embeddings and similarity search. The real fight happens in the “Augmentation” step. And that's where most pipelines completely fall apart.

This is about moving from demo-grade answers to production-grade reliability.

[BED: DUCK]

THE ARCHITECTURE YOU ACTUALLY NEED

Let me break down what a working RAG pipeline actually looks like. Not the theoretical version from blog posts. The one I'm running right now across thirteen sites.

You've got six components. The Loader grabs your documents. The Chunker breaks them into digestible pieces. The Embedder turns those pieces into mathematical vectors. The Retriever finds the best matches for your query. The Router — and this is the part everyone skips — classifies what you retrieved. Then the Generator creates your final answer.

Most tutorials spend ninety percent of their time on components one through four. Here's the contrarian take: the vector database is just the bookstore. The real art is in choosing which books to read and how to summarize them.

I spent three weeks optimizing my embedding model. Tried OpenAI's new text-embedding-3-large, experimented with Cohere's latest release. Got maybe a five percent improvement in retrieval quality. Then I added proper classification after retrieval. Fifty-three percent reduction in hallucinations, measured across a test set of four hundred queries that had been manually verified.

The vector part is solved technology. Pinecone, Weaviate, even basic FAISS if you want to run it yourself — they all work fine. Where your pipeline lives or dies is what happens next.

[BED: SWELL]

BUILDING THE AUGMENTATION ENGINE

This is where it gets interesting from an operations standpoint. Before you send retrieved chunks to your expensive LLM, you need a bouncer. Something fast and cheap that asks: is this chunk actually relevant? Does it contradict the other chunks? Is it recent enough to trust?

I use Claude Haiku for this classification step. Costs me seven cents per hundred classifications. The prompt is dead simple: “Rate this text chunk's relevance to the query on a scale of zero to ten. If it's below a seven, explain why it should be filtered out.”

Here's a real example from last week. User query: “How do I update WordPress plugins safely?” My retrieval pulled five chunks. Two were about plugin updates from my current documentation. One was about theme updates — related but not the same thing. One was from 2019 talking about FTP uploads. And one was completely off-topic about plugin development.

Without classification, all five chunks would have been dumped into GPT-4's context window. The model would have tried to synthesize everything, probably mentioning FTP uploads as if they're still relevant, maybe confusing plugins and themes. Instead, Haiku scored them. The two current plugin update chunks got nines. Theme update chunk got a six with a note about topic drift. The old FTP chunk got a three. Off-topic chunk got a zero.

Only the nines went to the final generation step. Clean context, focused answer, no hallucinated nonsense about outdated methods.

Don't ask your expensive LLM to reason with garbage. Build a bouncer for your context. This one change took my pipeline from something I was nervous to deploy to something I trust with live traffic.

[BED: DUCK]

The operational savings are real too. Before classification, I was sending an average of eight hundred tokens of retrieved context per query. After filtering, it's down to three hundred and twenty. That's a sixty percent reduction in Opus costs, plus faster response times because there's less to process.

THE TEST IN PROD MISTAKE

Now here's where most people completely blow it. They test their RAG pipeline on softball questions. “What's the capital of France?” works perfectly. “How do I install WordPress?” gets a clean answer. They ship it and think they're done.

Two weeks later, a user asks something like: “I'm getting a white screen after updating to WordPress 6.4, but only on mobile Safari, and only when I'm logged in as an admin. Could this be related to the plugin conflict I had last month with WooCommerce?”

Your beautiful RAG system retrieves chunks about white screens, chunks about Safari compatibility, chunks about admin areas, chunks about WooCommerce conflicts. None of them together. Your LLM tries to synthesize this partial information and generates confident-sounding nonsense.

Here's what I did instead. I ran a shadow pipeline for ten days. Every query that hit my site's search got processed by both the old system — which was just showing search results and letting humans figure it out — and my new RAG system. I didn't show users the RAG answers yet. I just collected them.

Then I manually reviewed two hundred of the weirder queries. The edge cases. The ones where users typed full sentences with multiple problems. That's where I found the real failure modes.

Like when someone asked about “fixing broken images in WordPress” and my system confidently recommended a plugin that had been compromised and removed from the directory six months ago. The plugin chunk was in my knowledge base because I'd written about it before the compromise. The security warning was in a different chunk that didn't get retrieved because it used different terminology.

If you aren't testing on the edge cases, you're not testing your pipeline. You're just testing your demo.

[MID-ROLL CTA]

Quick side note — if you want the exact architecture diagram I just walked through, I've put together a free blueprint that maps this whole system out. Head to buildlog.net slash RAG blueprint. Drop your email, get the PDF, and I'll send you the actual prompts I'm using for that classification step. One email a week with what I'm testing in production.

[BED: SWELL]

MONITORING WHAT MATTERS

Here's the part that separates hobby projects from production systems: you need to watch this thing like a hawk for the first month. Not just server metrics. Content metrics.

I track four numbers daily. Classification rejection rate — what percentage of retrieved chunks are getting filtered out. If this spikes above forty percent, my retrieval is pulling too much noise. Context token usage per query — this directly impacts my costs and response time. Answer confidence scores from my final LLM call — I ask GPT-4 to rate its own confidence and log anything below eight. And user feedback when they click “was this helpful.”

The confidence scores caught something interesting. Queries about very recent WordPress updates were getting low confidence ratings because my knowledge base was three weeks old. I hadn't thought about content freshness in my initial design. Added a weekly scraping job to pull the latest WordPress news and changelogs. Confidence scores on recent topics jumped from six to nine.

This isn't set-and-forget technology. It's more like having a really smart intern who needs periodic check-ins to make sure they're not going off the rails.

CTA

Your one task today — and I mean this — is not to spin up a vector database. It's to write down the five weirdest, hardest questions your users might ask your system. The multi-part ones. The edge cases that would make a human support agent pause and think.

That's your real test set. Build for those, not for “What is WordPress?”

Grab that architecture blueprint at buildlog.net slash RAG blueprint if you want to see how to build a system robust enough to handle the weird questions. Because that's where your actual users live — in the weird questions.

Now go build something that actually works.

Got a workflow you want me to break down? Send it in. Real problems from real operators — that's what this show is for.

If you're thinking about the business behind AI features like this — the unit economics, the pricing strategy, how to justify the infrastructure costs to your boss — check out our sister show Production Profits. For the business behind the code.

OUTRO

That's the build log for this week.

Ship something. Measure it. Tell me what happened.