Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.
Build Log. I'm Nick.
Here's what I shipped this week and what it taught me.
If you're measuring your RAG system's accuracy by just asking ‘does this feel right?', you're flying blind. I deployed a system that looked perfect in testing. It aced all my example questions. Then it went live. In the first week, it hallucinated answers on 17% of real user queries. Real people got confidently wrong information. Today, we're fixing that.
Stop guessing and start measuring.
You’ve probably heard that you need a ‘robust evaluation framework.'
Here's what actually happens when you run it.
This is the single biggest blocker between a demo-grade chatbot and something you can actually deploy to users without holding your breath. It's the difference between art and science. And it’s not academic. A wrong answer can cost you a customer, damage your reputation, or just waste hours of support time cleaning up the mess.
The Three Numbers That Don't Lie
My wake-up call was a query from a user named Sarah. She asked my support bot, “What's the process for upgrading my plan from Basic to Pro?” The bot retrieved the correct help doc. I saw it in the logs. Then it told her, “To upgrade, simply log into your account and click the ‘Upgrade' button on the billing page. The change is instant.” Sounds great, right? Except our upgrade process requires manual approval for fraud checks. It takes up to four hours. There was no ‘Upgrade' button. The model just made it up. It saw the context about upgrading and invented a perfect, logical, completely fictional process. That’s a faithfulness failure.
So we built a triage dashboard. It runs on three metrics.
First, Faithfulness. Does the answer stick to the retrieved context? It's your hallucination score. You're measuring how often the model goes rogue. In my case, that 17% failure rate.
Second, Answer Relevance. Does the output actually answer the question? A user asks, “What is our refund policy?” and the bot says, “Our mission is to deliver exceptional customer service.” That's a zero. It's not wrong, but it's useless.
Third, Context Relevance. Are the documents you fetched actually tight and useful? For a simple question like “What's the company's founding year?”, retrieving a 50-page annual report is a failure. You want the one-sentence bio. This measures your retrieval quality.
And this is where it gets interesting from an operations standpoint.
You don't need a team of humans to score this. We use Anthropic's Claude Haiku as a judge. For each question-answer-context set, we send it a scoring rubric. The prompt says, “On a scale of 1 to 5, how faithful is this answer to the provided context?” It costs us about seven cents per evaluation. We run this on a hundred queries a week. That's seven dollars. Seven dollars to catch a critical regression before my users do is the best insurance policy I've ever bought.
Faithfulness, Answer Relevance, Context Relevance. This is your new production triage dashboard. Ignore them at your peril.
[BED: SWELL, THEN FADE]
The Automation Pipeline
You might be thinking, “Nick, that sounds like a lot of manual work. Pulling queries, running them, scoring them.” You're right. If it's manual, you won't do it. Or you'll do it once and forget.
So we automated it. Completely.
The architecture is simple. We have a Python script that runs on a weekly cron job. Every Monday at 3 AM, it fires up. It goes into our RAG system's logs and randomly samples the last hundred real user queries. It re-runs each query through the exact same pipeline our users hit. It captures the question, the retrieved context chunks, and the final answer.
Then, it feeds each trio into the LLM judge—our Haiku instance—with that strict scoring rubric. The judge returns a score for each metric. Those scores get written to a Google Sheet. Seriously. It doesn't need to be fancy. A simple spreadsheet with a timestamp, the query, and the three scores. Over time, you see the trend lines.
This automation saved me four hours every single week. Four hours I used to spend manually spot-checking. More importantly, it caught a regression two weeks ago. Our average faithfulness score dropped from 4.8 to 3.2. I got an alert. We dug in and found a bug in our document chunking logic. It was splitting sentences in the middle of key facts. Users would have noticed that within days. We fixed it before a single complaint came in.
Automate this or you won't do it. That's the pipeline.
[BED: SWELL]
[MID-ROLL CTA]
You don't need to build that scoring rubric from scratch. I've made ours available. It's the exact system prompt we use for the Haiku judge. It defines each score from 1 to 5 for all three metrics. It'll save you hours of tuning. Head to buildlogpodcast.com/metrics to download it. No email required. Just grab it.
[BED: DUCK]
The Cost of Being Right
Now, here's the contrarian take. Everyone obsesses over accuracy. But what if your perfect, 99% accurate system takes ten seconds to respond and costs a dollar per query? It's useless. It's a lab experiment.
You've probably heard that more parameters mean better answers.
Here's what actually happens when you run it.
You must track latency and cost-per-query. They are not secondary concerns. They are primary. Users abandon slow bots. Full stop. And a system that isn't cost-effective will eat your margins alive.
We have a hard limit: 3 seconds for end-to-end response time. And a cost target of under twelve cents per query. To hit that, we can't use the biggest, most powerful model for everything. We use a mix. Claude Sonnet for generating the final answer—it's the right blend of capability and cost for us. But we use the cheaper, faster Haiku for the retrieval scoring and for initial query classification. It’s all about picking the right tool for the job.
The real metric you care about is the ratio of accuracy-to-cost. Is a system that's 95% accurate but costs a dollar per query better than one that's 90% accurate and costs a dime? For most real businesses, the answer is no. That 5% point jump isn't worth 900% more cost.
If you're not measuring latency and cost, you're only seeing half the picture. You're building a Ferrari when your users need a reliable delivery truck.
[BED: SWELL, THEN FADE TO BACKGROUND]
Your One Thing Today
So here is your action item. Don't try to boil the ocean.
Pick one metric. Just one. I'd start with Faithfulness. It's the most dangerous failure mode.
This afternoon, pull the last fifty questions from your chat log. If you're not in production yet, use your test set. Now, for each one, do the manual work. Look at the answer. Look at the source documents it retrieved. Ask yourself: did the model make anything up? Score it a 1 for a major hallucination, a 5 for perfectly faithful. Be ruthless.
That number you get—that percentage of faithful answers—is your baseline. It's your truth. It might be 50%. It might be 90%. It doesn't matter. What matters is that you now have a real number to improve. Everything, every change you make to your prompts or your retrieval, starts from that number.
Ship it, measure it, and iterate.
If you want the reflective version — why I made certain decisions, what I'd do differently — Signal Notes is where that lives.
For a deep dive on slicing your LLM costs without sacrificing performance, check out our sister show, ‘AI in Prod.' We just released an episode breaking down our multi-model architecture.
That's the build log for this week.
Ship something. Measure it. Tell me what happened.
