Retrieval Augmented Generation Evaluation Framework

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

Build Log. I'm Nick.

Here's what I shipped this week and what it taught me.

The Hook

Last Tuesday at 2:47 AM, one of my WordPress sites went down. Not the main revenue driver, but a content hub that feeds three other properties. I got the alert, rolled over in bed, and did something I couldn't have done six months ago.

I sent a text message to my phone. Not to a person. To an AI agent that lives in my infrastructure.

“Fix the database connection on site seven. Check the usual suspects first.”

Twenty-three minutes later, another text: “Database connection restored. Issue was max_connections limit. Temporarily raised to 200. Recommend permanent increase to 150. Site responding normally.”

I went back to sleep. The agent had diagnosed the problem, applied the fix, and gave me a recommendation for preventing it next time. No SSH sessions. No emergency calls. No lost revenue.

Today I'm walking you through exactly how I built this system and three others that handle my operations while I sleep.

Why Autonomous Agents Matter Now

[BED: DUCK]

You've probably heard the phrase “autonomous agents” thrown around like it's some distant sci-fi concept. Here's what actually happens when you run them in production.

I manage thirteen WordPress sites, five KDP publishing pipelines, and three 3D model marketplaces. That's roughly forty-seven different things that can break on any given day. For two years, I was the single point of failure for all of them.

Then I started building agents. Not the chatbot kind. The kind that actually DO things.

[BED: SWELL]

An autonomous agent is software that can perceive its environment, make decisions, and take actions without human intervention. But that's the textbook definition. The real definition is simpler: it's code that can think through a problem and fix it while you're doing something else.

The difference between automation and autonomy is decision-making. A cron job is automation. It runs the same script every hour. An agent is autonomous. It looks at the current situation and decides what to do next.

Here's why this matters right now. The cost of running language models has dropped ninety percent since last year. Claude Haiku costs four dollars per million tokens. That means an agent that handles my site monitoring runs for about six dollars a month.

Compare that to hiring a virtual assistant at eight dollars an hour. The agent works twenty-four seven. It never gets sick. It doesn't take vacation days. And it's getting smarter every quarter.

The Database Guardian Agent

Let me show you exactly how the database agent works. Because this isn't theory anymore. This is production code managing real revenue.

The agent runs on a fifteen-dollar DigitalOcean droplet. It checks all thirteen sites every ninety seconds. Not just “is the site up” checks. Deep health monitoring. Database connection counts, memory usage, disk space, response times, even SSL certificate expiration dates.

When it detects an anomaly, it doesn't just send an alert. It starts a decision tree.

For database issues, it first checks connection counts against the configured limit. If we're hitting max connections, it looks at the query log for long-running queries. If it finds any running longer than sixty seconds, it kills them and logs which plugin or theme triggered them.

If that doesn't resolve it, the agent temporarily increases the connection limit by twenty-five percent and schedules a review for the next morning. It documents everything in a shared Notion database with timestamps and reasoning.

But here's the operator detail that makes this actually work: the agent has limited permissions. It can restart services and adjust connection limits, but it cannot delete files or modify core WordPress configurations. I learned this the hard way.

In October, I gave an early version too much access. It decided a plugin was causing memory issues and deactivated it. The plugin was WooCommerce. On a site doing twelve hundred dollars a day in sales.

Three hours of downtime taught me that agents need guardrails, not superpowers.

Now each agent has a defined scope and escalation path. The database agent can fix connection issues and restart MySQL. For everything else, it creates a detailed report and wakes me up.

The Content Pipeline Agent

[BED: DUCK]

The second agent manages my KDP publishing pipeline. This one's more complex because it deals with creative decisions, not just technical maintenance.

Every Monday morning, the content agent reviews performance data from all my published books. It pulls sales numbers, review sentiment, and keyword ranking data. Then it makes decisions about what to publish next.

The process starts with trend analysis. The agent scrapes Amazon's bestseller lists in my niches, identifies rising topics, and cross-references them against my content calendar. If it finds a gap, it generates a book outline and adds it to my review queue.

But here's where it gets interesting from an operations standpoint. The agent doesn't just generate content. It manages the entire production pipeline.

When I approve an outline, the agent coordinates three different AI models. Claude Opus handles the heavy reasoning and chapter structure. GPT-4 writes the actual content. A local Llama model does fact-checking and consistency reviews.

The agent monitors token usage across all three models and switches providers if costs spike. Last month, OpenAI had API issues for six hours. The agent automatically routed all content generation to Claude until service restored. I didn't even know there was a problem until I checked the logs.

The economics are remarkable. Each book costs roughly thirty-seven dollars in AI usage to produce. My average KDP book generates four hundred and twenty dollars in the first ninety days. The agent has published eleven books since September with zero manual intervention after outline approval.

But the real value isn't the books themselves. It's the data. The agent tracks which topics perform best, which writing styles generate more engagement, and which keywords drive organic discovery. That intelligence feeds back into better content decisions.

What Most People Get Wrong About Agents

You've probably seen demos where someone asks ChatGPT to “be my virtual assistant” and pretends that's an autonomous agent. That's not even close to what we're talking about here.

Real agents have three components that those demos completely miss. Perception, memory, and action capability.

Perception means the agent can read its environment. Not just respond to prompts. My database agent knows the current state of all thirteen sites at any moment. It tracks trends over time. It can tell the difference between a temporary spike and a growing problem.

Memory means persistent context across interactions. When my content agent suggests a book topic, it remembers why it made that recommendation three weeks later. It builds on previous decisions instead of starting fresh every time.

Action capability means the agent can change things in the real world. It can restart services, create files, send emails, make API calls, and update databases. Most “agents” people build can only generate text. That's not agency. That's autocomplete with personality.

The biggest mistake I see operators making is starting too big. They want to build the agent that handles everything. Customer service, content creation, technical support, and business strategy all in one system.

That's backwards. Start with one boring, repetitive task that you do every week. Build an agent that does only that task. Run it for a month. Fix the edge cases. Then expand its scope.

My third agent handles 3D model marketplace uploads. It takes finished models from my design pipeline, optimizes file sizes, generates product descriptions, sets pricing based on comparable listings, and publishes to three different platforms.

This agent processes fifteen to twenty models per week and generates about eight hundred dollars in monthly revenue. The time savings alone are worth it. But the consistency is what actually drives sales. The agent never forgets to add proper tags. It never uploads the wrong file format. It never skips the marketing copy.

Humans make errors when they're bored or tired. Agents make errors when you haven't defined the process clearly enough.

The Technical Architecture That Actually Works

[BED: DUCK]

Let me give you the operator details on how this infrastructure actually runs. Because the architecture choices matter when you're betting real revenue on these systems.

All three agents run on the same DigitalOcean droplet. Eight CPUs, sixteen gigs of RAM, two hundred gigs of SSD storage. Total cost: sixty-four dollars per month. I could optimize this further, but reliability beats cost optimization when your business depends on it.

Each agent runs in its own Docker container. This isn't about being trendy with containerization. It's about isolation. When the content agent burns through RAM processing a large book outline, it doesn't affect the database monitoring agent.

The agents communicate through Redis. Not direct API calls between containers. This creates a natural queue system and prevents cascading failures. If one agent goes down, the others keep running.

For persistence, everything writes to PostgreSQL. Agent decisions, error logs, performance metrics, and state information. I can reconstruct exactly what any agent was thinking at any point in time.

The notification system uses Telegram because the API is simple and reliable. Each agent has its own bot token. Critical alerts go to my main chat. Routine updates go to separate channels I check when convenient.

And this is crucial: every agent has a kill switch. A single command that stops all autonomous actions and puts the system in monitoring-only mode. I've used this twice in six months when I needed to debug unusual behavior.

The model routing happens through LiteLLM, which gives me a unified interface to Claude, GPT-4, and local models. The agents automatically retry failed calls with exponential backoff. They track costs per operation and switch models if spending exceeds preset limits.

The Economics That Make This Worth It

[BED: SWELL]

Let me break down the real numbers, because this only makes sense if it actually saves money or generates revenue.

The database agent saves me approximately four hours per week. That's time I was spending on manual monitoring, troubleshooting connection issues, and optimizing slow queries. At my consulting rate, that's six hundred and forty dollars worth of time per week.

The content agent manages eleven active KDP books generating about two thousand dollars per month in passive revenue. Without the agent, I could maybe maintain three or four books while keeping up with my other projects.

The 3D model agent processes uploads I simply wouldn't have time to do manually. That eight hundred dollars monthly revenue stream wouldn't exist without automation.

Total monthly cost to run all three agents: about one hundred and thirty dollars. Model usage fees, server costs, and monitoring tools combined. The return on investment is roughly fifteen to one.

But here's the less obvious value: these agents don't get sick. They don't take vacations. They don't have bad days where they forget important steps.

The database agent has caught and fixed fourteen issues in the last ninety days that could have caused revenue loss. Three were during overnight hours when I was asleep. Two were during a family vacation when I was deliberately offline.

The peace of mind alone is worth the monthly cost. I can travel without constantly checking my phone for alerts.

What I'm Building Next

The next agent I'm building handles customer support for my 3D model marketplaces. This one's trickier because it involves direct customer interaction.

The agent will handle the ninety percent of support tickets that are routine. Download issues, license questions, file format requests. For anything complex, it escalates to me with a full context summary.

I'm also experimenting with an agent that manages my affiliate marketing campaigns. It tracks performance across different traffic sources, pauses underperforming ads, and allocates budget to winning campaigns.

The goal isn't to automate everything. It's to automate the repetitive tasks so I can focus on the creative and strategic work that actually moves the business forward.

Every hour I save on maintenance and routine operations is an hour I can spend building new revenue streams or improving existing ones.

Your Next Steps

[BED: DUCK]

If you want to build your first autonomous agent, start simple. Pick one task you do every week that follows a predictable process.

Content moderation, data backups, social media posting, inventory updates, or performance reporting. Something with clear inputs, defined logic, and measurable outputs.

Build the agent using Claude or GPT-4 with function calling capabilities. Start with read-only permissions. Let it analyze and recommend before it takes action.

Run it manually for two weeks while you watch every decision. Fix the edge cases. Expand the permissions gradually as you build confidence in its judgment.

The initial time investment is significant. Plan for forty to sixty hours to build and test your first agent properly. But once it's running smoothly, you'll recoup that time investment within a month.

The operators who figure this out first will have a massive advantage over the next two years.

For the newsletter crowd — Build Log Weekly hits your inbox every Monday with what I shipped, what I learned, and what I'm testing next.

If you want to see the actual code and architecture diagrams for these agents, check out the Build Log newsletter — link in the show notes.

Outro

That's the build log for this week.

Ship something. Measure it. Tell me what happened.