Ai Agent Frameworks Eval 2024

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

Hook

Stop.

If you're building AI agents in 2024 with just a simple `llm.call()` function wrapped in a while loop, you're building on a house of cards.

I've been running production agent systems for 13 sites over the last eight months. I've seen the failures firsthand. The infinite loops that burn through API credits. The hallucinations that compound across steps. The total system collapse when one tool returns unexpected output.

Everyone's talking about agents. Everyone's hyping the demos. But almost no one is talking about the frameworks that make them actually reliable enough to run in production.

Today, we're changing that.

This is Signal Notes. I'm Nick. Let's ship.

Context — Why "Prompt and Pray" Fails in Production

Let me paint the picture of where we are right now.

In 2023, the AI agent space was a carnival of cool demos. Remember the AutoGPT explosion? Everyone was running agents that would browse the web, write code, and supposedly build entire apps. They worked great for about three minutes. Then they'd spiral into nonsense loops, hallucinate fake API responses, or just stop responding entirely.

Here's what happened in those demos: a single prompt wrapped in a while loop. The agent would call the LLM, get a response, parse it, call the LLM again, get another response, and so on until it either succeeded or—more commonly—hit a token limit or infinite loop.

No persistent state. No error recovery. No observability into what the agent was actually thinking.

I call this the "prompt and pray" pattern. You prompt the model, you pray it does the right thing, and you have absolutely no mechanism to recover when it doesn't.

That worked in 2023 because the stakes were low. It was hobby projects and weekend experiments.

But 2024 is different.

We've moved from the era of prototypes to the era of production.

The foundational tech is finally here. OpenAI's GPT-4o, Anthropic's Claude 3.5 Sonnet, Google's Gemini 1.5 Pro—these models are powerful enough to handle complex multi-step tasks. They're cheap enough to run at scale. A complex agentic workflow with multiple LLM calls costs pennies, not dollars.

But here's the problem: the methodology is lagging behind the technology.

We have rocketship engines running on bicycle brakes. The models can do incredible things, but the way we're orchestrating them hasn't matured.

This matters right now because companies are starting to bet real money on these systems. I've talked to founders who are deploying agents for customer support, for data analysis, for automated content pipelines. These aren't experiments—they're revenue-critical systems.

And when a revenue-critical system fails because you used a while loop instead of a proper framework, that's not a technical problem. That's a business problem.

So let's talk about what a production-ready agent framework actually needs.

Point 1 — The Three Pillars of a Production-Ready Framework

After running agents in production for over 150 episodes worth of automated content, managing 13 sites, and deploying pipelines that handle real revenue, I've identified three non-negotiable pillars for any agent framework.

If your framework doesn't have these three things, it's not production-ready. Period.

**Pillar One: State and Memory.**

This is not just conversation history. A lot of people think state management means storing the last few messages in a list. That's not enough.

Production state is a structured database of steps, results, and conclusions that the agent can query and update.

Think about it this way: if your agent runs a SQL query, gets results, processes them, and then needs to reference those results three steps later, where are they? In a naive implementation, they're in the context window—which means they're competing for space with every other piece of information.

A proper framework stores each step's output in a structured key-value store. The agent can query it by step number, by result type, by timestamp.

This is what allows an agent to pick up a task days later. I have agents that run weekly reporting pipelines. They save their state after each step. If the Gmail API is down on Tuesday, the agent pauses, saves its progress, and resumes on Wednesday without losing a single piece of context.

**Pillar Two: Orchestration and Tools.**

An agent is only as useful as the tools it can wield. But managing tool calls is surprisingly complex.

Your agent needs to decide which tool to call, parse the function parameters from the LLM's response, execute the tool, handle the output, and then feed that back into the reasoning loop.

A good framework handles this cleanly. It defines tools as typed functions with schemas. The LLM gets the schema, generates a function call, and the framework validates the parameters before execution.

And crucially, the framework manages execution flow. It detects when the agent is stuck in a loop—calling the same tool with the same parameters and getting the same result. It implements retry logic with exponential backoff. It sets max iteration limits so your agent doesn't burn through $50 in API credits on an infinite loop.

**Pillar Three: Observability and Debugging.**

This is non-negotiable. You need a dashboard to see the agent's thought process, the steps it took, the tokens it used, and exactly where it failed.

You cannot fix what you cannot see.

I've been testing with Langfuse for tracing, and their open-source model has been a game-changer for debugging these complex, multi-step agentic workflows. Being able to replay an agent's entire execution—every LLM call, every tool invocation, every state transition—is like having X-ray vision for your AI system.

Without observability, you're debugging blind. Your agent fails, and you have no idea why. Was it a bad prompt? A hallucinated tool call? A rate limit? An API schema change?

Good observability answers these questions in seconds, not hours.

Think of it less like chaining API calls and more like building a central nervous system for your AI. The framework is the spine—connecting the brain (the LLM) to the limbs (the tools) and making sure everything communicates properly.

Point 2 — A Real-World Stack: How I Automated Client Reporting

Let me show you what this looks like in practice.

I run a weekly reporting pipeline for client analytics. The manual process took four hours every Monday morning: pulling data from 13 different client GA4 properties, summarizing trends, identifying anomalies, and drafting email updates for each client.

It was tedious, repetitive, and exactly the kind of task that should be automated.

Here's the architecture I built.

The orchestration layer uses LangChain. I know LangChain gets a lot of criticism—and some of it is deserved—but for complex multi-step workflows with diverse tool integrations, it's the most mature option available today.

The process starts with a webhook from my scheduler. Every Monday at 6 AM, a cron job fires a webhook to the LangChain application. The framework creates a new "session" with a fresh state store.

The first step is classification. A lightweight model—Claude Haiku—analyzes each client's GA4 data. It checks for significant traffic changes, conversion rate anomalies, and notable trends. Haiku costs about $0.25 per million input tokens. For this classification step, I'm spending roughly $0.002 per client.

For clients where Haiku detects significant changes, the framework triggers a more capable model. Claude Opus handles the heavy reasoning—analyzing the data, identifying root causes, and drafting the analysis. Opus costs more—about $15 per million input tokens—but it only runs on the subset of clients that need deep analysis.

This tiered approach is critical. You don't need a PhD-level model to check if traffic went up or down. You need a PhD-level model to explain why.

The agent's tool set includes BigQuery for data retrieval, the Gmail API for sending updates, and Google Docs API for creating shared reports. Each tool is defined with a typed schema. The framework validates every function call before execution.

Here's the flow:

Webhook from scheduler. Framework creates session with state store. Agent decides which clients to analyze based on Haiku's classification. For each client requiring deep analysis, Opus is invoked with the relevant data. Opus generates the analysis and drafts the email. The framework calls the Gmail API to send the update. State is saved after each step.

The result? The entire pipeline runs for roughly $0.07 per client per week . That's about $0.91 total for 13 clients.

It saved a full business day of work. Four hours every Monday, gone.

But here's the real value: the framework's state management means the system is resilient. If the Gmail API is down, the agent saves its state and retries later without starting from scratch. If BigQuery returns an error, the agent logs it, moves to the next client, and flags the failed one for manual review.

This isn't a fancy demo. It's a system that's been running in prod for 3 months, and it just works.

I monitored it closely for the first two weeks. After that, I set up alerts for failures and let it run. In three months, I've had exactly two failures—both caused by upstream API changes that I caught within minutes thanks to the observability layer.

Mid-Roll CTA

Struggling to map this out for your own use case?

I've built a simple comparison spreadsheet evaluating the top 5 agent frameworks against these three pillars—state, orchestration, and observability.

It's free. Grab it at [YourWebsite.com/agentstack](http://YourWebsite.com/agentstack). Just enter your email and it's yours.

Point 3 — The Contrarian Take: You Might Not Need a Monolithic Framework

Now let me throw a wrench into everything I just said.

Here's the hype: everyone says you need to adopt a huge, complex framework to do anything agentic. LangChain, CrewAI, AutoGPT, Microsoft's Semantic Kernel—the list goes on.

Here's the reality: for many focused tasks, a "framework" can be a ~300-line Python script using LiteLLM for model routing and a simple Redis cache for state.

The principles matter more than the platform.

I've seen teams spend weeks integrating LangChain for a task that could have been handled by 50 lines of code and a SQLite database. They over-engineer from day one because they think they need the full framework.

Let me give you a concrete example.

A founder I know wanted to build an agent that automatically triages customer support tickets. His team spent two weeks setting up LangChain, defining complex agent chains, and debugging the framework's abstractions.

I looked at the requirements. It was a simple classification task with three categories and a conditional response.

I built the same system in an afternoon with 150 lines of Python, LiteLLM for model access, and a Redis cache for conversation history.

It worked better, cost less, and was easier to debug.

Here's the mistake I see teams making: they start with the framework and try to force their problem into it.

Don't let the framework dictate your logic. Your logic should dictate your choice of framework—or whether you even need one at all.

Start by manually simulating the agent's workflow yourself. Sit down with a whiteboard and map out every step. What data comes in? What decisions does the agent make? What tools does it call? What happens when a tool fails?

If you can't whiteboard the exact steps and decision points, no framework will save you.

The framework is not a substitute for understanding your problem.

Now, my prediction—medium confidence—is that we'll see a surge of smaller, composable libraries rather than everyone standardizing on one or two mega-frameworks.

LiteLLM for model routing. Pydantic for schema validation. Redis or SQLite for state. Langfuse for observability. These lightweight tools compose well and don't lock you into a single paradigm.

The mega-frameworks have their place—especially for complex, multi-agent systems with diverse tool integrations. But for 80% of agent use cases, you don't need them.

CTA — Your Task for Today

Your task for today is not to install a new framework.

It's this: take one repetitive cognitive task you do weekly and whiteboard it.

Map out the decisions, the data sources, and the desired outcome. What triggers the task? What information do you need? What are the decision points? What are the failure modes?

That blueprint is worth more than any tool.

Once you have the blueprint, you can evaluate whether you need a framework at all. If you do, that comparison spreadsheet I mentioned will help you choose the right one.

Head to [YourWebsite.com/agentstack](http://YourWebsite.com/agentstack) to get it.

This is how you move from theory to a deployed system. Not by installing the trendiest framework. By understanding your problem deeply enough to know what you actually need.

Cross-Promo

If you're deep in the weeds of building AI-powered products, you'll love our sister show, "ML In Production," where we break down model deployment, monitoring, and everything that happens after the notebook.

Find it wherever you listen.

Outro

Thanks for listening.

This is Nick, helping you ship AI that actually works.

See you in the next one.