Local Rag With Ollama Tutorial

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

**Signal Notes. Episode 47. Local Rag With Ollama Tutorial.**

—

The Hook

You've heard that running a local LLM is the secret to private, powerful AI. Everyone says it. The privacy advocates. The open-source purists. The people who've never actually tried to deploy one.

The moment you try, you get a 10-gigabyte download. A cryptic command line error you've never seen before. A melted CPU fan that sounds like a jet engine preparing for takeoff.

I've been there. Three months ago, I spent a weekend trying to get an offline RAG system running on my production laptop. I had 13 sites to manage, client documents that legally couldn't touch OpenAI's servers, and zero patience for academic tutorials that assumed I had a PhD in distributed systems.

What if you could build a Retrieval-Augmented Generation system that's not only powerful but also runs entirely offline on a standard laptop? No API calls. No data leaving your machine. No monthly bill.

Today, we're deploying exactly that in under 15 minutes. I've been running this stack for internal wikis across my agency for two months. It saves me roughly six hours a week on document retrieval alone. And it cost me exactly zero dollars in API fees.

Here's the thing. This isn't a demo. This is a production-ready architecture I've stress-tested on a three-year-old MacBook Air with 8 gigs of RAM. If it runs there, it'll run in your environment.

Let's start with why you'd even want this.

—

The Context — Why Local Matters Now

The AI hype cycle tells you to send your data to an API. Every tutorial. Every startup pitch. Every LinkedIn post from someone who's never shipped a production system.

In production, that's a data privacy nightmare. I have clients in legal and healthcare. Their documents can't touch a third-party server. Period. And even if you don't have regulatory constraints, there's the latency problem.

Every API call adds 500 milliseconds to three seconds of network overhead. That doesn't sound like much. But when you're building an internal tool that your team uses fifty times a day, that latency kills adoption. People stop using it. The tool dies.

The alternative, local LLMs, has been stuck in the domain of hardcore researchers with ten-thousand-dollar gaming rigs. You'd need to understand CUDA versions, PyTorch builds, model quantization. It was a nightmare.

Until Ollama.

Ollama has fundamentally changed the game. It's the Docker of LLMs. A simple tool that lets you pull and run models like Llama 3.1 as easily as you'd run an Nginx container.

One command to install. Two commands to pull models. One command to run inference.

I've been running this for two months across my agency. We have an internal wiki with 400 pages of documentation. Before, finding the right procedure meant digging through Slack messages and outdated Google Docs. Now, I type a question, and the answer comes back with citations, generated entirely offline, in under two seconds.

The cost? Zero. The setup time? Fifteen minutes. The hardware? A machine you already own.

Let me show you the architecture.

—

The Architecture — Your Offline RAG Pipeline, Deconstructed

Forget the academic papers. Let's talk in operator terms.

Here's the simple, three-part pipeline we're building.

**Part one: The Ingestion Layer.**

This is where we chunk and embed. We'll use a simple Python script with SentenceTransformers to turn our documents into vectors. I've been using this for client onboarding documents, internal wikis, and even my own book manuscripts.

The process is straightforward. You load a PDF. You split it into chunks of around 500 tokens with some overlap. You generate embeddings for each chunk. Then you store those embeddings.

**Part two: The Vector Store.**

Think of this as the brain's hippocampus. We're using ChromaDB. It's lightweight. Runs in-memory. Stupidly simple to set up. No server required. No Docker container to manage. No cloud service to configure.

I chose ChromaDB because I've been burned by over-engineered solutions. When I'm building an internal tool, I don't want a distributed database. I want something that works with a single import statement.

**Part three: The Reasoning Engine.**

This is Ollama. We pull the nomic-embed-text model for generating embeddings. It's small, fast, and surprisingly accurate for document retrieval.

Then we pull a reasoning model. I recommend starting with Llama 3.1 8 billion . It's the sweet spot between capability and speed. On my MacBook Air, it generates answers in under a second for most queries.

Here's the key insight that most tutorials miss.

The pipeline is only as strong as its weakest component.

Most people obsess over the reasoning model. They want the biggest, most capable LLM. But if your retrieval is bad — if you're feeding the model the wrong context — the best LLM in the world will give you wrong answers.

I learned this the hard way. In my first iteration, I used a massive model with terrible chunking. The answers were confident but wrong. The citations pointed to irrelevant sections. It was worse than useless because it looked authoritative.

The fix wasn't a better model. It was better chunking, better embedding, and better retrieval.

Let me show you the exact commands to build this.

—

The How-To — From Zero to Local RAG in 10 Minutes

Stop reading tutorials. Here are the exact commands to go from a blank terminal to a functioning AI.

**Step one: Install Ollama.**

One command. Open your terminal. Run this:

curl -fsSL https://ollama.ai/install.sh | sh

That's it. The script detects your operating system, installs the binary, and sets up the service. It takes about thirty seconds.

**Step two: Pull your models.**

Two commands:

ollama pull nomic-embed-text

ollama pull llama3.1:8b

This is the big download. The embedding model is about 274 megabytes. The reasoning model is about 4.9 gigabytes. Combined, you're looking at roughly five gigs of downloads.

But here's the thing. This is a one-time cost. Once the models are downloaded, they stay on your machine. You can run them offline. No internet required. No API calls. No data leaving your laptop.

**Step three: The Python script.**

This is where most tutorials lose people. They show you a 200-line script with complex abstractions. I'm going to show you the sub-50-line version I've been running in production.

Here's the structure:

First, we load the PDF. I use PyMuPDF. It's fast, reliable, and handles most document formats.

Second, we chunk the text. I split on paragraph boundaries with a 500-token window and 50-token overlap. This gives the model enough context without overwhelming it.

Third, we generate embeddings. We call Ollama's embedding endpoint with the nomic-embed-text model. This happens locally, in memory, in milliseconds.

Fourth, we store everything in ChromaDB. The collection, the embeddings, the metadata, the original text for citations.

Fifth, we query. You type a question. The script embeds your question. ChromaDB finds the most relevant chunks. Ollama generates an answer based on those chunks. You get a cited, accurate response.

The “aha” moment comes when you run the script for the first time.

You type a question about your company's vacation policy. The script whirs for a second. Then it prints a perfect, cited answer. No internet. No API key. No monthly bill.

I've seen this happen in client meetings. People's jaws drop. They can't believe it's running on a laptop.

But here's the truth. This script has run on a three-year-old MacBook Air with 8 gigs of RAM. If it can run there, it can run in your production environment.

The complete script is in the show notes. It's commented. It's tested. It's ready to copy and paste.

Now, let me tell you why smaller models win.

—

### MID-ROLL CTA

If you want to skip typing all that code, we've shipped.

The complete, commented Python script and a sample document are in a ready-to-run GitHub repo. Get it for free by signing up for the Operator's Manual newsletter at signalnotes.com slash ollama .

It's one email a week with the tools and tactics we're actually running in production. No hype. No theory. Just the stuff that works.

Back to the show.

—

The Contrarian Take — Why Smaller Models and Smarter Pipelines Win

Everyone is chasing the biggest, most parameter-heavy model. The 70-billion parameter models. The 120-billion parameter models. The models that require four A100 GPUs just to run inference.

I'm here to tell you that's usually wrong for a deployed RAG system.

Here's the 70-30 rule.

Let your RAG pipeline do 70 percent of the work. The retrieval finds the perfect context. The embedding similarity scores are high. The chunks are relevant and complete.

The LLM's job then becomes simple. Synthesis and phrasing. Not recall. Not reasoning from scratch. Just taking the provided context and turning it into a coherent answer.

When you set it up this way, a smaller model works perfectly. Llama 3.1 8 billion handles this task effortlessly. Even Phi-3 mini , with only 3.8 billion parameters, gives excellent results when the retrieval is good.

I tested this rigorously. I built a golden dataset of 100 questions from my internal wiki, with verified answers. I ran the same queries through four configurations:

Configuration one: Big model, bad retrieval. The answers were confident but wrong 40 percent of the time.

Configuration two: Big model, good retrieval. The answers were accurate 95 percent of the time. But inference took four seconds per query.

Configuration three: Small model, bad retrieval. The answers were wrong 60 percent of the time.

Configuration four: Small model, good retrieval. The answers were accurate 93 percent of the time. Inference took 0.8 seconds per query.

The small model with good retrieval was almost as accurate as the big model. But it was five times faster.

In production, speed matters. A five-second response time kills user adoption. People stop using the tool. They go back to searching through Slack messages.

A one-second response time keeps them engaged. They use the tool. They trust it. They tell their colleagues.

Here's my testing framework. I don't judge on flaky, subjective quality. I test with precision and recall on a golden dataset. I measure how many answers are correct, how many are partially correct, and how many are wrong.

Often, a well-architected RAG with a small model outperforms a huge model with bad context. The small model with good context gives you accurate, fast, cheap answers. The huge model with bad context gives you slow, expensive, wrong answers.

Stop optimizing for benchmark scores. Start optimizing for latency and accuracy in your specific domain.

Your users don't care about MMLU scores. They care about getting the right answer in under two seconds.

—

The CTA — Your Action for Today

Your action for today isn't to become an ML engineer.

It's to install Ollama.

Just run that one curl command.

curl -fsSL https://ollama.ai/install.sh | sh

Then pull the Llama 3.1 8 billion model.

ollama pull llama3.1:8b

Feel how quick it is. Run a test query. ollama run llama3.1:8b and type “Explain what a vector database is in one paragraph.”

Notice how the response starts streaming almost instantly. That's the foundation.

Then, head to the show notes. Grab that free script. Replace the sample PDF with your own documentation. Your company handbook. Your client onboarding documents. Your internal wiki.

In one afternoon, you'll have a proprietary AI tool that no one else does. It doesn't leak your data to third-party servers. It doesn't rack up an API bill. It doesn't require an internet connection.

I've been running this for two months. It saves me roughly six hours a week. That's 24 hours a month. That's 288 hours a year.

All from a 50-line Python script and two Ollama commands.

This is what shipping looks like.

—

### CROSS-PROMO

If you're building more than internal tools, you need to hear our sister show, Production Ready .

This week, we're breaking down how we did a zero-downtime migration of a live ML pipeline with over 10 million daily inferences. The architecture. The rollback strategy. The monitoring setup. The exact playbook we used.

Find it wherever you get your podcasts. Search for Production Ready Podcast .

—

The Outro

This has been the Local Rag. Episode 47 of Signal Notes.

I've been your host, Nick. The operator who ships, not the thinkfluencer who posts.

Here's what I want you to remember from this episode. The AI hype cycle wants you to believe that local LLMs are hard. That you need expensive hardware. That you need a PhD. That you need to send your data to the cloud.

None of that is true.

One command installs Ollama. Two commands pull the models. One 50-line script builds your RAG system. Fifteen minutes from blank terminal to functioning AI.

The models I'm running on a three-year-old MacBook Air are good enough for production internal tools. Good enough to save me six hours a week. Good enough to replace my dependency on paid APIs.

Now get out of the hype cycle and go ship something.

We'll talk next week.