Fine-Tune Llama 3 For Document Summarization

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

OPENING

I have a file on my desktop. It's been there for six months. It's a 47-page legal document from a contract negotiation. I've opened it exactly twice. The first time, I skimmed it for fifteen minutes and missed a critical indemnification clause. The second time, I spent two hours reading it cover to cover and still had to re-read three sections to understand the implications.

That's the reality of document analysis. We're drowning in text, and our brains are not optimized for sustained attention on dense material. The average professional spends 2.5 hours per day reading documents. That's 600 hours a year. Most of it is inefficient.

But here's what I've been running in production for the last three months. I fine-tuned Llama 3 on a dataset of 12,000 annotated document-summary pairs. The model now processes a 50-page contract in 90 seconds. It extracts key terms, identifies risks, and generates a structured summary that I can review in under five minutes.

The accuracy improvement was 34% over the base model. Not theoretical. Not benchmark scores on a leaderboard. I tested this on production documents with real legal stakes.

Welcome to Signal Notes. I'm Nick. I run 13 sites, automated pipelines, and I ship things that work. Today: fine-tuning Llama 3 for document summarization. Not theory. Not speculation. What I tested. What broke. What shipped.

THE COST OF MANUAL SUMMARIZATION

Let's start with the problem. Because if you don't understand the cost, you won't understand why fine-tuning matters.

I spoke with a legal operations manager at a mid-sized firm. She manages a team of six paralegals. Each paralegal reviews an average of 15 contracts per week. That's 90 contracts total. Each one takes between 45 minutes and 3 hours to read and summarize.

That's 135 to 270 hours of reading per week. For one team. At one firm. In one city.

Now multiply that across every law firm, every financial institution, every healthcare organization that processes documents. The numbers are staggering. And this is where Llama 3 enters the picture.

But here's the thing. The base model is good. It's not great. When I tested the base Llama 3 8B on a set of 50 legal documents, it produced summaries that were factually accurate about 72% of the time. That sounds decent until you realize that 28% of your summaries contain errors. In legal documents, that's not acceptable.

The base model struggles with domain-specific terminology. It doesn't understand that "indemnification" has specific legal weight. It can't distinguish between a material breach and a minor one. It treats all clauses as equal.

Fine-tuning changes that.

WHAT FINE-TUNING ACTUALLY DOES

Let me be specific about what fine-tuning does to Llama 3. Because there's a lot of vague language around this topic, and I want to ground it.

The base Llama 3 model was trained on a general corpus. Trillions of tokens from the internet. It knows about everything. But it doesn't specialize in anything.

When you fine-tune, you're not retraining from scratch. You're taking the existing weights and adjusting them on a focused dataset. Think of it like this: the base model is a general practitioner who knows a little about every medical field. Fine-tuning is sending that doctor to a six-month residency in cardiology. They still know general medicine. But now they can read an EKG and spot arrhythmias.

The mechanism is straightforward. You take your training data — input documents and their ideal summaries — and you run it through the model in training mode. The model generates a summary, compares it to your ideal summary, calculates the error, and adjusts its weights to reduce that error. Repeat that 10,000 times across your dataset, and the model learns the patterns specific to your domain.

Here's what I found in my testing.

I fine-tuned Llama 3 8B on three different domains. Legal contracts. Medical research papers. And financial reports. Each dataset had 4,000 document-summary pairs. The training took about 6 hours on a single A100 GPU. Total cost: roughly $48.

The results:

Legal contracts: accuracy went from 72% to 91%.
Medical research: accuracy from 68% to 87%.
Financial reports: accuracy from 74% to 93%.

But accuracy isn't the only metric that matters. I also measured something I call "critical error rate" — the percentage of summaries that contained a mistake that would materially change the reader's understanding.

The base model had a critical error rate of 14%. The fine-tuned models dropped to 3%.

That's the difference between "this looks right" and "this is right."

THE TRAINING DATA — THE MOST IMPORTANT PART

Everyone asks about hyperparameters. Learning rate. Batch size. Number of epochs. And those matter. I'll get to them. But the single most important factor in fine-tuning success is your training data.

I learned this the hard way.

My first fine-tuning attempt used automatically generated summaries. I fed the base model documents, had it generate summaries, and used those as training targets. The logic seemed sound: use the model to bootstrap itself.

The result was a model that produced summaries that were slightly better than the base model. But not by much. The critical error rate only dropped to 11%.

Why? Because the training data contained the same errors the base model makes. I was training the model to replicate its own mistakes.

The fix was expensive but necessary. I hired three domain experts — a lawyer, a doctor, and a financial analyst — to create manual summaries. Each expert summarized 1,000 documents in their domain. I paid $15 per summary. Total cost: $45,000.

That sounds like a lot. But consider: that's less than one paralegal's annual salary. And the resulting model saves my team 20 hours per week. The ROI was positive in under three months.

Here's what I learned about building good training data:

First, length matters. Your summaries should be consistent in length. I aimed for 150-250 words per summary for documents that were 2,000-5,000 words. If your training data has summaries ranging from 50 to 500 words, the model will struggle to learn the right format.

Second, structure matters. Every summary in my training data followed the same template: document type, key findings, risk items, action items. The model learned to produce summaries in that exact structure.

Third, edge cases matter. I made sure 10% of my training data were unusual documents. Contracts with unusual clauses. Research papers with contradictory findings. Financial reports with anomalies. The model learned to flag these rather than smooth over them.

THE ACTUAL FINE-TUNING PROCESS

Let me walk through exactly what I did. Because I want you to be able to reproduce this.

I used the Hugging Face Transformers library with PyTorch. The model was Llama 3 8B, which has 8 billion parameters. I used LoRA — Low-Rank Adaptation — which is a technique that dramatically reduces the number of parameters you need to update.

LoRA works by adding small trainable matrices to specific layers of the model. Instead of updating all 8 billion parameters, you only update about 0.1% of them. The rest of the model stays frozen. This reduces memory requirements from roughly 60GB to about 16GB.

Here are the exact hyperparameters I used:

Learning rate: 2e-4. I started with 5e-4 and the model was unstable. Loss would spike every few hundred steps. Dropping to 2e-4 smoothed it out.

Batch size: 8. With gradient accumulation of 4 steps, giving an effective batch size of 32. Larger batches actually hurt performance in my testing. The model converged faster with smaller batches.

Number of epochs: 3. I tested 2, 3, and 5 epochs. At 2 epochs, the model hadn't fully learned the domain patterns. At 5 epochs, it started overfitting — performing well on training data but poorly on new documents. Three was the sweet spot.

LoRA rank: 16. Higher ranks (32, 64) improved performance marginally but required significantly more memory. Rank 16 was the best tradeoff.

Sequence length: 4,096 tokens. Llama 3 supports up to 8,192 tokens, but I found that 4,096 covered 95% of my documents. Longer sequences require more memory and training time.

Total training time: 5 hours 47 minutes on a single A100 80GB. The cost was $48.

But here's the operator aside.

The first time I ran this, I forgot to save the model checkpoint. Five hours of training, gone. I had to start over. Always set save_steps to a reasonable number. I use save_steps=500, which saves a checkpoint every 500 training steps. If the training crashes — and it will — you can resume from the last checkpoint instead of starting from zero.

EVALUATION — THE PART EVERYONE SKIPS

Here's a mistake I made that cost me two weeks.

I evaluated my fine-tuned model using the same metrics everyone uses. ROUGE scores. BLEU scores. BERTScore. All the standard NLP metrics. And they all looked great. ROUGE-1 went from 0.42 to 0.67. BERTScore went from 0.83 to 0.91.

But when I actually used the model on real documents, something was wrong.

The summaries were technically accurate. They hit the right keywords. The ROUGE scores were high. But they missed the nuance. A contract summary would correctly list all the clauses but fail to flag that a particular clause was unusually one-sided.

The problem was that my evaluation metrics were measuring surface-level similarity, not semantic understanding.

I switched to a different evaluation approach. I had the same three domain experts — the lawyer, the doctor, the financial analyst — rate the summaries on a 1-5 scale across four dimensions:

Factual accuracy: Did the summary get the facts right?
Completeness: Did it cover all important points?
Actionability: Could someone act on this summary without reading the original?
Risk flagging: Did it identify potential issues?

The base model scored an average of 2.8 across all four dimensions. The fine-tuned model scored 4.2. That's a 50% improvement in perceived quality.

But here's the key insight: human evaluation is expensive. The three experts cost me $15 per summary evaluation, and I evaluated 200 summaries. That's $3,000.

My solution: I used Claude Opus as a proxy evaluator. I gave it the same rubric as the human experts. The correlation between Claude Opus ratings and human ratings was 0.89. Not perfect, but close enough for iterative development. I used Claude Opus for rapid evaluation during training and saved human evaluation for the final model.

TRANSFER LEARNING — ADVANCED TIP

Here's an advanced technique that saved me a lot of money.

I needed to fine-tune models for three different domains. Legal, medical, and financial. Doing each from scratch would have cost $144 in compute and required 9,000 expert-written summaries. Instead, I used transfer learning.

First, I fine-tuned Llama 3 on a general summarization dataset. I used the CNN/DailyMail dataset, which has 300,000 article-summary pairs. This cost about $30 in compute.

Then, I took that fine-tuned model and fine-tuned it again on each domain-specific dataset. The second fine-tuning only needed 1,000 expert summaries per domain instead of 3,000. And the training time dropped from 6 hours to 2 hours.

The results were nearly identical. The transfer learning models scored 4.1 on the human evaluation, compared to 4.2 for the models trained from scratch. A 2% drop in quality for a 70% reduction in data and compute costs.

This works because the first fine-tuning teaches the model the fundamental patterns of summarization — how to extract key information, how to structure a summary, how to maintain factual accuracy. The domain-specific fine-tuning then adapts those patterns to the terminology and conventions of each field.

WHEN NOT TO FINE-TUNE

I want to give you the contrarian take. Because I've been running this for three months, and I've found situations where fine-tuning is not the right approach.

First: very small datasets. If you have fewer than 500 high-quality document-summary pairs, you're better off using few-shot prompting with the base model. I tested this. With 200 examples, fine-tuning actually performed worse than the base model with 5-shot prompting. The model overfit on the small dataset and lost its general capabilities.

Second: rapidly changing domains. If your documents change frequently — new formats, new terminology, new regulatory requirements — fine-tuning becomes a maintenance burden. You'll need to retrain every few months. In that case, a well-crafted prompt with retrieval-augmented generation might be more practical.

Third: when you need maximum flexibility. Fine-tuning specializes the model. That's the point. But if you need the same model to summarize legal contracts, medical research, and financial reports, you're better off using multiple prompts or multiple base models. I tried training a single model on all three domains. It performed worse on all three than the domain-specific models.

The lesson: fine-tuning is a power tool, not a universal solution. Use it when you have the data, the stability, and the focus. Don't use it when you're lacking any of those three.

MID-ROLL CTA

Before I get into the deployment details, I want to offer you something practical.

I've put together a resource — it's a step-by-step guide to fine-tuning Llama 3 for document summarization. It includes the exact code I used, the hyperparameter configurations I tested, and the evaluation scripts. It's free. No email required. Just a direct download.

You can find it at signalnotes.com/llama-finetune. That's signalnotes.com/llama-finetune.

Also, if you want to stay updated on what I'm actually running in production — not what I'm speculating about — subscribe to the Signal Notes newsletter. I send one email per week. Every one includes a specific technique I tested that week, with the exact code and the results.

DEPLOYMENT AND PRODUCTION LESSONS

So you've fine-tuned your model. It's producing great summaries. Now what?

I made several mistakes in deployment that I want to save you from.

Mistake one: running the model on CPU. The fine-tuned Llama 3 8B takes about 45 seconds per summary on CPU. On a GPU, it takes 3 seconds. I initially tried to save money by using CPU inference. The latency was unacceptable for my workflow. I switched to a single T4 GPU on a cloud provider. Cost: $0.35 per hour. Each summary costs about $0.0003 in compute.

Mistake two: no input validation. The model handles documents up to 4,096 tokens well. But when someone uploaded a 15,000-token document, the model truncated it silently. The summary was incomplete. I added a check that rejects documents over 4,000 tokens and suggests splitting them.

Mistake three: no confidence scoring. The model doesn't know when it's uncertain. It will confidently generate a summary even for documents that are outside its training distribution. I added a simple confidence check: if the summary contains phrases like "I'm not sure" or "this document appears unusual," flag it for human review.

The production pipeline now looks like this: <