Llm Inference Gpu Requirements 2024

Auto-generated transcript. Minor errors may exist. The audio is the authoritative version.

OPENING

Nick: Signal Notes. Episode 247. September 16th, 2024.

Nick: I ran the numbers last night. Between January and August this year, the average GPU memory required to run a single Llama 3.1 70B inference at acceptable latency went from 140GB to 180GB. That's a 28% increase in eight months. And nobody's talking about it.

Nick: Here's what I mean. If you're deploying LLMs in production right now, your GPU bill is probably the single largest line item on your infrastructure spreadsheet. I've got 13 sites running various inference pipelines. My GPU costs jumped 34% year-over-year. Not because I'm doing more inference — I'm doing roughly the same volume. The models themselves got hungrier.

Nick: So this episode is about one thing: how to understand what your LLM actually needs from a GPU, and how to cut that requirement by up to 50% without sacrificing quality. I've been testing these techniques on production traffic since June. They work.

THE REAL COST OF INFERENCE

Nick: Let's start with a baseline. If you're running a 70B parameter model — Llama 3.1, Qwen 2.5, Mistral Large — you need roughly 140GB of GPU memory just to load the weights at FP16. That's one A100 80GB. Or two. Or one H100 if you're lucky enough to have access.

Nick: But here's the thing nobody tells you. The weight memory is only the start. You also need memory for the KV cache — that's the key-value pairs the model stores during generation. For a 70B model with a 32K context window, the KV cache alone can eat 40 to 60GB. Suddenly your single A100 isn't enough for a single concurrent user.

Nick: I tested this on my own production pipeline. I have an agent that generates SEO-optimized blog posts. It uses a 70B model for the reasoning pass — planning the structure, deciding which sources to cite. In May, with a 16K context, it ran comfortably on a single A100. By August, after I upgraded to Llama 3.1 with its native 128K context support, the same pipeline required two A100s. Same task. Same output quality. Double the hardware cost.

Nick: That's not sustainable. Not for me running 13 sites, and not for any business deploying LLMs at scale. So I started digging into what actually drives these requirements, and where you can cut.

THE THREE LEVERS

Nick: After three months of testing, I've isolated three main levers that control GPU requirements for inference. Model size, batch size, and precision. Each one has a direct, measurable impact on memory and compute. And each one can be optimized.

Nick: Let's start with model size. This seems obvious — bigger models need more memory. But the relationship isn't linear. A 7B model uses roughly 14GB at FP16. A 70B model uses 140GB. That's 10x the parameters for 10x the memory. But the performance gain from 7B to 70B isn't 10x. It's more like 20 to 30% on most benchmarks.

Nick: So the first question you should ask yourself: do you actually need a 70B model for this task? I run classification across my sites — tagging content, categorizing products, routing support tickets. For that, I use Haiku. It's a small model, runs on CPU if I want, costs pennies per thousand classifications. And it's accurate enough that I don't need to involve the heavy models.

Nick: But for reasoning tasks — planning a blog post structure, analyzing a competitor's strategy, generating a complex prompt chain — I need the big models. That's where Claude Opus comes in. I route the heavy reasoning to Opus, which runs on Anthropic's infrastructure. I don't pay for GPU memory at all for those calls. I pay per token.

Nick: This is the operator aside. The architecture looks like this: A Python script running on a $20/month VPS receives a classification request. It checks the task type against a routing table. Simple classification goes to Haiku via API. Complex reasoning goes to Opus. Only the generation tasks — the actual content creation — hit my local GPU cluster. This single change cut my GPU memory usage by 40%. Not the inference volume, the peak memory. Because the big models aren't loaded unless they're needed.

BATCH SIZE AND THROUGHPUT

Nick: The second lever is batch size. This is where most people waste money. They load a model onto a GPU, send one request at a time, and leave 90% of the compute sitting idle.

Nick: Here's the math. A single A100 can process roughly 100 tokens per second for a 70B model. If you're sending one request at a time, that's 100 tokens per second total. But if you batch four requests together, the throughput doesn't drop to 25 tokens per request per second. It drops to about 60. So your total throughput goes from 100 to 240 tokens per second. That's 2.4x the output for the same hardware cost.

Nick: I tested this on my blog generation pipeline. Originally, I was generating posts one at a time. Each post took about 45 seconds. The GPU was busy maybe 30% of the time. The rest was overhead — prompt construction, API calls to the classification model, writing to the database.

Nick: I switched to batch processing. Now I collect incoming generation requests over a five-minute window. Then I send them to the GPU in batches of eight. Total time per batch: about 90 seconds. That's 11 seconds per post instead of 45. And the GPU utilization went from 30% to 85%.

Nick: But here's the catch. Batch size increases memory usage. The KV cache scales linearly with batch size. If one request needs 40GB for the KV cache, four requests need 160GB. Suddenly your single A100 is out of memory.

Nick: The solution is to match your batch size to your available memory. I run a quick profiling pass when the pipeline starts. It measures the KV cache size for a single request at the expected context length. Then it calculates the maximum batch size that fits within 80% of available memory. The remaining 20% is headroom for spikes.

Nick: For my setup — Llama 3.1 70B, 32K context, A100 80GB — the optimal batch size is three. Three concurrent requests. That gives me 2.1x throughput improvement over single-request processing. Not the theoretical 2.4x, but close.

PRECISION AND QUANTIZATION

Nick: The third lever is precision. This is where the big gains are hiding.

Nick: Most models are trained at FP32 — 32-bit floating point. That's 4 bytes per parameter. For a 70B model, that's 280GB. Nobody runs inference at FP32. It's not practical.

Nick: FP16 — 16-bit — cuts that in half. 140GB for a 70B model. That's the standard for most production deployments. But you can go further.

Nick: INT8 quantization drops to 8 bits per parameter. That's 70GB for a 70B model. INT4 drops to 35GB. Suddenly a 70B model fits on a single consumer GPU. An RTX 4090 has 24GB. Not quite enough. But an RTX 6000 Ada with 48GB? That works.

Nick: I've been running experiments with quantization since April. Here's what I found.

Nick: For classification tasks — sentiment analysis, topic tagging, intent detection — INT4 quantization causes no measurable accuracy loss. I tested it against a labeled dataset of 10,000 customer support tickets. FP16 accuracy: 94.2%. INT4 accuracy: 93.8%. Within the margin of error.

Nick: For generation tasks — writing blog posts, generating code, composing emails — the story is different. INT4 causes noticeable quality degradation. The output becomes less coherent, more repetitive. I ran a blind test with five human evaluators. They preferred FP16 output over INT4 output 78% of the time.

Nick: But INT8? That's the sweet spot. For generation tasks, INT8 quantization causes minimal quality loss. In my blind test, evaluators preferred FP16 over INT8 only 54% of the time. Essentially a coin flip. But INT8 cuts memory usage in half.

Nick: Here's my current setup. I run my classification pipeline at INT4. That's Haiku-sized models, 7B parameters, running on CPU or low-end GPUs. I run my generation pipeline at INT8. That's Llama 3.1 70B, running on a single A100. And I route the most demanding reasoning tasks to Claude Opus via API, where I don't manage the GPU at all.

Nick: The result: my GPU memory requirement dropped from 320GB — two A100s — to 80GB. That's a 75% reduction. And my monthly GPU bill went from $4,800 to $1,200. Same output volume. Same quality. Different precision.

THE CONTRARIAN TAKE

Nick: Here's where I disagree with most of what you'll read online.

Nick: Everyone says you need the most powerful GPU you can afford. H100s. A100s. The latest hardware. And sure, if you're training models, that's true. But for inference? Not always.

Nick: I run a production pipeline that generates content for 13 sites. Total monthly traffic: about 2.5 million page views. The inference workload is roughly 500,000 tokens per day. That's not a lot. A single A100 can handle that in about 90 minutes.

Nick: If I bought an H100 instead, my inference time would drop to about 45 minutes. But my hardware cost would triple. And the pipeline isn't latency-sensitive. The posts are generated overnight. Nobody is waiting for them.

Nick: The smarter play is to match your hardware to your workload. If you need real-time inference — chatbots, interactive agents — latency matters. You might need the expensive hardware. But if you're doing batch processing, scheduled jobs, background tasks? Buy older hardware. Buy consumer hardware. Quantize your models. Optimize your batch size.

Nick: I know a founder who runs a document processing startup. They process legal contracts. Heavy inference workload — about 2 million tokens per day. They bought four H100s. Cost them $120,000. Their utilization is under 20%. They'd be better off with eight A100s and a batch scheduler. Same throughput, half the cost.

Nick: The conventional wisdom is wrong because it assumes your workload is uniform. It's not. Some tasks need the big models. Some don't. Some need low latency. Some don't. The optimization is in matching the hardware to the task, not buying the biggest GPU you can find.

MID-ROLL CTA

Nick: I've been documenting all of this in a detailed optimization guide. It covers quantization, batch sizing, model routing, and hardware selection. It includes the exact scripts I use to profile memory and calculate optimal batch sizes.

Nick: You can get it for free at signalnotes.com/guide. No email required. No upsells. Just the guide and the code.

Nick: Also, if you're building with LLMs and want to discuss optimization strategies, I'm active on the Signal Notes Discord. Link in the show notes. I answer questions there every day.

MONITORING AND ADJUSTMENT

Nick: The biggest mistake I see: people optimize once and never look at it again.

Nick: GPU requirements change. Models get updated. Context windows grow. Your workload shifts. What worked in January might be wasteful in September.

Nick: I run a weekly monitoring script. It checks three things. First, GPU utilization — am I using more than 80% or less than 30%? If utilization is too low, I increase batch size. If it's too high, I check for memory pressure.

Nick: Second, KV cache size. This is the silent killer. The KV cache grows with context length. If your users start sending longer documents, your memory usage spikes. I track the 95th percentile KV cache size and compare it to available memory. If it's approaching 80%, I reduce batch size or increase quantization.

Nick: Third, latency. For batch processing, latency isn't critical. But if you're running any real-time tasks, you need to track it. I measure the time from request to first token. If it exceeds 500 milliseconds, I investigate.

Nick: Here's a concrete example. In July, I noticed my GPU utilization had dropped from 75% to 40%. Nothing had changed on my end. I checked the model provider's release notes. They had pushed a new version with a smaller KV cache implementation. My batch size was now too conservative. I increased it from three to five. Utilization went back to 70%.

Nick: If I hadn't been monitoring, I would have left 30% of my compute capacity on the table for months. That's real money.

THE ROAD AHEAD

Nick: Let me give you my prediction. Not as a thinkfluencer. As someone who runs this stuff on production traffic.

Nick: By Q1 2025, I expect GPU memory requirements for inference to increase another 15 to 20%. Models are getting bigger. Context windows are expanding. The trend is clear.

Nick: But the optimization techniques are also improving. Quantization is getting better. INT4 quality is approaching INT8 quality. KV cache compression is advancing. Speculative decoding — where a small model generates draft tokens and a large model validates them — is becoming practical.

Nick: My bet: by mid-2025, you'll be able to run a 70B model with 128K context on a single consumer GPU. Not at FP16. But at INT4 with KV cache compression and speculative decoding. The quality will be 90% of FP16. The cost will be 10%.

Nick: That's where I'm putting my infrastructure budget. Not into more hardware. Into optimization tooling. Into better quantization. Into smarter routing.

CLOSING

Nick: Here's what I want you to do this week.

Nick: One, profile your current GPU usage. Check your utilization, your KV cache size, your batch size. If you're not measuring, you're guessing.

Nick: Two, experiment with quantization. Try INT8 for generation. Try INT4 for classification. Measure the quality impact. I bet it's smaller than you think.

Nick: Three, route your tasks. Don't use a 70B model for everything. Use small models for simple tasks. Use APIs for the heavy reasoning. Save your local GPU for what it does best.

Nick: That's it for this episode. Signal Notes is produced by me, Nick. The show is part of the Build Log network.

Nick: If you want more on LLM optimization, check out our sister show, Build Log. We just dropped an episode on running Mixtral 8x22B on consumer hardware. It's wild.

Nick: Thanks for tuning in. Optimize your inference. Ship your work. I'll talk to you next week.