Field guide · LLM inference hardware · October 2026

Anatomy of a Token

Every chip that runs a language model comes with three headline numbers. Each one controls a different part of the work. Once you know which number controls which part, spec sheets start to make sense, and so do the trade-offs between a Mac Studio, a DGX Spark and a rack of H100s.

GB Memory capacity Does the model fit?

A yes-or-no gate. Having more of it doesn't make anything faster.

GB/s Memory bandwidth How fast does it write?

Sets tokens per second while the answer is generated (decode).

TFLOPS Compute How fast does it read?

Sets how long your prompt takes (prefill), and how many people one chip can serve at once.

Your mental model, checked

RightBandwidth sets tokens per second when one person is generating.
RightCompute is what makes reading the prompt (prefill) fast.
AddCompute also decides how many users one chip can serve at the same time. That's why datacenters care about it during generation too.
AddCapacity only tells you the model fits. A model can fit and still crawl if bandwidth or compute is missing.

Prefill and decode

One request, two different jobs

A model is a large pile of numbers called weights. Llama 3.3 70B at 4-bit is about 40 GB of them. To do anything, the chip streams those weights out of memory and multiplies them against your tokens. What differs between the two phases is how many tokens share each trip through the weights.

Prefill: reading the prompt

compute-bound

Model weights, streamed from memory each pass

Tokens using this pass: 9

WhyismyMacslowatlongprompts?Because

Memory buslight

Math unitsmaxed

One trip through the weights serves the whole prompt at once. The first answer token comes out at the end.

Decode: writing the answer

bandwidth-bound

Model weights, streamed from memory each pass

Tokens using this pass: 1

WhyismyMac…Becauseprefilliscomputebound.

Memory busmaxed

Math unitsidle

One full trip through the weights for every new token. Faded tokens are already stored in the KV cache.

Prefill handles your whole prompt in one pass. Each weight that arrives gets used for every prompt token at once, so the chip does thousands of operations for each byte it loads. Memory keeps up easily, and the math units are the bottleneck. Prefill decides how long you wait for the first word.

Decode writes the answer. Each new token depends on the one before it, so the model runs one full pass per token. Every active weight comes out of memory again and gets used for about two operations. The math units mostly wait for data. Decode decides tokens per second.

Writing speedtokens/s ≈ bandwidth ÷ bytes of active weights

Reading speedtokens/s ≈ compute ÷ (2 × active parameters)

Fits ifweights + KV cache + headroom ≤ memory

Example: Llama 3.3 70B at 4-bit is about 40 GB. An H100 moves 3,350 GB/s, so one user tops out near 3,350 ÷ 40 ≈ 84 tokens/s. A Mac Studio M3 Ultra moves 819 GB/s, so about 20. Real software lands at roughly 50–80% of these ceilings.

FLOPs per byte

Compute ÷ bandwidth is the chip's personality

Divide a chip's compute by its bandwidth and you get how many operations it can do for each byte it pulls from memory. Single-user decode at 4-bit needs only about 4 operations per byte: 2 per weight, half a byte per weight. Every chip below offers 30 to 460. So for one user, generation is always limited by bandwidth, and most of the compute sits idle.

The ratio still matters, because it tells you how lopsided the chip is. A low ratio means it writes quickly but reads long prompts slowly. A high ratio means it reads quickly but writes slowly.

Bubble size is memory capacity. Further right writes faster, higher up reads faster. Dashed lines connect chips with the same FLOPs per byte; the numbers sit at the right edge. Both axes are logarithmic.

Chip	Memory	Bandwidth	Compute	FLOPs per byte	Reads ÷ writes

Compute is the 16-bit dense peak. ≈ marks estimates: Apple doesn't publish TFLOPS, and the DGX Spark figure is derived from NVIDIA's FP4 number. "Reads ÷ writes" is how many times faster the chip processes prompt tokens than it generates answer tokens, for a 4-bit model after typical efficiency losses.

Big memory, small surprises

Three ways a chip disappoints

Capacity looks like the spec that matters most, because it decides which models you can run at all. But a chip is only as good as its weakest number for the job you give it. Bars below use a log scale across the chips on this page.

Fits, but writes slowly

DGX Spark · Ryzen AI Max+ 395 · 128 GB at ~260–270 GB/s

128 GB holds a 70B dense model easily. But every token has to drag the whole model across a 256-bit memory bus, so a 70B model in 8-bit writes at about 3 tokens per second. LMSYS measured the DGX Spark at 2.7 tokens/s on Llama 3.1 70B in FP8, while it read the prompt at 803 tokens/s. Its compute is fine; its bandwidth is the limit. These boxes work well with mixture-of-experts models like gpt-oss-120b that touch only a few GB per token.

Writes well, reads slowly

Mac Studio M3 Ultra · 512 GB at 819 GB/s, ~28 TFLOPS

512 GB holds DeepSeek-class models and writes them at a usable pace. With so little math per byte, though, a 30,000-token prompt from a coding agent can take minutes before the first word appears. Apple added matrix units to every GPU core in the M5 generation to fix exactly this. On an M5 MacBook Pro, Apple measured up to 4× faster time to first token than M4, while generation improved 19–27%, close to the 28% bandwidth increase. The M5 Ultra Mac Studio pairs 512 GB with 1.2 TB/s, and Apple claims up to 4× faster prompt processing than the M3 Ultra.

Fast, but too small

GeForce RTX 5090 · 32 GB at 1,792 GB/s

Strong bandwidth and strong compute, but 32 GB. A 70B model at 4-bit needs about 40 GB before any context, so it doesn't fit. Two cards can split the model, with the link between them becoming a new bottleneck. For models that do fit, it is one of the fastest single chips you can buy.

Try it

Same request, seven chips

Pick a model and a job. Each bar shows how long the whole request takes, split into reading the prompt and writing the answer. Tap a chip to see its memory and how it copes with many users at once.

Model

Weight precision

16-bit 8-bit 4-bit

Prompt

Answer

Reading the promptWriting the answerDoesn't fit

Memory

Serving many people at once

Limited by bandwidthLimited by computeNo room for more KV caches

Modifiers

What else moves the numbers

Quantization

capacitybandwidth

Going from 16-bit to 4-bit makes the weights about 3.5× smaller. The model fits in less memory and writes about 3.5× faster, since each token reads fewer bytes. Prompt reading only speeds up if the chip has fast low-precision math, like FP8 and FP4 on recent NVIDIA chips.

Mixture of experts

capacitybandwidthcompute

Total parameters decide what has to fit. Active parameters decide the work per token. DeepSeek V3 has 671B parameters but uses about 37B per token, so it needs a huge memory pool yet writes like a 37B model. That's why 512 GB Macs and MoE models go well together.

KV cache

capacitybandwidth

The model stores two vectors per token per layer so it never recomputes the past. It grows with context: Llama 3.3 70B uses about 320 KB per token, so a 128k-token context is roughly 40 GB. It eats capacity, and every new token reads all of it, which is why long conversations write more slowly.

Batching

computecapacity

A server reads the weights once per step and applies them to every user's next token. Throughput grows almost for free until the math units fill up, or until there's no room left for everyone's KV cache. This is how compute matters during generation, and why cloud tokens are cheap.

Speculative decoding

computebandwidth

A small draft model guesses the next few tokens, and the big model checks all of them in one pass, like a tiny prefill. When the guesses are right, idle compute turns into extra tokens per second.

Linking chips

capacitybandwidth

Splitting a model across chips adds their capacity. Splitting each layer across them adds bandwidth too, if the links are fast enough. That is what NVLink does inside a GPU server. Apple says four M5 Ultra Mac Studios linked over Thunderbolt 5 run inference up to 3× faster than one.

On-chip SRAM designs

bandwidthcapacity

Groq and Cerebras keep the weights in SRAM on the chip itself, which is far faster than any external memory but holds very little per chip. You get extreme tokens per second, paid for with racks of chips to hold one model.

Software

bandwidthcompute

The same chip can differ 2× between engines. llama.cpp, MLX, vLLM and SGLang each reach a different share of peak bandwidth and compute, especially on newer hardware where kernels are still catching up.

Cheat sheet

Reading a spec sheet in 30 seconds

Capacity first. Total parameters × bytes per weight (2 for 16-bit, 1 for 8-bit, about 0.56 for 4-bit), plus the KV cache, plus a few GB of headroom. If it doesn't fit, nothing else matters.
Then bandwidth. Divide it by the bytes of active weights. That's the best-case tokens/s for one user. Expect 50–80% of it in practice.
Then compute. 2 × active parameters × prompt tokens ÷ FLOPS gives seconds to the first word at perfect efficiency. Double it for a realistic figure.
Then your workload. Chat with short prompts lives on bandwidth. Coding agents, RAG and long documents push lots of prompt through, so compute starts to dominate. Serving many users needs compute and room for everyone's KV cache.