This is the first post in a series on “Inference Engineering”, the craft of serving large language models fast and cheaply. I’m writing the series as I learn the subject, partly to lock it in and partly because this material is surprisingly hard to find in one place. Here I cover the foundations: why inference has become its own discipline, how the field is organized, what we measure, and the roofline diagram that explains why a single chat request leaves an H100 almost entirely idle.

When a model generates text for one user, it does about one floating-point operation for every byte it reads from memory. An H100 needs about 295 to keep its compute busy. That gap is why generating tokens is memory-bound, why processing the prompt isn’t, and why most inference optimizations exist.

Software, then inference, then cost

Start with a simple chain: software, then inference, then cost. More and more of the code we ship is written with AI help, and every one of those AI calls is an inference request that someone pays for. Training a model is a one-time bill. Serving it is a bill that grows with every user, every day.

You can pay that bill to a closed API, token by token, or host an open-weight model on your own GPUs. Self-hosting can be far cheaper, but only if you serve the model well: later in this post, the same rented GPU comes out 30 times cheaper per token once it’s tuned properly. The model is the same either way. The difference is the serving stack.

That’s the core argument for the field. Open-weight models such as Qwen, Llama, and DeepSeek have closed most of the quality gap with closed models, and they’re free to download. When the model is free, the advantage moves to how well you serve it.

Inference also runs on a far wider range of hardware than I expected. At one end, a Raspberry Pi manages 1 to 5 tokens per second. At the other end is specialized silicon. Cerebras fits a whole model onto a wafer-scale chip and is reported to serve 1,000 to 3,000 tokens per second, and Taalas goes further by building a single model directly into the chip, with reported speeds of around 17,000 tokens per second. Meanwhile, Apple is pushing models onto phones and laptops so that user data never leaves the device.

The map: three layers

It helps to split inference engineering into three layers, each answering one question.

Figure 1. The three layers of inference engineering. You learn them from the inside out.

• Runtime: how fast can one GPU go? Attention variants, the KV cache, FlashAttention, quantization, batching, and speculative decoding. Everything here is about getting more useful work out of a single GPU every second.
• Infrastructure: how do we serve a million users? Splitting a model across GPUs (tensor, pipeline, and expert parallelism), running prompt processing and token generation on separate GPU pools, replicating engines, routing requests, and deploying close to users.
• Tooling: what lets me ship? Serving engines such as vLLM, SGLang, and TensorRT-LLM, plus orchestration such as Ray Serve. These package the first two layers so you can deploy in hours instead of months.

The order matters. A common failure is to jump straight to the tooling layer: install vLLM, deploy, and then get stuck when traffic grows and latency explodes, because you don’t know what any of the settings actually change. You can also go one level deeper. Beneath the runtime sit hand-written GPU kernels and, at the very bottom, the chip itself, which is where companies like Cerebras and Taalas compete.

The scoreboard: five numbers

Every inference decision moves at least one of five numbers.

1. TTFT (time to first token). How long the user stares at a blank screen. It’s dominated by prefill, the pass that processes the whole prompt, so long prompts hurt it most: a pasted 10,000-token contract, or a retrieval prompt stuffed with documents. Interactive apps aim for well under half a second.
2. ITL (inter-token latency). The gap between streamed tokens once the answer starts. It’s dominated by decode, which generates one token per pass through the model.
3. TPS (tokens per second). ITL turned upside down: TPS = 1000 / ITL in ms. An ITL of 50 ms means 20 tokens per second. Keep per-user TPS separate from system TPS, which is the total across all users.
4. P99 latency. The slowest 1% of requests. Averages hide this tail, and the tail is exactly what your most active users feel, which is why production service-level targets are written against P99.
5. Dollars per million tokens. The number your CFO asks about. Here’s a handy rule of thumb, where 278 is simply 1,000,000 ÷ 3,600:

$ per million tokens ≈ (GPU $ per hour ÷ tokens per second) × 278

An H100 rented at $2.50 an hour that produces 100 tokens per second costs about $6.94 per million tokens. Tune the same GPU to 3,000 tokens per second and the cost falls to about $0.23. Same hardware, 30 times cheaper, which is why the runtime layer matters to the business and not just to engineers.

These numbers pull against each other. Batching more users together raises total throughput but eventually slows each user’s stream. The job is to move one number without quietly wrecking the others.

The roofline: one diagram behind everything

This is the idea that made everything click for me. A GPU has two hard limits:

• Compute: how many floating-point operations (FLOPs) it can do per second. An H100 does about 989 TFLOPs/s in FP16 on its tensor cores.
• Memory bandwidth: how many bytes per second it can move from HBM, its main memory, to the compute units. An H100 moves about 3.35 TB/s.

Which limit you hit depends on one ratio, arithmetic intensity (AI): the number of FLOPs a piece of work performs for every byte it reads from memory.

AI = FLOPs performed / bytes transferred from HBM

When AI is low, the compute units sit idle waiting for data, and the work is memory-bound. When AI is high, data arrives faster than it can be used, and the work is compute-bound. The crossover is the ridge point:

ridge point = 989 TFLOPs/s ÷ 3.35 TB/s ≈ 295 FLOPs per byte

Below 295 you’re memory-bound on an H100. Above it you’re compute-bound. Plot attainable performance against AI and you get the roofline: a slanted line set by bandwidth that flattens into a roof set by compute.

Figure 2. The H100 roofline. Decode for one user sits far down the slope; prefill sits on the roof.

Decode: about 1 FLOP per byte

Let’s do the math. Take one projection inside a transformer layer: an input X multiplied by a weight matrix W of size d × d, stored in FP16 (2 bytes per number). During decode, X is a single token, so its shape is (1, d).

FLOPs = 2 × d²                     (a multiply and an add per weight)
Bytes = 2d (read X) + 2d² (read W) + 2d (write the output)
      = 4d + 2d², which is about 2d² for large d
AI    ≈ 2d² / 2d² = 1

An arithmetic intensity of about 1 is deep in memory-bound territory. On an H100 that means about 3.35 TFLOPs/s of useful work out of a possible 989, or roughly 0.34% of the GPU’s compute. The rest of the chip waits for the next weights to arrive from memory.

A quick sanity check makes this vivid. Llama-3-8B has about 8 billion parameters, so generating 100 tokens takes about 2 × 8 billion × 100 = 1.6 trillion FLOPs, which is less than 2 milliseconds of H100 compute. But every decode step also has to read all of the roughly 16 GB of weights from HBM. That takes about 4.8 ms per token, or nearly half a second for 100 tokens. Memory, not math, sets the speed.

Prefill: about 1,000 FLOPs per byte

Prefill processes all N prompt tokens at once, so X has shape (N, d) and the same weights get reused N times.

FLOPs = 2Nd²
Bytes = 4Nd + 2d²
AI    = 2Nd² / (4Nd + 2d²) = Nd / (2N + d)

For Llama-3-8B (d = 4,096) and a 2,048-token prompt, that’s 4,096 × 2,048 ÷ 8,192 = 1,024 FLOPs per byte, well past the ridge. Prefill is compute-bound.

Same model, same GPU, two opposite regimes. That one fact explains a lot of inference engineering: why TTFT and ITL behave so differently, why some systems run prefill and decode on separate GPU pools, and why so many optimizations focus on decode.

How optimizations move you around the roofline

Once you can place a workload on the roofline, you can predict what an optimization will do before you run it.

• Move right (higher AI): read fewer bytes per token. Shrinking the KV cache with techniques such as MQA, GQA, and MLA lives here, and so does FlashAttention.
• Move up (higher roof): quantization. FP8 tensor cores double the H100’s roof to about 1,979 TFLOPs/s, and FP8 weights take half the bytes, so quantization helps in both regimes.
• Climb the slope: batching. If 32 users share one forward pass, every weight read from memory now serves 32 tokens, so AI rises roughly in step with the batch size. In this simplified model, a d = 4,096 projection needs a batch of a few hundred to reach the ridge. That’s why servers pack in as many requests as their latency targets and memory allow.

Training vs inference: same math, different physics

The last idea ties it all together. Training and inference run the same forward pass through the same weights, but with opposite input shapes.

• Training feeds big batches of long sequences at once. Because the true next token is known at every position (this is called teacher forcing), all positions are processed in parallel and the GPU runs near its compute roof. It costs about 6 FLOPs per parameter per token, counting the forward and backward passes.
• Inference decode feeds one token at a time, because each new token depends on the one before it. It needs only about 2 FLOPs per parameter per token, yet without heavy batching it produces tokens far more slowly per GPU, because it’s memory-bound.

Training is a capital cost you pay once. Inference is an operating cost you pay for as long as the model serves users, and for a popular model, the inference compute spent over its lifetime can far exceed what went into training it.

What’s next

If decode leaves 99% of the GPU idle, the natural question is what all those memory reads are for, and whether some of them are wasted. In the next post I’ll follow a single prompt through the transformer, step by step, and show why recomputing the whole sequence for every new token is redundant. The fix is the KV cache, which turns out to be both the most important optimization in inference and the source of its biggest memory headache.

My main takeaway is a way of working as much as a set of facts. Define the configuration, measure the five numbers (TTFT, ITL, TPS, P99 latency, and cost per million tokens), place the workload on the roofline, predict what a change will do, change one thing at a time, and measure again.