The Roofline

The engine is finished and it is correct. Every call in Chapter 1’s loop is code you have read and checked. But correct is not the same as fast, and the Qwen3-0.6B model raises an immediate puzzle: it is small — six hundred million parameters, a bit over a gigabyte — yet generating text from it, one token at a time, is far slower than the raw arithmetic would suggest. A modern accelerator can do hundreds of trillions of floating-point operations a second; a single decode step needs only about a billion. By that measure a token should take microseconds. It does not. This last chapter of Part I is about why, and it gives you the one tool that explains it — the roofline.

The answer is not compute. It is memory. Every operation has two costs: the arithmetic it performs, counted in floating-point operations, and the data it moves between memory and the chip, counted in bytes. A processor has a ceiling on each — a peak compute rate in FLOPs per second, and a peak memory bandwidth in bytes per second — and any single operation is limited by whichever ceiling it hits first. Which one it hits is decided by a single ratio.

11.1 Arithmetic intensity and the ridge

That ratio is arithmetic intensity: the number of floating-point operations an operation performs for every byte it moves. It is a property of the operation, not the hardware. An operation with high intensity — many FLOPs per byte — keeps the compute units busy and is compute-bound. An operation with low intensity — few FLOPs per byte — starves the compute units while it waits on memory, and is memory-bound.

The hardware supplies the dividing line. Take the chip’s peak compute rate and divide by its peak memory bandwidth, and you get a number in the same units as intensity — FLOPs per byte — called the ridge point. An operation whose intensity is above the ridge is compute-bound; below it, memory-bound. To place any operation on the roofline, then, you need two counts from it: its FLOPs, and its bytes moved. forge’s benchmark code computes the bytes for a transformer directly from its shape.

    @property
    def weight_bytes(self) -> int:
        return self.params * self.dtype_bytes

    @property
    def kv_bytes_per_token(self) -> int:
        return 2 * self.layers * self.kv_heads * self.head_dim * self.dtype_bytes

Listing 11.1 · the two byte costs that dominate inference: the model’s weights, read in full on every step, and the key/value cache, which grows by this many bytes per token of context.

weight_bytes is the size of every trained parameter laid end to end — for Qwen3-0.6B, about 1.19 GB in half precision. kv_bytes_per_token is how much the key/value cache from Chapter 7 adds for each token already in the context. Those are the bytes that move. The FLOPs are simpler: running one token through the network is dominated by multiplying it against every weight once, which costs about two floating-point operations per parameter — roughly 1.2 billion for this model.

11.2 Why decode is memory-bound

Now put a single decode step on the roofline. To produce one new token, the engine reads all 1.19 GB of weights and the key/value cache for the context so far, and performs its ~1.2 billion operations. The arithmetic intensity is those FLOPs divided by those bytes — a little under one operation for every byte moved. forge’s estimator does exactly this comparison, timing the step against each ceiling and reporting which one wins.

def estimate(shape: ModelShape, gpu: GpuSpec, batch: int, context: int, prompt: int) -> Estimate:
    prefill_s = 2 * shape.params * prompt / gpu.flops_per_s
    step_bytes = shape.weight_bytes + batch * context * shape.kv_bytes_per_token
    memory_s = step_bytes / gpu.bytes_per_s
    compute_s = 2 * shape.params * batch / gpu.flops_per_s
    step_s = max(memory_s, compute_s)
    return Estimate(
        prefill_s=prefill_s,
        decode_step_s=step_s,
        decode_tokens_per_s=batch / step_s,
        memory_bound=memory_s >= compute_s,
    )

Listing 11.2 · time the step against both ceilings — bytes moved over bandwidth, FLOPs over compute rate — and take the larger. memory_bound records which ceiling set the time.

memory_s is how long the byte traffic takes at the memory ceiling; compute_s is how long the arithmetic takes at the compute ceiling; the real step cannot be faster than the slower of the two, so the step time is their max, and memory_bound is true when memory is the one that wins. For one token on an A100 — whose specifications forge records as 2,039 GB/s of bandwidth and 312 dense TFLOPs — the byte traffic wins by a landslide. Reading 1.66 GB at 2,039 GB/s takes about 0.8 milliseconds; the 1.2 billion operations at 312 TFLOPs would take four microseconds. The step is memory-bound, and the compute units sit idle more than ninety-nine percent of the time, waiting for weights to arrive. That 0.8 milliseconds per token is where the ~1,200-tokens-per-second ceiling comes from — not the arithmetic, the reading.

11.3 Prefill, and the road to Part II

Not everything is memory-bound. Consider prefill — the first pass over the whole prompt, from Chapter 7. It reads the weights once, but it pushes every prompt token through them in that single pass, so the same 1.19 GB of weights serves hundreds of tokens’ worth of arithmetic. Its intensity is hundreds of FLOPs per byte, well above the ridge: prefill is compute-bound. The same weights, read the same way, land on opposite sides of the roofline depending only on how many tokens ride along with each read.

That difference is the whole argument for Part II. If decode is slow because it re-reads the weights for every token, the ways to make it fast all attack that read. Batching makes many sequences share one pass through the weights. Quantization shrinks the weights themselves — half the bytes, half the read time. A paged key/value cache packs more sequences into memory so larger batches are possible. Each is a move against the memory wall this chapter just measured. Part I gave you an engine that is correct; Part II is about the roofline you just drew.

11.4 Correct, from the dimensions

The roofline needs no benchmark to place these operations — the intensities follow from the model’s shape and the chip’s published ceilings. The check computes the ridge for an A100, the intensity of a single-token decode step, and the intensity of a 512-token prefill, and confirms decode lands below the ridge while prefill lands above.

Decode’s 0.72 FLOPs per byte against a ridge of 153 is not a close call: it is more than two hundred times under the line, memory-bound past any doubt. Prefill’s 512 clears the ridge and is compute-bound. This is a first-principles estimate — it uses the model’s real dimensions and the A100’s published bandwidth and compute rate to predict which regime each operation is in, not to measure the exact milliseconds a specific run would take. What it establishes is the shape of the problem: for this engine, tokens cost memory traffic, and that is the thing Part II learns to spend less of.

Part I is complete. You have an inference engine that is correct — every layer built and checked against an independent implementation — and now a way to reason about its speed: the roofline that says, for this model, the enemy is memory. Part II takes that engine and makes it fast.