The Tokenizer

The model has never seen a letter. Every layer you built in this part operates on numbers: Chapter 2’s embedding turned integer ids into vectors, attention mixed them, the sampler chose the id of the next one. But the prompt in Chapter 1 was a string — "The capital of France is" — and the engine printed words, not numbers. Something turns the text into the integers the embedding expects, and turns the integers the sampler produces back into text. That something is the tokenizer, and it is the last piece of Chapter 1’s loop left to open.

A tokenizer is a fixed, two-way dictionary between text and integers. It is not learned by the engine and it is not part of the model’s weights; it ships alongside them, because the model was trained against these exact ids and would read any other numbering as gibberish. Its vocabulary is tens of thousands of pieces — whole words, word fragments, single bytes — each with an integer id, and a set of merge rules that decide how a run of text is cut into those pieces. That algorithm is byte-pair encoding, and forge does not reimplement it.

10.1 A library for the pieces, a layer for the rest

The byte-pair merges are a solved problem with a fast, exact implementation, so forge loads the released vocabulary through HuggingFace’s tokenizers library and calls it for the raw text. What forge adds on top is the handling the model’s chat format needs: recognizing the special tokens that mark the boundaries of a turn, and framing a user’s message the way the trained model expects to see it.

class Qwen3Tokenizer:
    def __init__(
        self,
        path: str | Path,
        eos: str = BASE_EOS,
        chat: bool = False,
        thinking: bool = True,
    ) -> None:
        self._tok = HFTokenizer.from_file(str(path))
        self.chat = chat
        self.thinking = thinking
        self._special_ids: dict[str, int] = {}
        for special in SPECIALS:
            ids = self._tok.encode(special, add_special_tokens=False).ids
            if len(ids) == 1:
                self._special_ids[special] = ids[0]
        self.eos_token = eos
        self.eos_token_id: int = self._special_ids[eos]

Listing 10.1 · the tokenizer loads the released vocabulary, then records the id of each special token by asking the library to encode it and keeping the ones that come back as a single id.

HFTokenizer.from_file(path) loads the released tokenizer.json — the vocabulary and merge rules the model was trained with. Then the loop over SPECIALS builds forge’s own lookup: for each marker like <|im_start|>, it asks the library to encode that exact string and, if the result is a single id, records it. A special token is one the vocabulary holds as one indivisible piece; the len(ids) == 1 check keeps exactly those. The end-of-sequence token — the id the sampler’s loop stops on — is pulled from that same table.

10.2 The markers that frame a turn

An instruction-tuned model does not read a bare question. It expects the question wrapped in markers that say here a user turn begins, here it ends, now the assistant speaks. Those markers are the special tokens, and getting them exactly right is what makes the model answer rather than ramble.

    def wrap_chat(self, text: str) -> str:
        wrapped = f"<|im_start|>user\n{text}<|im_end|>\n<|im_start|>assistant\n"
        # thinking=False PRE-CLOSES an empty reasoning block; True inserts nothing
        return wrapped if self.thinking else wrapped + "<think>\n\n</think>\n\n"

    def encode(self, text: str) -> list[int]:
        if self.chat:
            text = self.wrap_chat(text)
        out: list[int] = []
        for piece in filter(None, SPECIAL_SPLIT.split(text)):
            if piece in self._special_ids:
                out.append(self._special_ids[piece])
            else:
                out.extend(self._tok.encode(piece, add_special_tokens=False).ids)
        return out

    def decode(self, ids: list[int]) -> str:
        return self._tok.decode(ids, skip_special_tokens=False)

Listing 10.2 · wrap a message in the chat frame, encode by routing special markers to their ids and the rest through the library, and decode without discarding the markers.

wrap_chat builds the frame: the user’s text between <|im_start|>user and <|im_end|>, then <|im_start|>assistant to hand the turn over. encode is where forge’s routing lives. It splits the text on the special-marker pattern first, then walks the pieces: a piece that is a known marker becomes its recorded id directly; every other piece goes through the library’s byte-pair encoding. The property this protects is that a marker sitting against other text — answer<|im_end|> — resolves to the marker’s single id rather than being blended into the merges around it. The library, given these markers as known tokens, already isolates them on its own; forge does the routing explicitly anyway, so a special resolves to its id as forge’s own decision rather than an inherited one. decode runs the library in reverse and, critically, keeps the special tokens rather than stripping them, so what comes out is the exact text that went in.

10.3 Correct, without the released file

The released tokenizer.json is a large file, and it is not needed to test forge’s part. The byte-pair merges belong to the library; what forge owns is the round trip and the special-token framing, and both can be checked on a tiny vocabulary built in memory. The driver constructs a small byte-level tokenizer with the same library, registers the special markers, and hands it to forge’s Qwen3Tokenizer. Then it encodes a handful of strings — plain text, unicode, and one that already contains markers — and requires each to decode back to exactly what went in; and it encodes a chat-wrapped message and checks the frame’s markers land as their own ids in the right order.

Four strings go in and four come back unchanged, and the chat frame carries <|im_start|> twice — once to open the user turn, once to open the assistant’s — with <|im_end|> between them. That proves forge’s encode and decode are inverses and that the chat framing routes its markers to their ids. What it does not prove is the released vocabulary’s merges: those live in the library and the real tokenizer.json, and they ran in Chapter 1 when the engine read a real prompt. This check covers the layer forge wrote — the framing and the round trip — on a vocabulary small enough to build in a millisecond.

The tokenizer closes the loop. A string becomes ids, the ids run through the forward pass and the cache, the sampler picks the next id, and the tokenizer turns the growing list of ids back into text — every call in Chapter 1’s stream() is now code you have read and checked. What is left in Part I is not another piece of the engine but a way to reason about it: why the same correct engine can be fast or slow, and which of its operations decides. That is the roofline.