Why can adding one leading space change a word's token ID?

Answer: Spaces join the token

Spaces join the token ✓ — Many GPT-style tokenizers learn chunks that include nearby spaces. To us, red and a space before red still look like the same word with minor formatting; to the model, they may be different token IDs. That is why invisible layout can matter more than a reader would guess.

The word becomes shorter — The word is not literally shorter; the segmentation changed. A leading space can be bundled with the following letters as one learned piece, which may even reduce or change the token count. The useful lesson is that token length is not character length with a simple exchange rate.

The model reads syllables — Syllables are a human speech pattern, not the normal unit of GPT-style tokenization. Tokenizers are trained from text statistics, so they merge common written chunks rather than spoken beats. This is why a silent leading space can affect token IDs even though it adds no spoken syllable.