Why can adding one leading space change a word's token ID in a GPT-style tokenizer?

Answer: The space becomes part of the token.

The space becomes part of the token. ✓ — In GPT-style tokenizers, the tokenizer learns common chunks of text, and these chunks often include spaces. So 'red' and ' red' are different sequences of characters, and the tokenizer may assign them different token IDs. This is why adding a space can change the token ID.

The word gets shorter. — The word doesn't actually get shorter; the tokenizer just splits the text differently. A leading space can be grouped with the following letters as one piece, which might even change the number of tokens. The key is that token length isn't the same as character length.

The model reads it by syllables. — Syllables are a human speech pattern, not the usual unit for GPT-style tokenizers. Tokenizers are trained on text statistics, so they merge common written chunks rather than spoken beats. That's why a silent leading space can affect token IDs even though it adds no syllable.