Why don't GPT-style tokenizers feed every English word as single letters?

Answer: Sequences get too long

Words should stay intact — Keeping every word intact sounds tidy, but natural language has endless names, typos, compounds, and new terms. Pure word tokens create an open-vocabulary problem. Subword tokenization is the compromise: common pieces stay compact, while unfamiliar words can still be built from smaller parts.

Sequences get too long ✓ — A character-level input keeps every letter visible, but it makes the sequence much longer. BPE-style tokenizers compress frequent chunks, so the same context window can carry bigger pieces of text.

Characters are faster — Characters feel simpler to humans, but they are not automatically faster for a language model. They create many more processing steps and burn through the context window sooner. The alphabet is tiny; the problem is the long sequence created when every paragraph is spelled out letter by letter.