One emoji can be several UTF-8 bytes. What can a byte-level tokenizer do?
Show answer & explanation
Answer: Start from byte pieces
Start from byte pieces ✓ — Emoji are Unicode symbols encoded as bytes, and byte-level tokenizers start from byte values rather than human-visible characters. Those bytes may then be merged or left as separate token pieces. A tiny picture can therefore be computationally larger than it looks on screen.
Read Unicode code points — Unicode code points are a real layer, so this is a plausible guess. But byte-level BPE starts from encoded byte values and then applies learned merges. The surprise is that a visible emoji can be handled through lower-level byte pieces, not only through the character you see.
Always keep one token — One visible character is not guaranteed to stay one token. Emoji sit in Unicode and may enter the tokenizer as multiple UTF-8 bytes. This is the same trap as letter counting: screen units and model units do not line up neatly.
More Technology questions
- Why can a satellite redness index flag rusting Arctic rivers but still fail to tell which streams are chemically hazardous?
- A service publishes a fresh random number every minute, plus a public log in which each entry carries a fingerprint of the entry before it. What does that chaining add?
- Why can Cloudflare's lava-lamp camera feed improve encryption even though the cryptographic software that consumes it is deterministic?
- If an attacker learns a pseudorandom generator's seed and algorithm after watching several outputs, why can the later outputs become reconstructable?
- Why does a physical noise source provide security against an attacker who knows the code, while a phone game shuffle does not?
- At parking-lot speed, why do quiet EVs need alert sounds before tire noise helps?