ai/tokens-why-a-chatbot-never-reads-your-letters-only-numbered-pieces.md

Tokens: why a chatbot never reads your letters, only numbered pieces

AI · 6 min read ·

A chatbot never reads your letters. Before it sees anything, your text is chopped into pieces called tokens, and every piece is swapped for a number.

Those pieces are learned, not hand-written: start with single characters, then keep gluing the most common neighbouring pair together. Teach a tiny tokenizer yourself and watch "the" turn into one token.

Dark card with the title and three token chips: _the, _c and at, each with a small ID number underneath

$ ls ai/

see all →
ai/embeddings-how-a-computer-measures-that-a-kitten-is-like-a-cat.md Embeddings: how a computer measures that a kitten is like a cat How does a computer know that kitten is closer to cat than to car? It can't read meaning, but it can measure it. An embedding turns a word… 4 min read · 3 Oct ai/machine-learning-by-example-how-a-computer-learns-apples-from-oranges.md Machine learning by example: how a computer learns apples from oranges Normal programs follow rules a person wrote. Machine learning turns that around: you give the computer examples with the right answers, and… 11 min read · 1 Oct ai/how-a-chatbot-writes-one-guessed-word-at-a-time.md How a chatbot writes: one guessed word at a time A chatbot doesn't plan its reply and then type it out. It repeats one small step: look at the text so far, give every possible next token… 12 min read · 1 Oct