Most developers build with LLMs without understanding the one unit that governs both cost and behaviour: the token. This talk peels back the layer between the text you send and the numbers the model actually computes on, using TypeScript throughout instead of the usual Python.
Tokens as the currency you are billed in
Everything you pay for is denominated in tokens, not words or characters. When you send an input to an LLM it is broken into constituent tokens, and both the input tokens and the output tokens are billed, at different per-1K rates. The total cost of a call is just input tokens divided by 1,000 times the input rate, plus output tokens divided by 1,000 times the output rate.
What trips people up is that the token count rarely matches your intuition about the text. Sending “hello world” to Anthropic’s Claude 3.5 Haiku via the AI SDK reports 11 input tokens and 20 output tokens, even though you only typed two words. The output tokens make sense once you see the reply (“hello, how are you doing today? Is there anything I can help you with?”), but 11 input tokens for two words feels wrong until you know how tokenization works.
Encode/decode, and why the same text differs across providers
Every LLM ships with its own token vocabulary: the full set of words, subwords, and characters it knows, each mapped to a number. Calling the model runs an encode step that splits your text into the largest chunks present in the vocabulary and swaps each chunk for its number. The model does all of its thinking on those numbers, never on text, then emits output numbers that a decode step looks up and concatenates back into a string. Decoding is the trivial half: take an array of numbers, find each chunk, join them.
Because each provider trained a different vocabulary, the same prompt fragments differently.
That is why “hello world” costs three tokens on OpenAI but Google’s Gemini 2.0 Flash-Lite reports only four input tokens for its version while Anthropic reports 11.
You can watch this directly in TypeScript with js-tiktoken, the JavaScript port of OpenAI’s tiktoken, loading the o200k_base tokenizer used by GPT-4o.
Encoding a ~2,300-character passage yields fewer than 500 tokens, and encoding “hello world” (12 characters) yields exactly 3, so tokens sit somewhere between characters and whole words in granularity.
How vocabularies are trained (BPE) and the vocab-size tradeoff
Vocabularies are learned from a corpus, usually the same data the model itself trains on. The naive baseline is a character-level tokenizer: feed it “the cat sat on the mat” and it extracts the 10 unique characters as the entire vocabulary, so token count always equals character count. That is the worst case, because every extra token is more work for the model to process.
The improvement is to merge characters that frequently co-occur into subword tokens, the core idea behind byte-pair encoding. Starting from characters, you find common pairs (“th” and “he” both recur inside “the”), assign them their own tokens, and repeat, building groups of groups until “the” itself becomes a single three-character token. In the demo this turns an 11-character input from 11 character tokens down to 8 subword tokens once “th”, “he”, and space-prefixed pieces get their own entries.
Vocabulary size is the central lever, and it is a genuine tradeoff rather than something to maximise. The word “understanding” might split into five tokens (under, st, and, ing) at a ~1,000-token vocabulary, three tokens (under, stand, ing) at 50,000, and just two tokens at 200,000, and two tokens is far cheaper for the model to process than five. But you cannot scale vocabulary to infinity: a larger vocabulary needs a physically larger model to house it and more memory to run, so providers pick different points on that curve.
Why rare words and less-common languages cost more tokens
A tokenizer is only efficient on text that resembles its training corpus.
Feed o200k_base the invented word “frabjous” from Lewis Carroll and it fragments into four separate tokens, because that letter combination is rare in the data.
Frequency, not length, drives the count: common words collapse into one or two tokens while unusual ones blow up.
This generalises beyond vocabulary curiosities. Querying in a language poorly represented in the training data costs more tokens because the model has not learned its common letter combinations, and the same holds for programming languages. Twenty lines of JavaScript encode into fewer tokens than twenty lines of Haskell, which is a quiet, compounding advantage for mainstream languages in the AI era.
Key takeaways
- Tokens are the billing unit; total cost is input and output tokens each divided by 1,000 and multiplied by their separate rates.
- Encoding splits text into the largest known vocabulary chunks and maps them to numbers; decoding just looks the numbers back up and joins them. The model computes only on numbers.
- The same text produces different token counts per provider because each trained its own vocabulary (“hello world” is 3 tokens on OpenAI, 4 on Gemini, 11 on Claude).
- BPE builds subword tokens by iteratively merging frequently co-occurring pieces; bigger vocabularies mean fewer tokens per word but a larger, memory-hungrier model.
- Rare words, under-represented languages, and niche programming languages fragment into more tokens, so mainstream inputs are cheaper to send.
Sources
- Watch the video: youtube.com/watch?v=nKSk_TiR8YA
- Full transcript: how-llm-tokens-work
- Related: i-built-an-llm-from-scratch, what-are-ai-agents