Full transcript / extracted text of how-llm-tokens-work. Source: https://www.youtube.com/watch?v=nKSk_TiR8YA
So many devs are working with LLMs these days and they don’t really know about the fundamentals. I was teaching an AI workshop recently in Poland and I asked people to put up their hands if they knew what a token was. And I don’t know whether it’s my AI bubble on X or something, but I was pretty shocked because only about a third of the people in the workshop put up their hand. And so I wanted to put together a token deep dive for those of you who are using LMS but kind of need to catch up on what’s going on under the hood. And of course, we’re going to explain all of these concepts using TypeScript code. I promise I will never show a line of Python on this channel. That is for sure. Let’s start with a diagram here. Tokens are the currency of LLMs. When you send an input of Hello World to an LLM, that gets broken down into its constituent tokens. If you send hello world to OpenAI, that is three tokens build at an infantessimally small amount per 1K tokens. This is for GPT5, I think. Then the LLM will do some thinking and produce some output tokens. In this case, it said high and that is build at a different rate to the input tokens. So to get the total amount you spent, you take that number, divide it by a,000 and times it by a scent. Do the same with the input tokens. You add them together and you’ve got your total cost for that API call. I’m going to demonstrate that using the ASDK. For those who are not familiar with the ASDK, I’ve got a great free tutorial on my site AI Hero, which I’ll link uh up there. Now we’re going to generate some text using claw 3.5 haik coup a nice cheap model from anthropic and we’re going to send it hello world. We’re then going to log out two things. We’re going to log out the text that it produced and also the usage. This usage which we get from the ak is the amount of tokens that we used. We can see that it replied with hello how are you doing today? Is there anything I can help you with? Which used 20 output tokens but bizarrely 11 input tokens. That’s pretty weird considering we only prompted it with like two words. Hello world. Let’s do the same thing again, but this time we’re going to call Google. We’ll use Gemini 2.0 flashlight, and we’ll log out the usage and the text that it produced. We can see that Anthropic still at 11 and 20. But in the Google version, we only used four input tokens and 11 output tokens. So, this feels a bit like a mystery if you don’t know how tokens work. What is the relationship between tokens and text? And why does the same prompt sent to two different models end up with a different number of tokens? Okay, let’s reveal the mystery and talk about what tokens actually are. Every LLM has a different token vocabulary. And these tokens are all of the different words, subwords, and characters that it knows. Each of these words and subwords gets assigned a number. And the number is the token. When we call an LLM, it actually encodes the text that we send it into tokens. So, hello world gets split up into its largest individual tokens in the vocabulary. In this case, that might mean hello, space, world, and exclamation mark. And then it identifies the numbers in the vocabulary that match that word or subword. We can actually take a look at this in code in Typescript because OpenAI’s tokenizer is called tick token. And there’s a JavaScript implementation of it called JS tick token. We’re using O200K base, which is the tokenizer for the model GPT40. I’ve also got here some text at input.md, the wise owl of moonlight forest, where ancient trees stretch their branches toward the starry sky. We’re reading that big chunk of text into memory here, and we’re using the GPT40 tokenizer to encode a bunch of tokens, which are just an array of numbers. Then we’ll compare the input length in characters and the number of tokens that it produces. When we run this, we can see that the content length is nearly 2,300 characters, but the number of tokens is less than 500. and we end up with just a bunch of numbers here. If we go ahead and test this with something simpler like hello world, then here the content length is 12 and the number of tokens is only three. These tokens then are what gets fed into the LLM while it does its processing. Let’s look at a diagram of the full LLM process end to end. We pass in hello world. It gets split into the largest chunks that it knows about in the vocabulary. Those then get encoded individually into tokens by just looking up the number assigned to them in the token vocabulary. Then the LLM does some thinking here and it outputs some more tokens. In other words, all the computation here is being done on numbers, not on text. And these output tokens then get decoded back into text join together and you get the final output. So that’s the process. And while you think that the LLM is dealing with text, it’s actually dealing with these numeric representations of chunks of text. And though we get the whole picture here, we can decode these tokens here using tokenizer.deode. decode. Decode is just a function that takes in an array of numbers, tokens, and returns a single concatenated string. And when we run this, that is exactly what we get. By the way, if you’re digging this, then I have something cooking for you on aihero.dev. And I’ve got a newsletter where I post breakdowns like this, which will be in the description below. So, we know what tokens are now, but we don’t really know why they differ between model providers. To understand this, we need to know how these vocabularies even get built and why different model providers do them differently. Here’s the process for how tokenizers get trained. Let’s imagine you’ve got a corpus of text here. We’re going to use an extremely small corpus, just the single line, the cat sat on the mat. But in reality, this would be gigabytes, terabytes of data. The same data in fact usually that the model itself is trained on. Now, we can build an extremely simple tokenizer just from this one sentence. We can take this piece of text and extract out the unique characters in it. It’s probably simpler to look at this in code. I’ve created a character level tokenizer that takes in a data set and returns a tokenizer. And yes, it’s in a class. TypeScript classes are good, folks. I’ve been saying this for a while. We’ll get to this implementation in a minute, but let’s actually look at the usage. We have our data set of the cat sat on the mat. Again, in a real application, this might be gigabytes of text. Then we grab our tokenizer by instantiating a character level tokenizer. and I’m passing in an input of cats sat mat. We’re then encoding the input, turning it into tokens and logging out some information about the tokens. When we run this, we can see that the input length, the number of characters in cats mat is 11. Then the number of tokens that we produce at the end is also 11. This is because we only have characters as tokens. If we remove all these logs and just log something called tokenizer.mapping, we can see our entire vocabulary of tokens. And we can see there are only 10 tokens in the entire vocabulary. Space is three, n is 8, o is seven. So that means no matter what we do, the number of tokens is always going to be the same as the number of characters. And this is not good because the more tokens we have in the input, the more work it’s going to be for our LLM to process them. The vocabulary size is really, really important. Let’s say we only have about a thousand tokens in our vocabulary. If we take in the word understanding, it might produce five tokens under st and ing. If we bump that up to 50,000, then we might get larger subword chunks. So 50k might produce under standing, three tokens. But to 200k, we might split it into just two tokens. And two tokens is a lot more efficient for the LM to process than five tokens. So these are the trade-offs that different models and different model providers make. By the way, you can’t just scale this to infinity too because the larger your vocabulary, the bigger the model needs to be in order to house that vocabulary and of course the more memory it takes to execute. So then we know that our implementation of character level tokenizing is not going to be very good because we’re maximizing the number of tokens. So how do we get from character level to adding some more subwords? Well, we identify different groups that commonly occur together. For instance, th occurs in ‘the’ and ‘the’ he occurs in ‘the’ and ‘the’ as well. And then a occurs in cat, sat, and matt. I’ve got an example of this called the subword level tokenizer. Now, I’ve basically vibe coded this, so don’t look too closely at the implementation, but let’s take a peek at the usage down below. We have our same data set and our same input. And let’s take a look at how many tokens we get back from the input of cats, sat, and matt. We see here that the input catsmat is 11 and we get eight tokens back. We have achieved this by calculating some subwords in the data set. Let’s look at this by logging out tokenizer mapping. And we can see here that we now have 15 tokens at the and h are all in as our espace and tspace have each been assigned their tokens. This means our encode function is a little more complicated because it now needs to match the longest possible subword so that it can tokenize the input text efficiently. The implementation is below if you want to take a closer look. In our implementation, we only go one level deep and so we end up with groupings that are only two characters long, but real tokenizers identify larger and larger groups by identifying groups of groups. So for instance, in this example, TH and H often appear together. And so a group of three tokens, the makes sense. Finally, then we end up with a set of numbers that are each assigned to the individual token as we saw in our implementation. Finally, let’s look at how a tokenizer behaves when it encounters an unusual word. This is o fragous day from the Louiswis Carol poem. Frajus is just a word that Lewis Carol made up. So, it’s not going to be very frequent in the data set. When we run this through 0200K, Fragus is actually split up into four separate tokens here. And so, unusual words are split up into more tokens than words that occur more frequently in the data set. This also means that if you’re querying the LLM in a language that’s not present in the data set much, it’s likely to break it up into more tokens because it’s not used to those combinations of letters. This is true for spoken languages, but also for coding languages. And this is yet another advantage that more commonly used coding languages have in the AI era. It’s fewer tokens to send 20 lines of JavaScript than it is to send 20 lines of Haskell or something. So, let’s sum up here. Tokens are the currency of LLMs and you are charged by the token, but different model providers will treat these inputs differently and produce different numbers of tokens. Encoding tokens means you turn text into tokens. You take a piece of text, you split it into the largest tokens you can find, and then you turn them into the numbers that you have them stored in the vocabulary. Decoding is much simpler even. You just take the numbers, you find the relevant chunks in your vocabulary, and you join them together. And the whole LLM process involves an encoding step. than the LLM thinks, produces some output tokens which then get decoded into text. So tokens are really what the LLM does its thinking in. If you dug this video, then I have plenty more content like this coming. I’ve been full-time thinking about AI plus Typescript for a year now. And if you’re interested in learning more about AI or building with AI especially, then this channel is for you. I personally think that TypeScript is the language for building AI powered applications, whereas Python is probably the thing you want to use if you’re building models. So head to aihero.dev if you want to follow along. I have something dropping there very very soon. If you like this explanation then what should I cover next? Please comment below. Thanks for following along and I will see you very