Full transcript of i-built-an-llm-from-scratch. Source: https://www.youtube.com/watch?v=YmLp8qe87A0
What happens when you type a prompt into an LLM like ChatGBT or Claude or Gemini and press enter? The answer, as it turns out, involves over 80 years of research, billions of dollars, and some of the most elegant [music] math humans have ever invented. Now, to answer this question, I built an LLM from scratch. And I’m going to walk you through that full journey from keystroke to stream response with working code at every step. Now, our story actually starts back in 1950. Claude Shannon, the father of information theory, sat down with his wife, Betty, to play a game. And he showed her a passage of text with the next letter hidden. And she had to guess what came next, letter by letter. And he measured how often she was right. But what he was really measuring is how predictable is written language. And his finding was given enough context, the next letter is often nearly certain. Now, this means that language has deep statistical structure. And over 70 years later, that’s pretty much what an LLM is doing. It’s predicting what is coming next. And LLMs are statistical models of language. They extract patterns and relationships from massive amounts of text and they build a mathematical representation of how words relate to each other. And people often describe them as next token predictors. And while that’s true, there is a massive amount of machinery involved before you can get to the point of what comes next. And that’s really what I wanted to understand, right? We are increasingly interacting with these AI services on a daily basis whether it’s at home or at work and a lot of the details are hidden from us and most people kind of just handwave over oh it predicts the next token and and I wanted to dive deeper. I wanted to have a better understanding of all this stuff and so that’s exactly what I’m going to break down in this video. You’re going to walk away with a deeper understanding of how LLMs work. If that sounds good, let’s dive in. My name is CJ. Welcome to Syntax. Now before LLMs existed, we had chat bots of many forms over the past 60 plus years. And this really all started back in 1950 when Alan Turring published his paper computing machinery and intelligence. And this is where he proposed the famous Turing test which stated if a machine can converse indistinguishably from a human, can it think? And this really set the target for the next 70 years. Now in 1966, Joseph Weisenbound at MIT built a program called Eliza, which was essentially just a pattern matching program that mimicked a Rogerian therapist. It had no understanding whatsoever. Essentially, whatever the user typed, it would go through a series of if statements to determine how it should respond. And a program like this is known as a rule-based system. Essentially, a programmer has to go in and manually write out all the statements to detect based on what the user has typed how it should respond. So it has no built-in understanding. And famously Weisenbomb’s secretary was using the program and she actually asked him to leave the room so that she could talk to Eliza privately and he later wrote, “I had not realized that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” And we’re seeing this today, right? Like people are making comments about how dependent they are are on talking to chat GBT as if it’s their friend or their therapist even though there’s not a real human on the other side. Now later in 1972, Kenneth Colby at Stanford built a computer program called Perry which simulated a paranoid schizophrenia patient. It had internal states for anger, fear, mistrust, and it shifted its responses accordingly. And psychiatrists that sat down to talk to this chatbot couldn’t distinguish it from real patients. [music] And later in the year, Eliza and Perry were actually connected together. And you essentially had a therapist chatbot talking to a patient chatbot. After that, there were decades of rule-based systems. Alice was released in ‘95 and it had 41,000 handwritten patterns. [music] And Smarter Child, some of you might remember, was released in 2001 on AOL Instant Messenger. And this was one of the first chat bots millions of people used daily and could kind of be seen as a precursor to ChatGBT. It was kind of a a do everything chatbot. Now, every single one of these chat bots up until this point were based on rules or scripts or templates or decision trees. Essentially, a human had to sit down at a computer and program every single response in advance. So the jump from Eliza to chatgbt isn’t just a better chatbot. It’s a completely different mechanism. And before we dive into how LLMs work, let’s take a look at how a simple program like Eliza could be written. All right, so this is the simple chatbot I have set up here. And if I say hello, it will respond just like an LLM. Hello, how can I help you today? Or if I say hi, it will respond in the same way. So all of the code for everything I’m going to show you in this video is linked in the description. And we’re going to start here with this simple chat. And you can see I have a list of every possible greeting. And essentially if the user types any one of these greetings, it will respond with that exact response. Hello, how can I help you today? And we have this one function that handles all responses. So the user’s message is passed in and then we replace all the special characters, remove any empty white space. From there, we basically have a list of words that corresponds to the user’s message. And the first thing we do is check, is one of those greetings that we’ve defined, is that in the user’s message? And if it is, we immediately respond with, “Hello, how can I help you today?” So simple as that. If you were to just come across this chat UI and say, “Hi, you might initially think that there was an LLM on the other side because it responds in a very similar way to how chat GBT responds.” The next thing is we basically look at the user’s messages to determine how we can respond in various other ways. And the first check here is does the user’s message start with I feel. So I’ll say I feel happy today and then it will respond why do you feel happy today? So we essentially extract the thing after I feel. We replace any I’s with you’s that way if the therapist is responding they say you instead of I. And then it simply responds with why do you feel in the feeling that we extracted. So it extracted happy today and so it said why do you feel happy today? And I can say something like my boss gave me good remarks. And then it says, “Tell me more about your boss.” And so in the code, we have a matcher for my. So if the user’s message contains my, then they’re talking about some subject. We extract out that subject and then respond with, “Tell me more about your subject.” In this case, my the subject is boss. So as you can see, the code is super straightforward, but if you didn’t know what was going on behind the scenes, you might think that there was something smart on the other side. And beyond these matchers, we also have catchall responses. So these are just random continuations like please go on tell me more about that. How does that make you feel? So if I say something that we didn’t match against like he is cool and is a breakdancer it will respond with please go on. So we didn’t have any direct matchers for that particular sentence it picked a random continuation and now the user can keep chatting with the bot. So with just a few if statements we can make the user feel like there’s something smart on the other side. And that’s the basics of this chatbot. So we built a super simple chatbot. It’s just a bunch of if statements, but the UI that the user is interacting with is almost exactly the same as chatbt or cloud. I mean, it is right. It has an input box. The user types, it gets back some response. And I like to kind of abstract this and think about it as as a black box, right? It has some input, performs some process, and then gives some output. In this case, the process is just run the input through a bunch of if statements. But what’s interesting about this is we can actually replace that box with an LLM, right? prompt goes in, LLM does something, answer comes out. And that’s essentially what we’re going to do throughout this video. We’re going to slowly replace the pieces to get smarter and smarter machinery. And I actually like to think about the world of computing and technology in this way. Everything initially is a black box, but we can start to uncover how that thing works. And if we generalize it as inputs and outputs, we can even replace the black box with smarter machinery later on. Now, before you can prompt an LLM, that LLM needs to be created, right? When you type a question into chatgbt, that gets sent to a pre-trained model. So that training process has to happen ahead of time. And essentially that training process produces a model file that then lives on a server somewhere. And that’s actually what we’re interacting with. And I’m already getting ahead of myself because first we need to talk about what is a model. Now LLM stands for large language model. It’s actually crazy that I haven’t defined that this far into the video, but it has the word model in there. And a model is a neural network which is essentially a system of interconnected nodes called neurons that are organized into layers. And data flows through the input layer, passes through one or more hidden layers, and comes out through the output layer. And each connection has a weight. It’s essentially a number that controls how much influence one neuron has on the next. The concept is fairly simple, but the history is wild. And it starts back in 1943. Warren McCullik was a neurohysiologist and Walter Pittz, a self-taught logician, published a paper proposing that the brain’s neurons could be modeled as simple onoff logic gates. Connect enough of them in the right configuration and they could in theory compute anything. Now, this was just a mathematical thought experiment. Nobody could actually build it yet. 15 years later, in 1958, a psychologist named Frank Rosenblat actually built one, and he called it the perceptron. It was a physical machine, a roomsized contraption of wires and motors and photo cells that could be shown images on cards and learn to classify them. The Navy held a press conference. The New York Times reported it as the embryo of a computer that would be conscious of its existence. Then in 1969, two of the most prominent figures in AI, Marvin Minsky and Seymour Papair published a book called Perceptrons. They proved that a single layer perceptron couldn’t solve certain basic problems. The most famous exclusive or which is given two binary inputs output a one if exactly one of those inputs is one which is trivially simple for a human and mathematically impossible for a single layer perceptron. But multi-layer networks could solve exclusive ore. The problem was nobody knew how to train them effectively. And that breakthrough came in 1986. David Rohart, Jeffrey Hinton, and Ronald Williams published a paper in Nature describing back propagation, which is a method for training multi-layer networks. Essentially, you make a prediction, measure how wrong it is, and then propagate the error signal backward through every layer, adjusting each weight to make the prediction slightly less wrong. And then you repeat this billions of times. Now every neural network you interact with today whether it’s claude or chatbt or gemini is trained with some variant of this back propagation algorithm and so let me show you the code and I’m going to use the exact same problem that killed the perceptron exclusive ore. Okay, so this example here is called exor neural net and it will train a neural network in real time to solve exclusive ore. So if I pass in single layer, this trains a single layer network and you can see that it doesn’t reach a point even after 5,000 iterations where it has the correct output that we’re expecting. But if I pass in multilayer, this does reach a point where given these inputs, we’re getting our expected output here. And you can see that even around 800 iterations, so after 800 iterations, our loss is extremely low. And then our loss just gets lower and lower from there. And then we really reach a point where the loss isn’t much more after 5,000 iterations. So after all of those iterations, we have weight values for our neural network that given two inputs either false and false, false, true, true, false, or true and true, we get the expected outputs. And you can see this isn’t perfect, but we do round these values, right? This rounds down to zero. This rounds up to one, up to one, down to zero. Now, the main thing that I want you to see in the code is the fact that after we train these neural networks, it actually creates a file in the data folder that contains the weights for that neural network. And so when we talk about a model, it’s literally just a file with weight values inside of it. So for the single layer network, we’re connecting those two inputs to one single output. So we need a connection from the first input to the output and the second input to the output. And that connection has a weight and that’s why we see two weight values here. And then the final calculation that happens on the output node also includes a bias value. So the single layer neural network is literally just three numbers. If we look at the multi-layer weights, we can see it gets a little more complex, but essentially this neural network has four neurons in that hidden layer. And we need to connect each of the inputs to each of those four neurons. So that’s why we see two arrays of length four. The first array are the weight values for that first input because that first input needs to be connected to each one of those neurons. So we have four connections and then the second input needs to be connected to those same four neurons. So we have four more weight values and then of course four bias values for each of the neurons. Now those four neurons in the hidden layer need to be connected to one single output. And that’s why we see one last set of four weights because the four neurons will have one connection each. That’s four total to the output plus the bias value. So for this very simple neural network, it’s just a bunch of weight values in the model file. And of course this is very trivial, but every single neural network works in this exact same way. Every neural network you come across is going to have a weight file, a model file that has all of the weights that were calculated after that back propagation training. Now, if we take a look at the code, we have our inputs and then our target outputs. And so, every neural network is going to have a certain number of inputs, certain number of outputs. Our neural network has two inputs, either false, false, false, true, true, false, or true, true. And then one single output. And the expected output is false and false is false. False and true is true. True and false is true. And true and true is false. But every neural network you’re always going to need what are your inputs and then what are your expected outputs given those inputs. Now we have our train single layer function. And this takes in the number of epochs. You saw in the output we have 5,000 total epochs and that’s basically just what we’re defaulting to here. And essentially that’s the number of iterations where we will calculate what are our current outputs, figure out how far we are off from our expected outputs, adjust the weights and repeat. But that’s the total number of iterations. In this example, we’re always doing 5,000 iterations. You could also set this up to just keep going until your total loss is within a certain threshold. But this always just does 5,000 iterations. You can see that for those two weight values that we’re attempting to calculate, they start off as a completely random value. And basically for each iteration we calculate the overall value using those weights and the inputs and the bias and then we have a given output for that calculation. We then take a look at our target. So what is the value we’re looking for? We calculate our error and then we use calculus. We use the the derivative to figure out the delta how far we are off from our expected output. And then we adjust all of our weights and bias accordingly. Finally, after 5,000 iterations, we will have some calculated weight and bias values and we can determine what our current loss is overall. Now, train multilayer is very similar except for the fact that we have to come up with random weight values for every connection. The iteration code is very similar except we just have to work up through two layers. So, first we do from the input to the hidden layer and then we do from the hidden layer to the output. So, the code is very similar. Now, we just have two different deltas, right? we have the output delta that is how far we are off from the output to our expected values and then the hidden delta how far we are off from that hidden layer. So from there we update all of the weights accordingly and then repeat. So really the code from single layer to multi-layer isn’t that much more complex. It’s really just calculating between those layers and we can add any number of hidden layers we want. The code will be very similar. And so that’s the basics of a neural network. You set up your inputs, you set up your outputs, set up your hidden layers, and then just start iterating to adjust those weights until you reach a point where those weights give you the expected output given these inputs. Okay, so we’ve got the basics of a neural network. And as we showed, neural networks have inputs and outputs. And so essentially, the prompt [music] that you type into your LLM is going to be an input into that neural network. But the neural network doesn’t just accept a block of text or doesn’t just accept your question. That block of text needs to be broken down so that it could basically be turned into inputs for that neural network. And the first step in that process is known as tokenization. Now, a model breaks your prompt into pieces, not words, not characters. It’s something in between. A token is the smallest unit of text a language model works with. The word ‘the’ is one token. The word tokenization might be split into token and isization, two tokens. A space is often part of a token. A new line is a token. An emoji might be multiple tokens. The model doesn’t see words the way you do. It sees tokens. And how does it decide where to split? It uses an algorithm called BPE or bite pair encoding. And BPE was invented in 1994 by Philip Gage as a data compression technique. It has nothing to do with language models. And the idea was simple. Look at a sequence of bytes, find the pair that appears most frequently, replace it with a new symbol, and repeat. Essentially, it compresses data by finding common patterns. Now, 21 years later, in 2015, researchers Rico Senrik, Barry Hadau, and Alexander Burch at the University of Edinburgh realized this same algorithm was perfect for building vocabularies for neural machine translation. Instead of deciding in advance what all of the words are or what your vocabulary is, you let BPE learn a vocabulary by iteratively merging the most common character pairs in your training data. So common words like ‘the’ stay whole and then rare words get split into subword pieces. And the beauty of this is that one algorithm handles English, Japanese, Python code, TypeScript code, emojis, all without language specific rules. And tokenization isn’t just a pre-processing detail. It has real consequences, right? Token count determines cost. Every API call is priced per token. Token count determines what fits into the context. Every model has a maximum context window which is measured in tokens and if you exceed it something’s going to get cut. So let’s take a look at the code for the BPE tokenization. All right. So this demo is called the basic tokenizer and essentially you can drop in some text [music] and then it will show you how it essentially broke that training text down into tokens. So with the cat sat on the mat you can see at each merge the token pairs that it came across. And then finally we get our overall vocabulary. And so this came across six unique tokens total. And the main idea with BPE is this pair merging. The most frequent pairs are merged. And of course with a very simple training text, right? It’s literally one sentence. It’s going to find each word as a as a unique pair. But in the code, we actually can specify how many merges to do maximum. But if we change this to something like three and train it on the same small bit of text, you’ll see that it actually finds a as a unique token because at appears multiple times in the trading data. So cat, sat, and matt. Now because the word ‘the’ appeared twice, it actually got its own token and then everything else were just extra letters added on. So those appeared as individual letter tokens. So if we dive into the code, one of the first things to look at is this regular expression. And every tokenization algorithm first runs all of the training data through this regular expression to split it up because BPE never merges across word boundaries. So even before we start doing these merges, we need to define ahead of time like what are the whole groups we’re going to use to actually find the individual merged elements within them. And in this case, our groups are actually words. Um but if you look at the algorithm for GPT2 or GPT4, they have a predefined really complex regular expression because they’ve defined some rules up front of how to split things ahead of time. But our initial step here is to just split on whole words. So the things that we’re going to be looking at to actually merge are the individual words by splitting on spaces essentially. Now the next step is actually an optimization and that is we determine the frequency of all of those words. So our regular expression splits on spaces and then every single unique piece of text in there we count the number of occurrences and the number of occurrences is basically going to give us a weight as to how much we actually care about that token in the training data. So in the simple sentence, the cat sat on the mat, ‘the’ appears twice. So it’s going to have a higher weight than some of the other words that we’re looking at. And then we get into our actual training algorithm. Now, from there, we’re going to iterate up to our max number of merges. And like I showed earlier, we have this set to 10,000. But you essentially get to decide ahead of time how much iteration you want to do, and that’s going to determine how large your vocabulary actually gets. And with a really large max merge size, we’re more likely to find all the unique tokens in in a given training set. And then we have the bulk of the work. So this essentially looks at every character pair to find the most common occurring ones and also takes into account the weight. So how often that particular word occurs. And so we find the most common occurring pair that has the highest weight and that becomes a new piece that we’re going to then merge on in the next iteration. So we take that best pair that we’ve found, merge all of the groups, in this case merge all of the words together and then repeat to find the next most commonly occurring pair. So to see a more interesting example, I’m going to plug the B movie script into this tokenizer. And uh we can see it work to actually do all of the merges and find all of the unique tokens in the B movie script. So after the training, this found 2,88 unique tokens, which essentially is all of the unique words in the B movie script. And so you can see it found all the whole words. But if we do reduce our max merges to something like a thousand and then try this again, we are going to see tokens in our vocabulary that are essentially broken up words like you can see the word according got broken up into four tokens because we only have a certain to token budget size and the word according didn’t appear that many times in the overall B movie script. So that’s the basics of tokenizing. Okay, so we’ve analyzed a large data set, extracted out all the possible tokens using this BPE algorithm, and that gives us a vocabulary where each token in the vocabulary has a numeric ID. Essentially, it’s the index of that token in the vocabulary. So these IDs are just arbitrary numbers and they don’t tell us anything about the actual meaning of those tokens or how they relate to each other. So we need to turn those tokens into something the model can actually reason about. And the idea behind that goes all the way back to 1884 when logician and philosopher Goatlab Frige coined the context principle which states never to ask for the meaning of a word in isolation but only in the context of a proposition. 70 years later in 1957 a British linguist named Jr. FTH put it more memorably. He wrote, “You shall know a word by the company it keeps.” And today we call this idea distributional semantics. And we’ve built it into the machines. Every modern language model begins by mapping words into a vast numerical space where neighbors share meaning. And those maps are called embeddings. And that one sentence, you shall know a word by the company it keeps, is the thesis behind every embedding model ever built. Now, in 2013, Thomas Mikolov and his team at Google built off of these ideas in their paper, Wordtoveck. They trained a neural network, an embedding model, to predict words from their neighbors in large amounts of text. It was a simple setup, but when they looked at the vectors the model produced, they discovered structure that no one had taught it. For instance, take the vector for the word king, subtract the vector for the word man, add the vector for the word woman, and the closest result is queen. Now, nobody told the model about gender or royalty. These concepts just emerged purely from the statistics of which words appear near other words. And this was essentially FTH’s idea, you know, a word by the company it keeps implemented as math. Now, what is a vector? Essentially, it’s a list of numbers that identifies a point in highdimensional space. The simplest version is a vector in 3D space or three numbers X, Y, and Z. However, embedding vectors are much larger. The wordtovec paper used 300 dimensions, and GPT3’s largest model uses 12,288 dimensions. Each number captures some feature the model learned during training, not something a human named, but together they encode meaning as a position in space. Words with similar meanings end up as nearby points. For instance, happy and joyful are close together, and happy and refrigerator are far apart. Now, we can measure exactly how close two vectors are using a formula called cosine similarity, which essentially looks at the angle between two vectors. A score of one means they’re identical. A score of zero means they’re completely unrelated. Now, modern LLM embeddings are direct descendants of wordtovect, just scaled up from individual words to entire context. So, let’s take a look at how to train a simple word tovec model. All right, this demo is called train embeddings and you can pass in a couple of words and it will train an embedding model in real time and then show you the comparison of the generated vectors between these words that you pass in. And you can see for each of the words that we passed in, we see the actual generated vector. So these are the embeddings that are generated for each of these tokens. But the cool thing to see with this demo are the analogies that we actually get based on this training set. So we have the the classic word math of king minus man plus woman is queen. But it also works in reverse. So queen minus woman plus man gives us king. And then we have a few other interesting examples. So we have the same royalty where prince minus boy plus girl is princess. This is a fun one. kitten minus cat plus dog gives us puppy. And so this is really cool to see that even with our small training set, we’re able to get some really interesting word maths that actually make sense to us as humans that understand language. Now, the main thing you need when you’re training a model like this is a corpus, which is all of the text that you’re training the model on. So here we have a list of a little over a hundred sentences, and they’re all just statements. So the cat sat on the mat. The kitten is a baby cat. She loves her pet cat. And then there’s an entire section for royalties. And again, the thing to note with this training data is we’re showing it those relationships by putting both king and queen in similar context. So king is a man who rules. Queen is a woman who rules. A prince is a young man of royal blood. A princess is a young woman of royal blood. So by having those statements and just swapping out the words, the model will learn that princess is associated with woman and young and prince is associated with young and man. So there’s plenty more examples that show us king, queen, princess, prince in context. So the model can actually learn those relationships. Now let’s look at the actual training. And in the wordtovec paper they actually talk about two different architectures. There’s the skipgram gram architecture and then there’s also seba which is continuous bag of words. We implemented the skipgram gram architecture. Now the first step in training skipgram gram is to create pairs of words from the training data. So you can see here we have this variable called window size and you can configure this however you’d like but we set it to something sensible like five or six. And essentially every word gets paired with every other word that’s five or six whatever your window size is away from that given word. So for example in the training data take the word king we’re going to create pairs of king is king u king man king who king rules and then we do that with every other word too. So we have man a man is man king man who man rules. So we create all of these pairs and that’s then going to allow the model to learn those groupings and see that oh fairly often in the training data when I see king it’s very often paired with man or when I see queen it’s very often paired with woman or when I see king or queen it’s very often paired with kingdom. So this is the very first step we build up all of those pairs across all of the training data. So you can see in the run we had 107 sentences and this created 3,970 pairs of words that we’ll train on. Now the other thing to see is the actual weights file. So just like with the exclusive or neural net, we have a file that contains all of the weights. And the cool thing about word detovec is it’s one of the simplest neural networks where essentially our embeddings is just a list of every word in our vocabulary and then the vector for that given word. So the first step was to tokenize that input training text. We learned about tokenization in the last section. So you tokenize it and then we create a vector which is just an array of numbers for every single one of those tokens. So when you look at this embedding weights file, it’s literally the entire vocabulary and for every vocabulary word, we have a vector array. So just like when we were training exclusive ore, we need some initial random values for those vectors. So you’ll see we’re creating that randomized array for our vocabulary size times the dimension. And in this case, we did a dimension of 32. So every single one of these vectors is of length 32. Now of course you can make that dimension whatever you want. The more dimensions the more information that it’s going to store. But for this we have 32 dimensions. So that means for every word in our vocabulary we’re going to start off with an array of length 32 filled with random values. And then we have the actual training loop. So for each epoch we’re going to take all of our pairs of words. And our first step is to push the target and the context closer together. So all of the pairs of words that appear next to each other are going to be somehow related. So our back propagation in a sense here, our way of nudging these values is to say if those two words are close together, we’re going to push their vectors closer together. And then we also do the opposite. So for every vocabulary word that never appears next to that given word in our training data set, we push those vectors further away. So we kind of have this push and pull on each epoch where all of the words that are related, their vectors start to get closer and closer together and all of the words that are unrelated, that is, they never appear next to each other, get pushed further and further apart. So after the model has been trained, we have vectors for every word in our vocabulary and now we can start to do maths with it. So we have all of the analogies that you saw in the web UI. We have those set up here. And so in order to do that math, we need to look up the vector for any one of those words. And so essentially once we’ve trained the model, all of the vectors live here. And then when we’re doing inference, when we’re just comparing the words or doing math with them, we can literally just look up their vector values. So this code here says for each of those three words we’re about to do the maths on, pull their vector value out of that weights file and then just do the math. So take that vector minus that vector, add that vector, and then that gives us our resulting vector. Now, the resulting vector isn’t going to be perfect. It’s just a vector. But we then compare that vector to every other word in our vocabulary, and we choose the one that’s the closest. So that’s why you see like with queen that was the closest vector in our vocabulary to the result of performing that maths. So that’s it for training embeddings. Now this is a very simple embedding trainer. The data set is really small and it only creates vectors for individual words. Now modern embedding models have massive data sets. Literally every text written by humans in the entire history of humans. And instead of just embedding the vocabulary, they also embed statements and essentially context from all of that training data. So you basically take this exact same concept and then just scale it up to whole sentences or paragraphs instead of just individual vocabulary words. So we’ve got tokens, we’ve got embeddings, but what actually processes them? The answer is a transformer. [music] And every major LLM that you might use today, whether it’s GPT or Claude or Gemini, are all a form of a transformer. And this all started with a problem. By 2016, Google’s translation models were built on a type of neural network called an LSTM, short for long, short-term memory. LSTMs read text one word at a time, carrying a running memory of what they’d seen so far. They worked, but they were slow. Each word had to wait for the previous one to finish processing. To help with longer sentences, researchers had bolted on an add-on called attention. Now, attention came from a 2014 paper by Dimmitri Bodnau, Kungan Cho, and Yosua Benjio. Older translation models compressed a whole sentence into one fixed-sized vector before translating. Short sentences were fine, but long ones broke. And Bodnau’s idea was let the model look back at the input at each step and focus on the words that matter right now. And it worked, but it was still bolted on to the LSTM. So, it was still slow. Now in 2017, Yakab Yuskarite at Google proposed dropping the sequential part entirely using only attention and eight researchers built it all equal contributors and they titled the paper attention is all you need. Now that paper has been cited over a 100,000 times and every one of those eight authors has left Google and several of them founded billion-dollar companies. Now the architecture they described is the engine running under everything. And the crazy thing is like that paper is only 15 pages long and it’s available for free for anyone to read. So the blueprint for the technology powering a trillion dollar industry is literally just a PDF that you can sit down and read right now. Now underneath everything, a transformer is a neural network, but it’s a very specific kind. And at the hardware level, it’s almost entirely matrix multiplication. The intelligence isn’t any clever logic. It’s in the billions of numbers inside those matrices. All of which started random and got tuned during training to produce useful outputs. Each transformer block has two main operations. Attention where tokens exchange information with each other and a feed forward step where each token gets processed on its own. Both have their own learned weights and both are trained in the same way by running predictions, measuring error and nudging every weight in the direction that reduces it. Now if you zoom in on a block, self attention is the core operation. For each token, the model generates three vectors from its embedding. A query, a key, and a value. Think of the query as what this token is looking for. The key is what each token offers, and the value is what gets passed along when there’s a match. The model compares every query against every key, scores the matches, and mixes the values accordingly. The output is a new vector for each token, the same shape as the input, but now carrying information from the rest of the sentence. So the word bank in the sentence bank of the river was muddy ends up with a vector shaped mostly by river and muddy. Whereas the word bank in the sentence I went to the bank to deposit money ends up shaped by deposit and money. So the same word in different vector out meaning resolved by context. Then multi head attention runs that operation many times in parallel with a different set of query key and value weights each time. Each run is a head and nobody tells the head what to focus on. They start random and during training they end up specializing. One head might learn to track which pronoun refers to which noun. Another might track verb tenses. Another might track position. It’s emergent and not designed. [music] And those outputs get combined back into a single vector per token. And feed forward layers come next. Each token’s vector, now contextenriched from attention, passes through a small two-layer network. The feed forward step processes each token on its own, refining the signal before it moves on. And finally, you stack these layers. So attention plus feed forward is one transformer block or one layer. And modern LLMs stack dozens or hundreds of these layers. And every block is the same input and output shape, one vector per token. Each layer builds on the last. So early layers tend to capture syntax, middle layers capture meaning, late layers capture reasoning. And that’s what deep learning means. It’s many stacked layers. So that’s the whole architecture. Tokens come in, get embedded, get passed through stacks of attention and feed forward and a prediction comes out the other end. Everything else, whether it’s GPT, cloud or Gemini, is this same recipe just scaled up. So let’s take a look at how I implemented this to train my own simple transformer model. Okay, this next demo is called train transformer and the arguments we pass in are the number of epochs and then we have arguments here for the actual generation of the next token. So that’s temperature and top P which we’ll talk about later and then you have the number of layers in that transformer and then the number of tokens to predict. So transformers only predict one token but we’ll talk about in the next section how we can predict multiple. And we essentially created a transformer that can write simple children’s stories. So if I run this, this will actually use a cached model. Uh because I did train this and training this on my MacBook with about 14 cores took over an hour and a half. So I’m not training on GPUs, just just my CPU. So even training this tiny model took a very long time. But as you can see, our input text is once upon a time. That’s what we start with. And then we ask the transformer for the next token and it gave us a and then we ask it for another token and it gave us small. And so if we play around with the temperature and top PE, we’ll see that it predicts different next tokens. So once upon a time a young. So based on the training data, which I’ll show you next, there are different once upon a times in different characters. So you can see that with different values here, it’s producing a different next token. Now the corpus for this training was a little over 30 short children’s stories. So once upon a time there was a king. A young prince lived in a castle. There was once a wicked old man. So we have 30 different short stories. They’re about four or five sentences each. And this is what we actually train the transformer on. So it’s learning the relationships between characters and place and setting. And that’s going to allow it to predict the next word when we’re writing out our own stories. And the actual output of our transformer isn’t a token. It’s actually a probability distribution. So we essentially have our vocabulary and for every single token in our vocabulary, the model outputs what is the probability that any one of those tokens will be the next token given the input prompt. And that’s all the output is. It’s not a single token. It’s a probability distribution. And then using temperature and top P, we actually choose which one of those tokens will be the next token. And as we’ll talk about in the next section, we basically feed that next token back in. And that’s how we can get it to generate more than one token. Now let’s take a look at the actual transformer weights file. And you can see here we have our vocabulary size. So that’s what words did this transformer learn based on our training data. And then we have our context length. And you can see this is tiny tiny tiny compared to the models that you talk to on a daily basis. But essentially this transformer has been trained to look at a sequence of 32 tokens and predict what comes next based on those sequences of 32 tokens. And just like we saw with the embedding model, we actually have the vocabulary stored inside of this weights file as well as the merges which actually allows us to tokenize any new incoming input text. Now we have a separate embedding model for this transformer and it has embedding weights as well as positional weights. So the input and output to this transformer is literally the list of every possible token. That’s the input and then the output is the list of every possible token. But our prompt isn’t every possible token. It’s just a few words. So this positional embedding essentially allows the model to learn, okay, the the input here isn’t everything. It’s just these few tokens in this given order. So we actually train this positional embedding so that given a user’s prompt, it turns it into a list of every possible token, but with the weights tweaked in a way that the model knows what order those tokens are in. And then we have a list of weights for every single block in the transformer. So we talked about every block has attention, which then goes through feed forward. That gives us the output which is the probability distribution of which token comes next. But we have multiple blocks. So for every single block we have the weights for that query key and value and the weights for the feed forward network for that given block. And all of these start off random just like they do in all of our other models that we’ve trained, but we tweak them as we train the transformer. And every block has that exact same setup of weights for query key and value plus weights for the feed forward network. Now, given that our model has an embedding size of 32 and there are six layers, this model we created actually has 52,000 parameters, that is 52,000 different values that got tweaked. And you might contrast that with uh GPTOSS 120B, which in the name has 120 billion different parameters in the model itself. So, this is peanuts compared to some of even some of these local models. And even those local models are peanuts compared to the the models that exist on OpenAI’s servers and anthropic servers. So this is a 52,000 parameter model that we created here. Now diving into the actual code for the transformer. For each block, it first needs to go through and compute the query, the key and the value. So we do that here with matrix multiplication. So given those input vectors, we create the query key and value matrices and then we perform the dotproduct. So we do the dotproduct of query times key and that’ll give us the value which essentially gives us the scores that corresponds to the overall attention of any given token to any other token in the input prompt and then those scores will give us weights and that gives us the overall attention values which we again do matrix multiplication with which gives us the attention matrix for output. Now I’m not going to pretend like I understand all the intricacies of all this because it’s a lot of heavy maths. Um, but if you think about the diagrams from the the last section, this is roughly how the code corresponds. And ultimately, I’m showing it to you because it’s not magic. It’s just math. It’s just numbers mixing them around. And some really smart people figured out how to mix those numbers around in the right way. So from this one pass with the query key and value, we have the attention matrix. Now we can compute the feed forward based on that attention matrix. So we take the input matrix which is the user’s prompt plus positional information add that to the attention matrix and then pass that through our feed forward network and that is just simple matrix multiplication to give us our overall output. Now the way all of this comes together is what I just showed you happens for one specific block. So given input and positional information pass that through a block that then gives us an output and then we can pass that through to the next block. So, however many blocks we have, it’s going to pass it through, perform the same each time, but the input and output size is always the same. We’re just passing it from one block onto the next. And then we end up with our final list of probabilities. So that is given these input tokens and positional information, what is the probability for every single token in our vocabulary that it will actually come next based on the data that we trained on. Okay, so we just saw that the actual output of a transformer is a probability distribution, which is essentially a score for every token in the vocabulary. And those raw scores are called logits. Now to actually turn those logits into probabilities, we use an algorithm called softmax. And this makes it so that all of the probability scores across every single token add up to one. From there, the model doesn’t pick the most probable top token. It samples. And as we saw in our example, there was a temperature variable. A low temperature results in a very predictable output distribution. And a high temperature value results in a more creative, less predictable distribution. For instance, if we pass a temperature of zero, the transformer would predict the exact same token every single time. And then a higher value like 1.5 will be much more random and sometimes incoherent. And then our other parameter is top p. And this is essentially a cutoff. So only sample tokens covering a certain percentage of the probability distribution. So a value of one means give every possible token a chance. A value of 0.2 two means only the top 20% of tokens are considered. It essentially shrinks down the possible list of next tokens [music] and those two parameters are how we can get different outputs from the exact same input prompt. Now everything we’ve walked through so far tokenization embeddings the transformer layer softmax sampling produces one single token. To get the next token we append the token to the input and then run this entire process again. So we get new attention, new distribution, new predicted token, append, repeat. And this whole process is known as auto reggressive generation. Essentially, the model has no plan. It doesn’t know how the sentence ends when it starts. So every coherent paragraph from an LLM emerges one blind step at a time. Now, when you send your prompt off to a production LLM like ChatGpt or Claude or Gemini, it doesn’t see just your prompt. It actually sees a structured package that includes the system prompt which is essentially hidden instructions written by the AI provider themselves. Things like you’re a useful chatbot and it also contains [music] the full conversation history. So every single message that you’ve sent in that chat so far and then finally your latest prompt or your latest message at the end. So the model has zero memory between requests. The conversation history is the actual memory of that conversation and that’s what feeds into the auto reggressive loop. So with that in mind, your prompt in the history and everything that’s gone into your current conversation has to fit into a specific token limit known as the context window. And we saw in our simple transformer example, it had a context window of 32 tokens. But modern models have context windows anywhere from 100,000 to millions of tokens. And essentially the context window is all the transformer can see when it’s producing that next token. So that’s why when you have a long conversation, the answers from an LLM might start to degrade or you might actually start to see hallucinations because with a longer context, there’s more things the model [music] needs to look at when predicting the next token. So it might ignore previous instructions because maybe later messages have more weight in the overall attention of your chat history. Now, at this point, we really have the full picture. Your prompt gets broken down into tokens. Those tokens are according to some vocabulary that was predefined based on the model’s training data ahead of time. Then those tokens get turned into embeddings which actually give the overall prompt meaning. Then the embedded tokens are passed through the transformer. Attention gathers context from all of those tokens and passes it through multiple blocks. And finally a probability score comes out the other end which is given all possible tokens in the model’s vocabulary. What’s the probability that any one of those tokens will be the next token? We then do sampling using temperature and top P to pick that next token. And then we append that token to the prompt and repeat in a loop. Now, everything we showed in this video is actually just the pre-training phase of a transformer. Essentially, we train a transformer on a massive chunk of the internet. And when I say the internet, I mean it. One of the biggest known data sets is called Common Crawl, which is a nonprofit that’s been archiving the web since 2008. And it has pabytes of raw text. On top of that, there’s Wikipedia, Reddit, GitHub, digitized books, academic papers, news archives, and a lot of copyrighted material that AI companies won’t necessarily admit that they actually trained on. But all of this is what transformers like ChatgPTC cloud and Gemini are trained on. Now, the result of this training is essentially just a really fancy autocomplete, which is basically what we built in this video. Now what really takes this model to the next stage of being more useful as a proficient chatbot is known as fine-tuning. And tuning is a secondary phase where we take an existing model and then pass it through a new set of training data. So our new data set is a massive refined list of question answer pairs in the style of what we would like the chatbot to actually respond in. We then slowly fine-tune the weights within the model so that its output actually gives us more of a chatbot-like response rather than just a fancy autocomplete. And then finally, that fine-tune model goes through one more step of tuning known as RHF or reinforcement learning from human feedback. Essentially, real humans are hired to sit down and rank the responses of the model. Good chatbot-like responses get ranked high. Bad responses get ranked low. Now, what defines good and bad is entirely up to the company that trains the model and also up to the judgment of those human rankers. Now, the result of all of this is just a model that behaves like a chatbot, right? It answers questions. It’s able to have back and forth conversations, but it’s still just a standalone model, right? It’s it’s a weights file. It just lives on a server somewhere. It doesn’t know any more than it’s actually been trained on, and it can’t reach outside of itself. It it can only produce next tokens. So this is where the idea of tools come in. You can essentially include a list of available tools and how to call them in the context of your request. Things like get the current weather or search the web for the specific term or run this code file or search the files on your file system. So these tool descriptions essentially tell the model what they can perform and then the models are further fine-tuned to output tool calls where instead of just outputting a response like a helpful chatbot, it will actually output structured code that says please call this tool. So the model itself isn’t reaching out over the web or anything like that. It’s literally producing a JSON object that says call this tool. And then that’s paired with a harness like a code editor or a desktop app that can see that the output of the model was a tool call. And then the harness can in turn actually do what the tool call said to do like execute code or search the web. The harness will then take the result of that cool tool call and append it to the conversation history. So now the next call to the model has all of that relevant output information in the context so it can produce a more accurate answer. Now there’s many other techniques that are used in modern LLMs to increase the quality of the types of answers that it outputs like mixture of experts and thinking modes. But that’s all we’re going to cover at a high level in this video. So if you’re interested in any of those, let me know down in the comments and we’ll definitely get into it in a future video. Now, this brings us back to the beginning of this 70-year journey where in the summer of 1956, four researchers, John McCarthy, Marvin Minsky, Nathaniel Rochester, and Claude [music] Shannon, gathered about 10 people on the top floor of the Dartmouth math department in handover, New Hampshire. They had a Rockefeller Foundation grant for $7,500 and a plan for 8 weeks of work. They called it the Dartmouth Summer Research Project on artificial intelligence. And that was the first time the term artificial intelligence was ever used. They stated, “We propose that a two-month 10-man study of artificial intelligence will be carried out. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” And they thought one summer would be enough. Now 70 years later, we’re still at it. [music] And a single architecture from a single paper powers products used by billions of people. And we might be one paper away from the next shift because we don’t have to be satisfied with the status quo. The transformer might not be the final architecture. It’s just the architecture of the current moment that was backed by billions of dollars in research funding. And researchers are actively exploring alternatives. The next breakthrough might look nothing like what we covered today. Now, everything I discussed in this video is based on public research. The papers are public. There are even openweight models you can download and run yourself. And like I mentioned, all the code I wrote is linked in the description for you to check out. Plus, there’s plenty of other projects on how to build an LLM from scratch you can check out on GitHub. But personally, as I’ve done this research and and kind of dug more into it, the more I feel like LLMs really just are the most sophisticated pattern matching autocomplete things we’ve ever built. I really don’t think that they have true understanding. Um, and I feel that the the less magic there is, the less handwaving there is in in terms of trying to understand the underlying tech, the closer we’ll get to really understanding how we can create better methods of working with LLMs or or better create better understanding of the statistics of language. And I’ve also thought about how the fact that LLMs are just trained on written text, but human intelligence is more than just text. It’s text is just one form of external communication. It’s not actually how the brain itself runs. So for me, that actually reinforces the idea that LLMs are predicting the next token based on language statistics, not necessarily like replicating what’s happening with human intelligence and what’s actually happening in the brain. So, in my opinion, AI is probably better described as something like alien intelligence rather than artificial intelligence because it’s intelligent, but it’s not necessarily a fake version of human intelligence. It’s something entirely different. So, that’s all I’ve got. Thank you so much for making it to the end of the video. If you have any questions, leave them down below. If I made any mistakes, let me know as well. I’ll use the corrections feature of YouTube and I’ll add it so future viewers of the video will see it. And also, if there’s any topic you want to see me dive deeper into or explore in a future video, let me know as well. All right, see you in the next one.