In 1950 Claude Shannon had his wife guess the next letter of a hidden passage and measured how often she was right, proving written language has deep statistical structure. Seventy years later that is still all an LLM does, but “it just predicts the next token” hides an enormous stack of machinery, and CJ builds every layer of it from scratch to show what is really under the hood.
From rule-based chatbots to learned machinery, and what “understanding” means
Conversational machines are old. Turing’s 1950 paper set the target with the imitation game, and by 1966 Weizenbaum’s ELIZA was faking a Rogerian therapist purely through if-statements and pattern matching, with zero understanding of anything. It was convincing enough that Weizenbaum’s own secretary asked to be left alone with it, which he found alarming and which echoes uncannily in how people talk about ChatGPT today. Colby’s PARRY (1972) added internal emotional states and fooled psychiatrists; ALICE (1995) hand-wrote 41,000 patterns; SmarterChild (2001) put a do-everything bot on AOL Instant Messenger. Every one of these was rules, scripts, templates, or decision trees that a human authored in advance. The jump from ELIZA to ChatGPT is therefore not a better chatbot but a completely different mechanism: understanding that is learned from data rather than programmed by hand. The useful mental model throughout the video is the black box, input goes in and output comes out, and progress is just swapping a dumber box (a pile of if-statements) for smarter machinery (a trained network) behind the same chat UI.
A neural network is just a file of weights, trained by backpropagation
A model is a neural network: interconnected neurons in layers, data flowing input to hidden to output, and every connection carrying a weight, a single number saying how much one neuron influences the next. The lineage runs from McCulloch and Pitts modelling neurons as logic gates (1943), to Rosenblatt’s room-sized physical Perceptron (1958), to Minsky and Papert’s 1969 proof that a single-layer perceptron cannot even compute XOR (output 1 when exactly one input is 1). Multi-layer networks could solve XOR, but nobody knew how to train them until Rumelhart, Hinton, and Williams described backpropagation in 1986: make a prediction, measure how wrong it is, push the error backward through every layer nudging each weight slightly less wrong, and repeat billions of times. CJ trains exactly that XOR network live: the single-layer version never converges, the multi-layer one drives loss near zero by roughly 800 iterations. The load-bearing insight is deliberately deflating. After training, the program writes a file of weights, and that is all a model ever is. The single-layer XOR net is literally three numbers (two weights plus a bias); the multi-layer version is a handful of small arrays. Every model you use, from a toy net to Claude, is the same object scaled up: a weights file produced by backprop.
Tokenization and BPE: why token count is cost and context limit
The network takes numbers, not prose, so a prompt is first broken into tokens, the smallest units a model works with, sitting somewhere between characters and words. “the” is one token; “tokenization” might split into “token” + “ization”; spaces, newlines, and emojis all tokenize in their own ways, so the model never sees words the way you do. The splitting rule is Byte Pair Encoding, and its history is a nice accident: Philip Gage invented BPE in 1994 as pure data compression (find the most frequent byte pair, replace it with a new symbol, repeat), and only in 2015 did Sennrich, Haddow, and Birch at Edinburgh realise the same merge loop builds excellent vocabularies for neural machine translation. Rather than fixing a word list in advance, you let BPE learn one by iteratively merging the most common pairs, so frequent words like “the” stay whole while rare words fragment into subword pieces, and a single algorithm handles English, Japanese, Python, and emoji with no language-specific rules. In the demo, capping merges low forces long or rare words to shatter (as “according” splitting into four tokens), which is the whole point: your merge budget sets vocabulary size. Tokenization is not a preprocessing footnote. Token count sets cost because every API call is priced per token, and it sets what fits because the context window is measured in tokens, so exceeding it means something gets cut.
Embeddings: meaning as position in vector space
Token IDs are arbitrary indices that say nothing about meaning, so the model needs to turn them into something it can reason over. The idea is old philosophy: Frege’s 1884 context principle (never ask for a word’s meaning in isolation) and Firth’s 1957 line, “you shall know a word by the company it keeps,” which is the thesis of distributional semantics and of every embedding model since. Mikolov’s Word2Vec (Google, 2013) trained a network to predict words from their neighbours and discovered structure nobody had programmed: take king, subtract man, add woman, and the nearest vector is queen. Gender and royalty emerged purely from co-occurrence statistics, Firth’s idea implemented as arithmetic. A vector is just a list of numbers naming a point in high-dimensional space; where 3D needs three numbers, Word2Vec used 300 dimensions and GPT-3’s largest model uses 12,288, each dimension capturing some learned feature no human named. Similar meanings land near each other (happy and joyful close, happy and refrigerator far), and closeness is measured with cosine similarity, the angle between two vectors, where 1 is identical and 0 is unrelated. CJ’s skip-gram trainer reproduces the magic on a tiny corpus: pairs built with a window of five or six words, related vectors pushed together and unrelated ones pushed apart each epoch, and afterward king − man + woman ≈ queen, kitten − cat + dog ≈ puppy. Modern embeddings are direct descendants of Word2Vec, just scaled from single words to whole sentences and contexts.
The transformer and attention: killing the sequential bottleneck
By 2016 Google’s translators used LSTMs that read one word at a time carrying a running memory, which worked but was slow because each word waited on the last, and long sentences overflowed the single fixed-size vector they were crushed into. Bahdanau, Cho, and Bengio’s 2014 attention mechanism let the model look back over the whole input and focus on the words that matter right now, but it was still bolted onto a sequential LSTM. The 2017 “Attention Is All You Need” (eight equal-contributor authors, since cited over 100,000 times, all eight now gone from Google) dropped the sequential part entirely and kept only attention, and that 15-page free PDF is the blueprint under every major model today. Underneath, a transformer is a neural net that is almost entirely matrix multiplication; the intelligence lives in billions of numbers that started random and got tuned. Each block does two things: self-attention, where tokens exchange information, and a feed-forward step, where each token is refined on its own. In self-attention each token emits a query (what it is looking for), a key (what it offers), and a value (what it passes on); every query is scored against every key and the values are mixed accordingly, so “bank” in “bank of the river was muddy” ends up shaped by river and muddy while “bank” in “went to the bank to deposit money” is shaped by deposit and money, meaning resolved by context. Multi-head attention runs this many times in parallel with separate weights, and the heads specialise on their own during training (one tracking which pronoun binds which noun, another tracking tense or position), emergent rather than designed. Stack attention plus feed-forward into a block, stack dozens or hundreds of blocks with identical input and output shape, and you get depth where early layers capture syntax, middle layers meaning, late layers reasoning; that stacking is what “deep learning” means. CJ’s own transformer that writes children’s stories has an embedding size of 32, six layers, a 32-token context, and just 52,000 parameters (against GPT-OSS-120B’s 120 billion), and took over 90 minutes to train on a MacBook CPU.
From one token to a chatbot: sampling, the loop, and everything after pre-training
A transformer does not emit a token; it emits a probability distribution over the whole vocabulary. Raw scores (logits) are turned into probabilities that sum to one by softmax, then the model samples rather than always taking the top token. Temperature controls spread (0 is deterministic, 1.5 is near-incoherent) and top-p is a cutoff (0.2 keeps only the tokens covering the top 20% of probability mass), and together they yield different outputs from the same prompt. That produces one token; append it to the input and run the whole pipeline again, which is autoregressive generation, so the model has no plan and every coherent paragraph emerges one blind step at a time. In production the model never sees just your message: it sees a structured package of system prompt, full conversation history, then your latest turn, because the model has zero memory between requests and the history is the memory. All of that must fit the context window (32 tokens in the toy model, 100k to millions in real ones), which is why long conversations degrade or hallucinate as later messages outweigh earlier instructions in attention. Everything shown is only pre-training, a fancy autocomplete over a scrape of the web (Common Crawl, Wikipedia, GitHub, books, and undisclosed copyrighted text). Turning that into a useful assistant takes fine-tuning on curated question-answer pairs, then RLHF where paid humans rank responses (with “good” defined by the company), and finally tool use, where the model is fine-tuned to emit a structured JSON tool call that an external harness executes and feeds back into the context. CJ closes on the 1956 Dartmouth workshop that coined “artificial intelligence” and planned to crack it in one summer, and on his own view that LLMs are the most sophisticated pattern-matching autocomplete we have built, better called alien intelligence than artificial, since they model language statistics rather than the brain.
Key takeaways
- A model is nothing more than a file of weights; every network from a 3-number XOR net to a frontier LLM is that same object scaled up, produced by backpropagation.
- Understanding in an LLM is learned from data, not authored in if-statements; that is the entire difference between ELIZA and ChatGPT.
- Tokens are the unit of both money and memory: BPE decides the splits, token count sets cost, and the context window is measured in tokens.
- Embeddings put meaning in space so relationships become arithmetic (king − man + woman ≈ queen), with cosine similarity as the ruler and dimensionality (300 to 12,288) as the capacity dial.
- The transformer’s win was replacing the LSTM’s sequential read with parallel attention (query/key/value, multi-head, stacked blocks); “attention is all you need” is a free 15-page PDF.
- Generation is autoregressive sampling one token at a time via softmax + temperature + top-p; there is no plan, only next-token prediction.
- Pre-training yields autocomplete; fine-tuning, RLHF, and tool-calling harnesses are what make it feel like an assistant.
Sources
- Watch the video: youtube.com/watch?v=YmLp8qe87A0
- Full transcript: i-built-an-llm-from-scratch
- Related: how-llm-tokens-work, retrieval-augmented-generation