Transformers, the tech behind LLMs | Deep Learning Chapter 5

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs. It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop. ▶ this video

I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other. ▶ this video This even opens up a more data-based approach to linguistics, where it is also heavily used.

This video explains the direction-encodes-trait topic very well IMO: ▶ this video It’s the first in a series of three that I can very highly recommend. > there's no way that there's a 1-to-1 correspondence between concepts and dimensions. I don’t know about that! Once you go very high dimensional, there is a lot of direction vectors that are almost perfectly perpendicular to each other (meaning they can cleanly encode a trait). Maybe they don’t even need to be perfectly perpendicular, the dot product just needs to be very close to zero.

This is good, but what bade me finally understand the transformer architecture [0] and attention [1], are 3Blue1Brown's videos. 0. ▶ this video 1. link

3Blue1Brown has a fantastic video on GPT that I'd highly recommend called "But what is a GPT? Visual intro to transformers" [1]. You'll need to have some background in mathematical reasoning, but not too much. He does a great job of drawing pictures to show the mathematical transformations that happen under the hood. [1] ▶ this video

3Blue1Brown is really a godsend. I had been trying and failed to get an intuition for the self-attention mechanisms, and also why you want to stack you transformer layers. I went through a lot of material (Key-Query-Value interpretations, SVD of these, etc.) but his comment in chapter 5 video [1] made it click for me: "But you should think of the primary goal of this network that it (the word embedding) flows through as being to enable each one of those vectors to soak up a meaning that's much more rich and specific than what mere individual words could represent". Here is my intuition: Transformers are merely a Convnet for token sequences. In the sense that information is first local and each data means something wrt its immediate neighbors. You extract data locally, and then you dezoom to and redo exactly the same at a higher level. Convnet have been shown to reproduce edge detecting kernels first (vertical edge), then assemble them into basic shapes (vertical+oblique: corner), then bigger shapes (something round), then link shapes together (round+round separated by a horizontal shape with rectangles on top), and then it's a car! [2] Transformers do the same. They assemble meaning among tokens ("ju" + "mps" : a conjugated verb), then assemble these at a higher level ("cat jumps" -> noun+verb), then it's a sentence "the cat jumps over the wall", then a paragraph, then a chapter, then a book. All differentiable so any deviation from something coherent can tweak its knobs high

LLMs are explicitly designed to handle, and also possibly 'learn' from different tokens encoding similar information. I found this video from 3blue1brown very informative: ▶ this video Also, think about how a LLM would handle different languages.

Super cool visualization Found this vid by 3Blue1Brown super helpful for visualizing transformers as well. ▶ this video

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGPT a lot of questions. I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out. [0] link [1] link [2] ▶ this video [3] link

It is regarded to be "the best" book on the topic by many. I found just like what Giles Thomas wrote that the book focuses on the details and how to write the lower level code without providing the big picture. I am personally not very interested in that as these details are likely to change rather quickly while the principles of LLMs and transformers will probably remain relevant for many years. I have been looking for, but failed, to find a good resource that approaches it the way 3blue1brown [1] explains it but then go deeper from there. The blog series from Giles seem to take the book and add the background to the details. [1] ▶ this video

LLMs are completely deterministic. Their fundamental output is a vector representing a probability distribution of the next token given the model weights and context. Given the same inputs an identical output vector will be produced 100% of the time. This fact is relied upon by for example link a lossless compression system that would not work if LLMs were nondeterministic. In practice most LLM systems use this distribution (along with a “temperature” multiplier) to make a weighted random choice among the tokens, giving the illusion of nondeterminism. But there’s no fundamental reason you couldn’t for example always choose the most likely token, yielding totally deterministic output. This is an excellent and accessible series going over how transformer systems work if you want to learn more. ▶ this video

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need link 2/ Transformers from Scratch link 3/ Andrej Karpathy has really good series of intros: link Let's build GPT: from scratch, in code, spelled out. link GPT with Andrej Karpathy: Part 1 link 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning ▶ this video Attention in transformers, visually explained | Chapter 6, Deep Learning link Full 3Blue1Brown Neural Networks playlist link