Let's build GPT: from scratch, in code, spelled out.

+1, Andrey is an amazing educator! I'd also recommend his ▶ this video and checking out his github repos. MinGPT, for example, implements a small gpt model that's compatible with HF API, whereas more modern nanoGPT shows how to use newer features such as flash attention. The quality of every video, every blog post is just so high.

Andrej Karpathy, the director of AI at tesla, published an AMAZING video about how the current models are built which I think everyone in the tech space should watch. It is intuitive, easy to follow and quite frankly the best video I have seen on the topic. Here is the link: ▶ this video If you conclude, after watching this, that this is all intelligence is - namely cascading optimization and data structure problems chained in a row, then you and a fair number of people here wont ever find common ground. That is not to say that you're wrong, it just seems as if there is a missing component here that we haven't yet discovered.

Very impressive. Recently I watched this really amazing lecture on building GPT from scratch from Karpathy, I was blown away: ▶ this video

Great compilation, would be great to see Vision Transformer (ViT) included. Andrej Karpathy's GPT video is a must have companion for this ▶ this video I was going nuts trying to grok Key, Query,Position and Value until Andrej broke it down for me.

Attention is all you need and GPT-2 were well known at that point. Many might doubt whether this approach leads to "general" intelligence – depends on the definition. BTW, Karpathy has a nice video tutorial about building an LLM: ▶ this video

Yes, it's doable. Your model won't be as large and as performant as a real large scale model, but you could train something. You could watch this for a start: ▶ this video

I initially thought this was his older video, but I see this video was published just an hour ago (from the time of this comment). I was thinking of this other video he had published early this year: "Let's build GPT: from scratch, in code, spelled out" ▶ this video Karpathy is generally well-reputed as a good tutor, especially for complex topics in AI / ML.

I love all the series about writing your own X in 100 lines of code. It gives you the understanding of technology and removes a lot of unnecessary details. The great examples of this are 'A from-scratch tour of Bitcoin in Python' link and 'Let's build GPT: from scratch, in code, spelled out' ▶ this video from Andrej Karpathy I wonder if anybody tried to collect all such projects together and built his own 'Internet in just 100 lines of code'

Those Computerphile videos[0] by Rob Miles helped me understand transformers. He specifically references the "Attention is all you need" paper. And for a deeper dive, Andrej Kharpaty has this hands-on video[1] where he builds a transformer from scratch. You can check-out his other videos on NLP as well they are all excellent. [0] link , link [1] ▶ this video

Here are a few resources that might help get you started: 1. A thread explaining the internal working of transformers: link 2. Paper by DeepMind which provides pseudo-code for important algorithms for Transformer models: link 3. Another thread specifically on large language models: link Once again these are not courses per se, but do provide intuitive explanations for how transformers work. There is also the nanoGPT series of videos by Karpathy on youtube. First video here: ▶ this video

I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": ▶ this video Additionally, for more comprehensive resources on Transformers, you may find these resources useful: * The Illustrated Transformer by Jay Alammar: link * MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: link * Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): link ... These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.

Andrej Karpathy's 2 hour video and code is really good to understand the details of Transformers: "Let's build GPT: from scratch, in code, spelled out." ▶ this video