Deep Dive into LLMs like ChatGPT

Even if you have some basic understanding of how LLMs work, I highly recommend Karpathy’s intro to LLMs videos on YouTube. - ▶ this video - link

I highly recommend Andrej Karpathy's Deep Dive into LLMs like ChatGPT: ▶ this video . It's a 3.5 hour general intro to the topic that I think does a great job of making sense of the tech and how it works under the hood, without needing to build up a foundation of math and programming.

This is like the LLM era version of the search bubble that prevented people from having the same search results for ostensibly identical searches. Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well: [0] ▶ this video

Since you're posting here, you're looking for the shortcut. The shortcut is Karpathy's "Let's Build GPT: from scratch, in code, spelled out" video: link Then there is a good video that dives into LLMs and how they work that is quite approachable: ▶ this video From there, flesh out knowledge with his other videos, where he goes both extremely light and extremely deep: link Anyway, I really like's Karpathy's video because he's very good at explaining LLMs at every level.

Depending on how much time you have, the first thing I would recommend is to watch Karpathy's 'deep dive into LLMs' video ( ▶ this video ). It is 3.5 hours but it will give you so much instinctive understanding of how these things work and will likely save you a lot of time. Purely from a technical standpoint, things aren't that difficult really. I'd recommend get yourself an API account with all major providers (because each frontier LLM evidently has their own strengths and weaknesses) and start experimenting with your data. The feeling you'll get when you first get an API response will keep you going! :) Having said that, a caveat is, LLMs do hallucinate from time to time. If your data needs accuracy, definitely don't do the "throw everything at LLMs and let them do the work" but I'm sure you'll figure it out from trial and error. Good luck!

I think you'll get a lot of use from this video from Andrej Karpathy: ▶ this video It is long, but don't get scared off. He goes over a ton of different stuff related to model training, but makes it very easy to understand.

This is a good resource, however for about 99.99% of people, you are most likely to just use a foundation model like ChatGPT, Claude, Gemini etc. so this knowledge/training will get you neither here or there. I would suggest you look into another Karpathy's video -- Deep Dive into LLMs like ChatGPT. ▶ this video

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGPT a lot of questions. I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out. [0] link [1] ▶ this video [2] link [3] link

> The models still _generate_ response token-by-token, but they pick tokens _not_ based on tokens that maximize probabilities at each token. This is also not how base training works. In base training the loss is chosen given a context, which can be gigantic. It's never about just the previous token, it's about a whole response in context. The context could be an entire poem, a play, a worked solution to a programming problem, etc, etc. So you would expect to see the same type of (apparent) higher-level planning from base trained models and indeed you do and can easily verify this by downloading a base model from HF or similar and prompting it to complete a poem. The key differences between base and agentic models are 1) the latter behave like agents, and 2) the latter hallucinate less. But that isn't about planning (you still need planning to hallucinate something). It's more to do with post-base training specifically being about providing positive rewards for things which aren't hallucinations. Changing the way the reward function is computed during RL doesn't produce planning, it simply inclines to model to produce responses that are more like the RL targets. Karpathy has a good intro video on this. ▶ this video In general the nitpicking seems weird. Yes, on a mechanical level, using a model is still about "given this context, what is the next token". No, that doesn't mean that they don't plan, or have higher-level views of the overal structure of their response, or whatever.

There is a very good intro on how LLM work by Kharpaty. It's ▶ this video and it's 3.30 hours, so way below the 100 hours mark :)

I agreed when I read your comment, but that turned out to be almost directly from the video ▶ this video From around the 2:20 mark he says: “[…] actually ends up being only about 44 TB of disk space. You can get a USB stick for like a TB very easily, or I think this could fit on a single hard drive almost today” So it’s just slightly altered from what was said in the original video. And the LLM rewritten version of it also says “roughly” where he said “almost”, and I guess 44 TB is pretty roughly or pretty almost 32 TB. Although I’d still personally probably put it as “can fit on a pair of decently sized hard drives today” (for example across two 24 TB drives). Regardless, it’s close enough to what was said in the source video that it’s not something the LLM just made up out of nowhere.

That's post-training No, it is not. Read the paper. They are discussing an emergent property of the context itself: "For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model." I'm talking about tasks like multiplying two 4-digit numbers (let's say 8-digit, just to be safe, for reasoning models), which 5th or 6th graders in the US are expected to be able to do with no problem - without using a calculator. So am I. See, for example, Karpathy's discussion of native computation: ▶ this video (starts at 1:46:56). The first few tokens in the context actually serve as some sort of substrate for general computation. I don't pretend to understand that, and it may still be something of an open research topic, but it's one more unexpected emergent property of transformers. You'd be crazy to trust that property at this stage -- at the time Karpathy was making the video, he needed to explicitly tell the model to "Use code" if he didn't want it to just make up solutions to more complex problems -- but you'd also be crazy to trust answers from a 5th-grader who just learned long division last week. Feeling quite comfortable being classed in the same camp with the greatest living physicist, honestly. Not a great time for you to rest on your intellectual laurels. Same goes for Penrose.