The spelled-out intro to language modeling: building makemore
Andrej Karpathy2 HN recommenders · recommended in 2 months, 2024–2024
tutorial · intro · evergreen · 2024–2024 on HN
Andrej Karpathy builds a bigram character-level language model from scratch, introducing torch.Tensor and the framework of training, sampling and loss evaluation. It is the first step toward a GPT-style Transformer in his makemore series.
HN recommends it as a long but worthwhile way to demystify how language models work technically.
Up next
16 mo on HNThe spelled-out intro to neural networks and backpropagation: building micrograd
Andrej Karpathy
17 HN recommender · 2022–2025
14 mo on HNBut what is a neural network? | Deep learning chapter 1
3Blue1Brown
16 HN recommender · 2017–2025
10 mo on HNAlphaGo - The Movie | Full award-winning documentary
Google DeepMind
10 HN recommender · 2020–2025
10 mo on HNTransformers, the tech behind LLMs | Deep Learning Chapter 5
3Blue1Brown
13 HN recommender · 2024–2026
10 mo on HNUSENIX Security '18-Q: Why Do Keynote Speakers Keep Suggesting That Improving Security Is Possible?
USENIX
10 HN recommender · 2018–2026
11 mo on HNLecture 1 | Machine Learning (Stanford)
Stanford
11 HN recommender · 2008–2020
10 mo on HNMarI/O - Machine Learning for Video Games
SethBling
10 HN recommender · 2015–2026
8 mo on HNDeep Dive into LLMs like ChatGPT
Andrej Karpathy
10 HN recommender · 2025–2026
10 mo on HNComputer program that learns to play classic NES games
suckerpinch
9 HN recommender · 2015–2025
9 mo on HNRL Course by David Silver - Lecture 1: Introduction to Reinforcement Learning
Google DeepMind
9 HN recommender · 2017–2026
8 mo on HNLet's build GPT: from scratch, in code, spelled out.
Andrej Karpathy
14 HN recommender · 2023–2025
7 mo on HNBay Area Vision Meeting: Unsupervised Feature Learning and Deep Learning
Google TechTalks
9 HN recommender · 2011–2013
4 comments from Hacker News
You've made something interesting. Many people have already answered this here [1], so there's no point in me explaining it again. Dropping some pointers that will be helpful for you: - link - Highly recommend this one ▶ this video - link - link 1: ex: link
I resource I found invaluable in demystifying LLM's was Andrej Karpathy's Youtube series. In particular, his "Makemore" video built a basic LLM in the same vein as OP's. It's long (2h), but well worth the time if anyone is curious about the technical workings of LLMs. ▶ this video
For reference, this project is from Eureka Labs, which is the new education startup from Andrej Karpathy. * link It appears to be a generalized version of the bi-gram project, which he links to here: * ▶ this video It's cool to see him putting in even more effort on the edu work. It's a real gift he has, and was visible all the way back in his grad student projects.
This is beautiful. I'd actually been going down this same rabbit hole with sqlite, I hadn't gotten far enough to bring a neural net into it. I'd been inspired by the makemore lecture series[0]. At the 1hr mark or so, he switches from counting to using a nn, which is about as far as I've gotten. Breaking it down into a relational model is actually a really great exercise. [0] ▶ this video