Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70

I didn't make the connection until now, but Jim Keller was on the Lex Fridman podcast twice [1, 2] and it was awesome. In terms of founders this is pretty great team. [1] ▶ this video [2] link

Jim Keller talks about his impression working with Elon - skip to 1:20:50 of this (effing amazing) interview: ▶ this video Edit: Also Jim talks about Elon’s engineering skills at 50 minutes into this interview: link Disclaimer: I’m not an Elon fan at all.

Loved the episodes with Jim Keller [0] and Brian Kernighan [1] as well. Lex really does a great show and has amazing guests. [0]: ▶ this video [1]: link

I highly suggest viewing one of the Alpha's original designers, Jim Keller, recent chat with MIT Prof Lex Fridman ▶ this video I was stunned by Jim's total command of Computer Architecture, and his deep insights about the subject. I will never again utter the phrase 'modern computer' without requisite awe.

Yeah that's how RISC vs CISC is taught in class, I've heard that same thing. I think it's an outdated paradigm though, if it's not been wrong all along. A CISC chip can still officially support instructions, but deprecate them and implement them very slowly in microcode, as is done with some x86 instructions. And a RISC chip manufacturer might just have used this phrase as a marketing line because designing RISC chips is easier than starting with a monster that has tons of instructions, and they needed some catchy phrase to market their alternative. They then get into the gradual process of making their customers happier one by one by adding special, mostly cold, silicon targeted for special workflows, like in this instance. Ultimately, the instruction set isn't that relevant anyways, what's more relevant is how the microcode can handle speculative execution for common workflows. There's a great podcast/interview from the x86 specification author: ▶ this video

Carmack was actually mentioning he asked Jim Keller about Moore's Law. For more details about Jim's thoughts I recommend this interview ▶ this video

I found this to be an excellent explanation for what is going on in these x86 CPUs that makes them so powerful: ▶ this video (Starting from ~6:30)

I just wanted to say this about Intel vs AMD. Back when Intel was the "underdog" it couldn't compete with Athlon/Opeteron CPUs from AMD with its Pentium IV line, fan boys from both sides continually escalated the armchair warfare on Slashdot and Anandtech forums when Intel finally came out with the Conroe architecture that blew the competition away. I've seen this pendulum swing from one side to the other and back. Putting Intel marketing (awful) and AMD's marketing (awful again...), and their fan bases (toxic) aside, can we acknowledge the fact that working on a computer architecture is an extraordinarily complex task and that requires brilliant people all working together? I'll defer this fascinating topic, which most software engineers and tech enthusiasts are completely and utterly unaware of, to this talk by the legendary chip designer (x86 spec coauthor, A4/A5/Zen/Tesla AI ASIC architect) - Jim Keller and his interview with Lex Fridman: ▶ this video Designing & manufacturing computer chips is hard af, having worked in semiconductor manufacturing for over 12 years (primarily on the backend side of the Fab). Edit: Redacting some specifics

> By 2025, either these models will have hit a wall on diminishing returns and it will take a complete pivot to some other approach to continue to see notable gains, or the products will continue to have improved at compounding rates and access will have become business critical in any industry with a modicum of competition. Is there a single example in AI, or even technology as a whole, where simply continuing to apply one technique has led to compounding growth? Even Moore's Law, according to none other than Jim Keller[1], is more a consequence of thousands of individual innovations that are each quite distinct from others but that build on each other to create this compounding growth we see. There is no similar curve for AI. In this case, GPT-3 (released in 2020) uses the same architecture as GPT-2 (released in 2019), expanded to have ~100x more parameters. It's not hard to see that compounding this process will rapidly hit diminishing returns quickly in terms of time, power consumption, cost of hardware, etc. Honestly, if Google, Amazon and Microsoft didn't see increased computational cost as a financial benefit for their cloud services, people might be willing to admit that GPT-3 is a diminishing return itself: for 100x parameters, is GPT-3 over 100x better than GPT-2? It seems that the big quantum leaps in AI come from new architectures applied in just the right way. CNNs and now transformers (based on multi-head attention) are the ways we've found to scale this thing,

I'm generally against rewrites as they're often shortcuts for new developers that instantly reject a code base as 'too complicated' without understanding it. However, I have had success in rewriting my own code base a number of times. This is a unique situation of a one-man-band that understands the business case, handles support, and every bit that went into every decision of the code base. With this kind of clarity, when you've spent years with the problem domain, you can uniquely rewrite a project to get at the real business case that, now with years of experience, you see what your customers actually wanted you to solve in the first place, or maybe what they evolved to want in the end. Jim Keller advocates rewrites in a similar way as the only way to progress in chip design. I think we could rewrite very basic things in exciting ways with this sort of attitude. For instance, we know a lot more about what we need in a desktop OS, an email client, a search engine, etc. Basic things that have gotten a lot of cruft over the years as they evolved. Taking a fresh look could be rewarding. I guess one could reframe this as 'first principals thinking" but with the caveat that the problem needs to be truly understood. Here's Jim Keller talking about it: ▶ this video

I'm not qualified to truly understand this, but when Lex first interviewed Jim Keller, Jim basically said regarding processor design -- 'yeah, if you just guess the same result as last time, you get a 50% speed increase.' First Interview (where that poorly paraphrased quote resides): ▶ this video Second Interview: link

Jim Keller (cpu designer): "I joke, like, I read books. And people think, 'Oh, you read books'. Well, no, I've read a couple of books a week, for [50] years."[1] Thought you might like. [1] ▶ this video