Refterm Lecture Part 1 - Philosophies of Optimization

Casey Muratori's refterm video series is a great example of how to do application design. It focuses on writing a terminal emulator for Windows, but almost all the concepts apply to nixes as well -- even down to the level of OS primitives like memory mappings to create a ring buffer, etc. He builds a nearly complete terminal that appears to better support unicode and escape sequences than windows-terminal, in drastically less code and with literally orders of magnitude better performance. In particular, I like his focus on experimentation to figure out a reasonable upper bound on performance (so you can measure success/failure), "non-pessimisation" (don't do things extrinsic to the actual task), understanding the actual machine and its capabilities (he doesn't use the term, but sometimes called "mechanical sympathy"), and tactics for isolating "bad code" that you probably have to use (at least to get started). ▶ this video

> Casey Muratori has a real in-depth course on why Python is slow and how to make it fast I don’t think that’s a fair characterisation of the course and that description does it a disservice. The course is not about Python, it’s about programming with performance in mind. The video on Philosophies of Optimisation is a good primer on the tenets of the course. ▶ this video

Anything object oriented is slower than anything procedural. Boxing and unboxing is expensive, running constructors and destructors is expensive, copying objects is expensive, virtual tables are expensive. And stuff isn't aligned greatly in memory, we can expect lots of cache misses and branch mispredictions. The fastest possible way is to have C strings in an array and run a function over it. If there are advantages to OOP, speed is not one of them. Here's a great talk about software speed: ▶ this video

He isn't saying that you should spend time on optimising your code, rather that you should write performant code from the beginning by not giving the cpu more work than it needs to do to perform the task, typically by avoiding OO paradigms, followings things like SOLID, etc. Then if needed, real optimisation can handle the hotspots in your code and is typically something that only a few of an orgs best engineers can do properly, but if everything is slow then its an insurmountable task for these few engineers. If you want to understand what he means with this watch the refterm series. ▶ this video

I'm going to pre-emptively link Casey Muratori's Refterm Lecture Part 1[0], because invariably when this topic comes up people are all "but optimization takes time and we gotta push the new features right now or the market will eat us." Computers are so ludicrously fast these days that in the majority of cases you do not need optimization at all. You do need de-pessimization though. ▶ this video

Agreed, but most people here haven't paid for the rest of the course, and Casey sometimes forgot that here he's addressing a wider audience. For instance he fails to explain why he didn't bother addressing the tail of his unrolled loop. He does in the course, but here he's just assuming it's irrelevant, and doesn't address again potential criticism like "he doesn't even bother to write correct code, look at that lazy unrolling!" . Thankfully there's a free lecture that explains the broad concept with an example. It's this lecture that convinced me to try out his course, where I hope he'll go into more details: ▶ this video

I under the impression that we are building software the wrong way. This video from Casey Muratori helped me point to some reasons: ▶ this video

I like Casey Miratory's "non-pessimization" approach in Philosophies of Optimization [1] [2]. He breaks "making a program go faster" into 3 modes: (1) "optimization" which includes counting expected cycles or memory bandwidth, careful benchmarking, designing custom algorithms, etc which can be fun but is expensive and is only rarely applied; (2) "non-pessimization" which is simply avoiding adding extra unnecessary work for the cpu, which should be done 100% of the time but isn't in the industry today; (3) and "fake optimization" which is shallowly repeating 'optimization tips' and conclusions from unrelated cases of (1) without actually measuring it. [1]: ▶ this video [2]: link

But perfect performance isn't even the benchmark, it's "not ridiculously slow". This is what is meant by "Computers are fast, but you don't know it", you don't even know how ludicrously fast computers are because so much stuff is so insanely slow. They're so fast that, in the vast majority of cases, you don't even need optimization, you just need non-pessimization: ▶ this video

Honestly you don't even have to think about optimization to make your software not horrifically slow on modern hardware, instead you have to think about non-pessimization. Simply put, avoid needless complexity and gratuitous waste in the first place. See Refterm Lecture Part 1 - Philosophies of Optimization by Casey Muratori: ▶ this video

I’ve found this to be an unsuccessful approach in practice. Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify. Customers leave in disgust because the site is slow. No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere. Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing. I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content. Invariably, doing the right thing from the beginning would have been cheaper. Listen to Casey explain it: ▶ this video You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer. Learn your craft properly so you can do more with less — including less developer time!

The fallacy of premature optimization rears its ugly head again. When Knuth wrote that quote, he meant something very specific with the word optimization: low level micro optimizations. Spending a lot of time trying to get every little ounce of performance from every little CPU instruction. High level reasonable design decisions are not "optimizations" in this sense at all. You should think of them as non-pessimization. Choosing Python or another slow language is premature pessimization. You just make your program slower for no reason. So choosing a language based on how programs written in it perform is simply non-pessimization. If you are interested in an expansion on this idea, checkout Casey Muratori's lecture about philosophies of optimization: ▶ this video The article makes claims about software being "fast enough" and not needing further optimization. But what you call "fast enough" is probably 10000 times slower than what it could be. I'm not joking. People who program in slow languages have a really skewed perception about performance. If a page takes 5 seconds to load, they don't see that as a problem. If a server takes 500ms to respond to a request, they don't see that as a problem. This is simply unacceptable.