Unix and Google

I wrote a non-programmer summary to some of my friends, because my mind is blown. Here it is (warning: it's very informal). This is some next level 80/20 stuff, I'd argue it's even 99/1 (consultant speech for "being efficient" by seeing if you can exploit Pareto's principle [0]). A guy on YT [1] simulates in 7 minutes how the two biggest operating systems were built in the sixties (Multics by IBM and Unix by 2 guys in a garage). Nowadays we only have descendants of Unix, this simulation shows IMO two things: 1. How a chain of command screws up productivity (I doubt that anyone at IBM had the power/autonomy to change complete structure of the product. This was needed, but such change management is insane). 2. How the pipe command in Unix changed the mathematical nature of developing an operating system (in fact, it's even better than 99/1, when you think about it). In normal human language, the difference is: IBM programmed modular blocks. Unix programmed modular blocks. IBM then programmed how those modular blocks should interact. Unix created a pipe command and allowed computer users themselves to implement how these modular blocks should interact. At the time, computer users were all savvy enough like that. Result: when IBM created a feature, it needed to integrate it with all the other features that came before. When those 2 guys in a garage (Unix) created a feature, all they needed to do was create that feature. So mathematically you could say (where each term is the amou

I was going to comment on this very thing in the other thread about software glue[1]. In that article there is a youtube video on Multics vs Unix[2] which really outlines why microservices were always doomed. Someone should coin a new law for how programmers have to rediscover Brooks's law every 5-10 years. The issue with microservices, as it has always been, is that you need an enormous company to brute force the communication pathways and maintenance overhead for it to all work. And by work, I don't mean function efficiently (as the Multics vs Unix video shows). I mean just function . Just work at all. The Multics team had all the devs and the Unix team was two guys doing laps around the Multics team. Because they had the mathematics on their side. Remember the bad old days of memory thrashing? That's what happens to teams that do not have enough bandwidth to properly maintain the dozens of services they are responsible for. Your organization gets frozen. This is what we all get for taking advice from the Googles and Facebooks of the world. Google has like a billion lines of code in a monorepo. They do not do things remotely like 99% of the businesses out there. They are sitting on huge piles of money that lets them be incredibly inefficient for decades . [1] link [2] ▶ this video

Unlike previous OSes, UNIX provided the pipe facility to allow you to combine the functionality of many small programs. Ex. ls | grep ... | sort ... | more Before that every command-line program had to support the full suite of additional functionality like search, filtering, paging, etc. This UNIX/Pipe approach allowed for greater reuse and much less code. This is explained and simulated in this video: ▶ this video

If we analyze the computational complexity of software architectures the way we do regular algorithms, we see that UNIX was much more efficient than previous OSes/approaches. It's demonstrated in this short video: ▶ this video

Programs/functions like left-pad do not follow the unix philosophy (which the author did say). The idea of the unix philosophy is to write less programs. Not write a ton of tiny programs, but a small number of composable programs. So instead of left-pad, right-pad, middle-pad, and anything else to get specific need you can use sprintf which can format a string in general, one program that can do implement the features of all of those. It is about “coding the permitter, not the area”, so if you have NxM features to build, don’t write N*M individual programs/functions, instead write N+M programs/functions that can be composed together to get all NxM features. This video explains it well: ▶ this video

This video explains why UNIX was a moonshot, and why it could be developed by a small number of people because of it: ▶ this video

I made a simulation and video which shows why the pipe is so powerful: ▶ this video I showed it to Doug McIlroy, and while he thought there was more to the story, he didn't disagree with it.

This video shows how Unix was mathematically more efficient than Multics, which goes a very long way at explaining its success and longevity: ▶ this video

Sort of related but an interesting idea is take it one step further with the Feature Oriented Active Modeler (FOAM) [1,2] paradigm and use code to model your whole system, which generates diagrams model, and runnable code in whatever language needed. The project is still young and it may not be practical today with the available tooling but it seems like a cool idea and project. It is influenced by the unix principle of “coding the perimeter not the area” which is essentially factoring your dev tasks into building NxM capabilities, but instead of building NxM things individually build N+M tools that can be composed into NxM capabilities [2]. So with FOAM the idea is if we want to maintain a model of our software, and build it as well, what if we can use one composable tool to generate both, rather than model everything and code it separately. [1] link [2] link [3] ▶ this video

I would say that the best programmers don't change the multiplier, but the exponent, so that the factor by which they're better than the average increases with the size of the problem. This is why you might not see as much of a big difference with a test lasting a few hours, as you would with one lasting weeks, months or years. I explain this a bit by means of a simulation in this video: ▶ this video

> The whole thing was invented by a "software consulting" firm I don't know the whole origin story. It definitely does feel like something Martin Fowler would come up with. But I blame Google for really making it a trend: ▶ this video And you can see they understand the whole problem with microservices. It's the same thing The Mythical Man-Month was trying to tell everyone decades ago[1] > When n people have to communicate among themselves, as n increases, their output decreases Microservices exasperates this. It is the Multics model. Each microservice implements its own, often wildly different, API. Which every bit of code that needs to use that microservice has to go and implement. [1] link

No. pg has a good article on doing things that don't scale[0] and showed how the AirBnB guys went and setup the first listings manually, but it's actually far more general than just kicking off the network effect. Moats are things that don't scale. Open Source software scales perfectly. Billion dollar companies are billion dollar companies because they can use Open Source software wrapped around some Moat-able structure. The more Open Source Software that exists, the bigger the walled garden. When Google spends a billion dollars sending cars all around the world taking pictures, they are doing so in a way that the Open Source world can't centrally compete against, as they don't have the billions to get that done. This means that all Open Source mapping software will be missing a crucial feature that users rely on, and will therefore go unused. But meanwhile the libraries that those Open Source Software use, Google can freely use within their walled garden. On the other hand, Microsoft paid a great deal of money to researchers with Encarta and had this same exact structure and Wikipedia came in and disrupted them. It takes more time for the decentralized approach to come online, but as long as it's possible and the data is kept high quality Google Maps will slowly lose market share to OSM. Open Source software is generally created as libraries, whereas most companies build frameworks. Frameworks are easier to setup and give control to the makers, but they do not scale. Why is