TigerStyle! (Or How To Design Safer Systems in Less Time) by Joran Dirk Greef

I second this recommendation. I'm impressed by what TigerBeetle has accomplished, and would love to see more of the industry approach software engineering with their philosophy and rigour. The TigerStyle video is well worth watching for anyone who wants to build defect-free software systems, or recognizes that this is the correct thing to aim for. ▶ this video

Joran from TigerBeetle here! This is a “walking sim” real visualization of our Deterministic Simulation Testing harness, the VOPR. Typically we do DST headless in the terminal but this is a visualization, more or less the same thing but with a gaming engine on top. It’s running real TigerBeetle code, but with simulated network/storage and fault injection, and multiple replicas in a cluster, all compiled to WASM and running in the browser, and you can interact with the simulation to poke it and watch the consensus recover. Here’s the full background context and explanation from our talk, TigerStyle, for which we developed this sim: ▶ this video See also a shorter explanation: link

You're right! In the past, distributed databases written without Deterministic Simulation Testing (DST) would typically take around ~10 years (~5 years for the consensus protocol and ~5 years for the storage engine). And then not without bugs. However, and we write about this also in the post, but TigerStyle and DST enabled us to ship TigerBeetle to production in 3.5 years, in less time, to a higher standard, and to be the first distributed database with an explicit storage fault model (Kyle Kingsbury added new storage fault injectors to Jepsen) solving Protocol-Aware Recovery. Our Jepsen audit is here (also linked in the post): link For more on TigerStyle as a methodology, hope you enjoy this talk: ▶ this video

We apply this pretty much fully at TigerBeetle [1]. Reason being that TigerStyle is not only about shipping safer software, but shipping it in less time [2], factoring in the Total Cost of Ownership to end users. In other words, not only optimizing our development time, but also all the time spent in production with subsequent fixes. [1] link [2] ▶ this video

Appreciated your post, Lewis! From the beginning, TigerStyle was intended not only for TigerBeetle as infrastructure, but for software (up and down the stack) in general. Nobody wants buggier, slower software. And so TigerStyle as a methodology was codified primarily to address this problem: How to write “higher quality software in less time”? [1] Before designing TigerBeetle, I was finding the principles more and more valuable, regardless of whether I was doing frontend or backend work, and so brought them into TigerBeetle. Of course, software higher up is likely to use some dependencies, but there's always a cost, and this principle (“do more with less”), at least in my experience, tends to prove valuable when applied. [1] Some of the background motivation behind TigerStyle: ▶ this video

> How does TigerBeetle's existence support the argument that rolling your own dependencies is better than using off-the-shelf tools? At least from my own experience, without TigerStyle [0], there's no way we could have shipped TigerBeetle from 0 to production in 3.5 years, with the same quality, and to the same tolerances: link Whereas, for example, if we'd used off-the-shelf dependencies such as Raft or RocksDB, it would have been a nightmare, and taken longer, just to fix them up to solve our storage fault model and implement Protocol-Aware Recovery correctly. Even then, we'd have been stuck with a systems language of the last 30 years (C/C++), when we could have invested in a language for the next 30 and with a more powerful toolchain. Of course, we'd also have been without Deterministic Simulation Testing, because most software dependencies are typically not designed for DST. But finally (and to your point about sustainability), without TigerStyle, we wouldn't have had the company, or the investment, team, and customers we have today, or be as far along financially. Not saying that you need to share our engineering philosophy. But it works for us. [0] ▶ this video

Joran from TigerBeetle here! We didn't design our docs because it was "a fun thing" (as suggested) but rather because we simply care deeply about the experience of developers reading our docs. For example, concerning performance and offline use, which were further reasons we gave in the post. We have a high bar for taking on dependencies. We don't take on dependencies automatically without justification. It's just not a philosophy that we share, to assume or to insist that everything needs to be a dependency. (The discussion on CSP in our post was also not given as motivation, but as an example of the thought process that went into this. Again, as per the post, it's a matter of defense-in-depth. We have plans for how our docs will be used in future, that you may not be aware of, and security is important.) Finally, we're happy with the result, the project was small and didn't take long. We're used to "painting" things like this fairly quickly. It's just easier for us than trying to "sculpt" off the shelf dependencies. That's not to suggest that everyone needs to paint like we do at TigerBeetle, but it's equally true that not everyone needs to sculpt either. [1] [1] To understand our engineering methodology, and why we prefer to paint than sculpt, see TigerStyle: ▶ this video

> Zig will be easy to understand and use but too unsafe to use for important applications This is an outside perspective on Zig, and I have to say, not an informed one. If you'd like to understand Zig (and what I mean) better, this video is a good start: ▶ this video Zig is, right now, being used for high-assurance systems where correctness is a terminal value, and it provides many tools and affordances to assist in doing so. It isn't a good choice for projects which give lip-service to correctness, but for ones which actually mean it, and are willing and able to put in the effort and expense to achieve it, it's an excellent choice. I'm willing to gloss that domain as "important applications", but perhaps you meant something different by that term.

I recall learning TLA+ around 2015/2016 to solve a hard bug in an OpenStack deployment. I was miffed that I was some 17 years into my career and had only learned about it then. It was so useful! Since then I've used it for similar purposes as this. I try to share with my team when I do this kind of work. But often folks are highly resistant to it so I use it on my own when I'm working on a hard project. Spending the time up-front to work out the design is often more cost effective in terms of time and money than iterating in production towards a, "more correct design." Getting it right first sounds hard, and it's challenging, but it's worth it for the kinds of projects where mistakes are costly, difficult to diagnose and to fix. And I think more software developers are starting to realize this [0] (even if they're using different methods). [0]: ▶ this video Update : grammar/spelling

Hey, Joran from TigerBeetle here! You raise a fair question. Coincidentally, one of the reasons we picked Zig was for how readable it was, and strikingly so, even for high level programmers who might not understand systems programming or C. Because Zig reads like TypeScript, and we were working in payment switches where the majority of programmers could read that. This particular switch, in fact, had this same business requirement, that programmers should be able to at least read the systems language. But generally, our experience has been that people who understand C will understand how to maintain Zig [0]. Zig's toolchain is also more accessible, and across all platforms. Zig's compiler is already being used by Uber for hermetic builds. It's also easy to learn. You can pick up Zig in a week and be comfortable in a month. Zig has a simple grammar. I love how, when we have someone join the team, we never have a discussion about how to learn Zig, as if it's a difficult language to master (like C++!). Rather, there's excitement around learning the language, even ahead of starting at work, and within a day or two they're committing. We made this decision for TigerBeetle in July 2020, and didn't take it lightly. We had already followed Zig's progress for 2 years by that point, and many factors were considered [1][2][3]. C was the other contender, given that we had to handle memory allocation failure. The crux of the decision, then, was whether to invest in a systems language of t