"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson

"FDB can tolerate f failures with only f+1 replicas." Wait a minute, I know that formula... Viewstamped Replication?? I need to read the foundation DB paper. (I mainly read CRDT stuff so hopefully it's understandable). --- In general I'm really impressed foundation DB folks. The talk "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson blew my mind. TL;DR they spent the majority of their initial dev effort into making a simulation of the database, then when they were happy with that plugged in real storage, time and networks at the end. Well worth a watch for anyone interested in distributed systems & reliability. ▶ this video

Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: ▶ this video

Soooo stoked. This should make futzing with it to get started a lot easier. I've been thinking about going to the FoundationDB summit to hopefully learn more... Does anyone have any good resources for learning about it? I've tried a few times but find the documentation a bit hard to grok. Maybe I'm just slow. Also if you haven't seen it, this is one of the most amazing talks on distributed systems (because it's about testing them) I've ever seen (by Will Wilson from FoundationDB pre-Apple merger): ▶ this video Truly mind blowing how good FoundationDB is supposed to be for its use cases, and this Mongo layer is just the business when it comes to helping adoption.

This talk was about Foundation DB was brought up recently, and it's pretty amazing. I recommend watching the whole thing, but to be brief they are taming the "infinite interleavings" problem through determinism. "Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson ▶ this video They wrote an interesting Actor DSL that compiles to C++ and is completely deterministic, and they torture this deterministic engine with generated test cases on a cluster every night. I guess you could say that the whole cluster is necessarily non-deterministic, but an individual node is deterministic, given an ordering of the messages it receives.

One of their engineers gave a great talk at Strangeloop ( ▶ this video ) that shows some of the amazing simulations they ran FoundationDB through. It's well worth a watch if you want to understand the punishment that they put this database through to make it stand out.

FondationDB is such an incredible resource, but it’s so hard to access for the general public. It’s no wonder so few people know about it. I’ve always thought that if there were easier ways to interact with it, or some dev-friendly layers built on top (or public) it would become way more popular. it has shown how well it scales with Apple and snowflake, but is not an obvious choice for non-internet scale applications. If you don't know about FDB; there's an amazing video about how they test it that really got me into learning more: ▶ this video

The great thing about FDB is that they have a testing infrastructure that is unlike any other database: ▶ this video If there is a Swift compiler bug that causes data loss I will bet money they will catch it (I will also bet they will find at least one Swift compiler bug)

Tangent about FoundationDB: this is a great video that explains how the team tested it ▶ this video Spoiler: they even had their own custom power supplies used to test against power failures.

It is a tall order, but but it's doable. The most important thing is to reduce the surface area of your application. Our business logic is almost entirely pure. The 'procs' have no side effects, except for things that happen on the other side of a clojure protocol (java interface). That means all side effects can be stubbed out during tests. Our "user" code has no access to threading libraries. Threading happens in "kernel" code. Watch ▶ this video for a good example of how this has already been done in practice.

There is a great talk about this testing, it goes all the way to deterministic seeds and other basics. The level of determinism is admirable. ▶ this video

>>> But for the Web site for my startup, early on it was clear that I needed a good key-value store as a Web user session state store (server). So, I designed one, coded it up, and have been using it. It seems to work fine. Well depending on what error scenarios you might run into your code will work until it runs into those scenarios. Few ideas what could go wrong you can get from this video: ▶ this video

I'll give you some brief thoughts from my experience working at FoundationDB, but if you really want the in-depth answer to what makes our simulation framework an enabling technology for everything we do, you should take a look at the talk my colleague Will Wilson gave at Strange Loop [0]. Everything in our core is written in Flow, our asynchronous programming environment. The architecture of our server is essentially single threaded, with (pseudo-)concurrent actors allowing a single process to satisfy multiple roles at once. The interfaces that these actors use to communicate -- with the disk, over the network, with the OS in general -- are abstracted and implemented in Flow as well. Each of these interfaces not only has at least one "real" implementation, but at least one, and sometimes more, simulated implementations. Since we run multiple actors in a single process all the time, running yet more actors, pretending to be different machines, all still in the same process, was an obvious step. These workers communicate with each other via the _same_ network interfaces that real-world workers do in real clusters. In the end we are able to become our own adversaries, pushing the limits of the system in ways that just don't happen often enough in the real world to test and debug in the wild. Our simulated implementations are allowed to present any behavior that can be observed in the real world, or justified by the spec, or implied by the contract. And they can do so much more