Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017

The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: ▶ this video Here's the slides: link It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.

Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) ▶ this video

Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: ▶ this video

Bryan Cantrill draws some excellent parallels between this incident and software systems in his "Debugging Under Fire" talk (beware of semi-automated systems): ▶ this video

I’m surprised I haven’t seen this linked yet, but Bryan Cantrill of dtrace, Sun, lawnmower, Joyent, etc, fame gave an amazing talk for Monktoberfest 2016, titled “Oral Tradition in Software Engineering”, which features The Story of Mel [1]. Highly recommend checking it out — there are loads of little gems and stories like this throughout. All of his other presentations are great too and definitely worth a listen if you like this sort of thing [2]. A couple of my favorites are “Fork Yeah! The Rise and Development of Illumos” [3] and “Debugging Under Fire: Keep your Head when Systems have Lost their Mind” [4]. [1]: link [2]: link [3]: link [4]: ▶ this video

I suppose it all depends on how much infra needs to be stood up for the absolute necessities of the business to operate. Does the company need that internal ticketing system in place to process external client transactions? Probably not, but it'll need it eventually (so maybe that moves to 2nd tier restore process?). My company's RTO is 24hrs to processing new client transactions. Restoring old ones will definitely take longer, but at least new ones can proceed. If your own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage ( ▶ this video ). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.

I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other). That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": ▶ this video So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.

Not (I think) the exacttalk/blog post gp was thinking of - but worth watching IMNHO: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" ▶ this video Ed: oh, here we go I think? > Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist. link

> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind ▶ this video

> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk[1] about the time an operator accidentally rebooted an entire datacenter with a single miss-typed command. The general solution to this is building sanity checks into the software. The user just asked to reformat 500 hosts, but almost all previous uses olf the 'reformat' command affected less than 10 hosts. Maybe we should ask for verification from an actual human if they really intended to run this unusually destructive command. Why doesn't YouTube have this kind of sanity check in their automated takedown/strike/channel-deletion tools? Google wrote automation that can decide to delete a channel with a long history and many successful videos. Why doesn't that automation have basic sanity checks that ask for operator input when asked to do an unusually destructive action like deleting a 10 year old channel with a huge history?

> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. Bryan Cantrill has a great talk[0] about dealing with fires where he says something to the effect of: > Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible [0] Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: ▶ this video

I was! (Along with co-author Dave Pacheco.) And I still have the dream that we'll finish it one day: we had written probably a third of it, but then life intervened in various dimensions. And indeed, as part of our preparation to write our book (which we titled The Joy of Debugging ), we read Wheeler's Debugging . On the one hand, I think it's great to have anything written about debugging, as it's a subject that has not been treated with the weight that it deserves. But on the other, the "methodology" here is really more of a collection of aphorisms; if folks find it helpful, great -- but I came away from Debugging thinking that the canonical book on debugging has yet to be written. Fortunately, my efforts with Dave weren't for naught: as part of testing our own ideas on the subject, I gave a series of presentations from ~2015 to ~2017 that described our thinking. A talk that pulls many of these together is my GOTO Chicago talk in 2017, on debugging production systems.[0] That talk doesn't incorporate all of our thinking, but I think it gets to a lot of it -- and I do think it stands at a bit of contrast to Wheeler's work. [0] ▶ this video