NYC Tech Talk Series: How Google Backs Up the Internet

FWIW... "Backups are a tax you pay for the luxury of restore" "How Google Backs Up the Internet" ▶ this video Detailed notes: link

The "petabyte for a century" problem statement he mentions near the top is fun: can you preserve 1 PB with better-than-even odds of all the bits coming back right in a century? He wrote something about how he saw the durability situation in 2010: link . He seems to define the problem to allow for maintenance, e.g. check-and-repair operations are discussed. Broadening a little, I read it as "if you're trying to be more cautious than you usually want for commercial/academic online storage or even backup, what do you do? And would it work?" A lot (but not all) of the author's Queue piece talks about stats from online storage, which doesn't have some wins you can get if you're entirely about long-term durability. In online systems, heavy ECC seems out of favor compared to replication for performance reasons, but additional LDPC or RS at the app layer can absorb a substantial % of your volumes having problems, or a ton of random bad blocks. (In a more near-line context, Backblaze uses RS: link ) Same for tape -- offline LTO is slow and drives are costly, but the cost/TB and the rated lifespan seem like advantages over HDDs for this specific goal, at least in a narrow engineering sense. A pile of cryptographic hashes that fits on one device can help you check for and localize errors without sending everything over a network/doing the full ECC dance. If the hash being broken and data tampered with is in your threat model (hey, weird things can happen in a century), you can also

I also remember that talk. I haven't verified by watching again, but if anyone's interested I suspect this is the one: ▶ this video

This should serve as warning to anyone relying blindly on cheap cloud to keep themselves up and running. This also remind me of importance of offline tape backup. Google still uses tapes[1], yet some of the companies have started thinking they can eliminate tapes and move their backups to cloud. [1] ▶ this video

There may occasional bugs, but I don't know of any cases when Gmail actually lost emails. According to this talk [0], the bug referred to by that article caused some users' emails to be temporarily inaccessible, but all the emails were eventually recovered. [0]: ▶ this video