Shouting in the Datacenter

I had the "vibration" problem a few times in me IT career, train, construction work and once elevators. It is the kind of issue that are really hard to diagnose the first time but after the first time it's part of the standard "diagnose suite". I love to show this video about the effect of noise on HDD (be sure to put the volume down) ▶ this video

One of the little gems I found while I was putting together the video for this post was "Shouting in the datacenter", an older clip (from the pre-HD era) demonstrating the consequence of doing exactly what the title says: ▶ this video Warning: loud!

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: ▶ this video (EDIT: replaced the link I found via YouTube's search with the original, thanks robotmlg!)

Also interesting: a Sun engineer uses dtrace to show HDD latency spiking after he yells at his disks ▶ this video

i just deleted a many-paragraph response that was getting long-winded(incoherent) and started over -- yes, i concur. like many, i dislike the "i want to help save the world" generic posts, but i think most here are being quick to dismiss TFA. to others (not the parent): just want to help place a human face onto this guy, who has (not known to me until now) not only published a bunch of works (whose titles i find very relevant to me as a sysadmin), but was the guy in front of the camera in the Shouting in the Datacenter clip[0] !! things are difficult, but i hang on to hope. and undoubtedly, i have tons to learn from this guy. [0]: ▶ this video

In the photo of him on his last day [0], there's a cassette deck on his desk. That could be something mundane, but I'd like to believe something crazy happens if you yell at it [1]... [0] link [1] ▶ this video

I believe it because it's a plausible variant of what I call the "Fus Ro Data Loss" vulnerability: shouting at hard drives causes them to resonate in a way that affects their ability to access data. ▶ this video

I recall a video of a guy temporarily reducing hard drive performance by shouting at it edit: here it is! ▶ this video

You clearly haven't read the full article as Jim Salter writes about the mechanical stuff at the end of the article. Also, you want to reduce vibrations because of this: ▶ this video (Shouting in the datacenter)

Didn't know Brendan Gregg was at Intel now. link He is also the star of the Shouting in the Data Center viral video ▶ this video True genius!

> For example, imagine an evil SSD which had a 1% chance of rolling a sector back to a previous version, a 1% chance of saying a write failed when it didn't, a 1% chance of writing data to the wrong sector number, a 1% chance of flipping some bits in the written data, and a 1% chance of disconnecting and reconnecting a few seconds later. There are stories from the ZFS folks of dealing with these issues and things ran just fine. While not directly involved with ZFS development (IIRC), Bryan Cantrill was very 'ZFS-adjacent' since he used Solaris-based systems for a lot of his career, and he has several rants about firmware that you can find online. A video that went viral many years ago, with Cantrill and Brendan Gregg, is "Shouting in the Datacenter": * ▶ this video * link (making of)

As long as you don't traumatize the hard drives by yelling at them, feat. brendangregg & bcantrill: ▶ this video