NYCBSDCon 2014: Serving one-third of the Internet via FreeBSD
BSDTVCited 5× on HN, 2014–2019
talk · intermediate · 2014–2019 on HN
Scott Long of Netflix presents the architecture of the OpenConnect content delivery network at NYCBSDCon 2014. He explains why Netflix runs FreeBSD 10 on its appliances and what the future holds.
Cited as the primary source for why Netflix uses FreeBSD and UFS rather than ZFS on its CDN boxes, and for its GPL reasoning.
Up next
9 mo on HNHarder Drive: Hard drives we didn't want or need
suckerpinch
12 HN recommender · 2022–2025
9 mo on HNThis Video Is Sponsored By ███ VPN
Tom Scott
9 HN recommender · 2019–2026
6 mo on HNA New Way to look at Networking
Google
6 HN recommender · 2011–2018
6 mo on HNNet Neutrality: Last Week Tonight with John Oliver (HBO)
LastWeekTonight
7 HN recommender · 2014–2017
4 mo on HNCharacters, Symbols and the Unicode Miracle - Computerphile
Computerphile
4 HN recommender · 2014–2025
5 mo on HN8 Frames in 16ms: Rollback Networking in Mortal Kombat and Injustice 2
GDC Festival of Gaming
4 HN recommender · 2020–2025
4 mo on HNA New Way to look at Networking
GoogleTalksArchive
4 HN recommender · 2012–2025
4 mo on HNWhy SpaceX is Making Starlink
Real Engineering
4 HN recommender · 2019–2020
3 mo on HNAWS re:Invent 2015 | (NET403) Another Day, Another Billion Packets
Amazon Web Services
3 HN recommender · 2017–2020
3 mo on HNEmail Hates the Living
m
3 HN recommender · 2015–2022
3 mo on HNRicardo Signes - 闇のEメール伝説
yapcasia
3 HN recommender · 2011–2021
3 mo on HNDefeating Ssl Using Sslstrip (Marlinspike Blackhat)
TheSecurityTube
3 HN recommender · 2014–2018
5 comments from Hacker News
Pretty sure it's still UFS+J. They've mentioned in the past that the benefits of ZFS don't really apply to the CDN boxes: if a drive dies, it's simply removed from the local cache, and files are re-downloaded/redistributed as necessary across the remaining cache drives the next time the appliance refreshes from its upstream origin. Additionally, ZFS prefetching was actually less performant in some cases from what I remember -- or at least providing no benefit -- because of how it couldn't really anticipate the I/O patterns of video segment requests from users. Additional reference: - link ("…but caching is futile.") - ▶ this video
Netflix delivers all video streaming content via FreeBSD which makes up 37% of all internet traffic. link They also contribute to FreeBSD core. It is not just an "appliance" as some people seem to want to point to. link link ▶ this video In addition, a guy I met, who worked at Yahoo, told me about 25% of Yahoo still runs FreeBSD as of two years ago.
Scott Long has summarized it a few times, eg, from link : • Availability of expertise, outstanding community • Works well, good vendor support • No GPL • Features used: – SUJ – gmirror – boot drive only – AIO – Dtrace, HWPMC – TCP Stack, modular CC It's missing "Integrates tightly with nginx: Async I/O and sendfile()" which Scott included in ▶ this video . In the video he elaborates a bit more about the GPL... edit: formatting
Actually it turns out they don't really have a need for most of the features of ZFS. So they don't use it. ▶ this video
> I'm assuming whatever file system is on the disks... Netflix uses UFS+J on FreeBSD 10. Here are some notes from their talk at NYCBSDCon 2014: - 400,000 stream files per appliance. - 5,000 - 25,000 client streams per appliance. - 300 - 500 streams coming off each disk all the time. - Attempt to buffer 1MB ahead, but caching is futile. - Result is completely random disk workload. - System becomes limited by disk latency and CPU load. Video here: ▶ this video