illumos day 2014: The Dream is Alive! Running Linux containers on an illumos kernel

I really enjoyed this talk[1] by Bryan Cantrill, part of which is about what it took to emulate Linux's syscalls from within SmartOS. It's lengthy, but enjoyable. One of the applications that they tried to port directly to before the emulation layer was htop, and he has a good rant about what that took because of its use of /proc and how that made porting it very difficult. Given that, I'm sure this wasn't easy. Kudos! [1] ▶ this video

Video of this talk. ▶ this video Though it is from 2014, it is interesting on comparison between linux and smartos(solaris). There is also a recent, shorter video link

A stable syscall ABI is a Linux peculiarity and was never a part of the UNIX tradition. Here's a talk where Bryan Cantrill describes the tightly coupled Solaris kernel/libc combination as a "welded unit", and mocks the Linux "libc ecosystem" and the complexity of ENOSYS handling in glibc. ▶ this video

You might need that when you want to reimplement Linux, the Joyent team did that on their OS (derived from solaris) so that user can run linux binaries on a solaris kernel (so thay have dtrace, zfs, mdb, ...) Bryan Cantrill did a bunch of conferences on that (one here: ▶ this video ) The idea behind is that Linux is only a list of syscalls, if you are able to reimplement them, you reimplement linux, you don't need anything else. On the contrary if you want to reimplement a BSD you need to reimplement their libc (and perhaps some other libraries)

From my perspective, if this rumor is true, it's a relief. Solaris died the moment that they made the source proprietary -- a decision so incredibly stupid that it still makes my head hurt six years later. Fortunately, Solaris was open long enough that we in the open source world were able to fork it with illumos[1]. And because illumos became the home for many of us that brought Solaris its most famous innovations (e.g., ZFS, DTrace and zones), it should come as no surprise that we've continued to innovate over the last six years. (Speaking only for Joyent, we added revolutionary debugging support for node.js[2], ported KVM to it[3], completed and productized Linux-branded zones[4], added software-defined networking[5] and developed first-class Docker integration[6] -- among many, many other innovations.) So illumos (and derivatives like SmartOS, OmniOS and DelphixOS) is vibrant and alive -- but one of our biggest challenges has been its association with the name "Solaris": I don't think of our system as Solaris any more than I think of it as "SVR4" or "SunOS" or "7th Edition" or any of its other names -- and the very presence of Solaris has served to confuse. And indeed, it is my good fortune to be working with a new generation of engineers on the operating system -- engineers for whom the term "Solaris" is entirely distant and its presence as an actual (if proprietary) system befuddling. So if the rumor is true (and I suspect that it is), it will allow everyone to know wha

The Simcity case reminds me of Bryan Cantrill's talk on lx-branded zones: apparently, there was a bug in rpm which didn't touch Linux, because Linux wasn't doing things fast enough to trigger the condition. So there was some code added to check if the running program was rpm, and to pause it briefly if it was. Apparently the comment the people who wrote it left amounted to, "we wrote this, and even we feel dirty about it." You can find the actual talk at ▶ this video

FreeBSD emulating Linux has way less performance overhead than either emulating Windows. FreeBSD's Linux "emulator" consists in a substituted syscall table and a kernel module for implementing syscalls that aren't native to FreeBSD (like epoll).[0][1] Here's an example of what that means. When a FreeBSD binary wants to delete a file it calls the syscall unlink(2). So it's executing as a regular old binary, does a regular syscall, specifies the unlink syscall number, the kernel looks up that syscall number in that binary's (FreeBSD's) syscall table, and jumps to that code. When you execute a Linux binary on FreeBSD you also execute it as a regular old binary except that you specify to use the Linux syscall table instead of FreeBSD's (IIRC this is done by the elf loader, so you're not even saying "linuxrun ./mybinary"; you're just saying "./mybinary"). It's executing more or less identically to a FreeBSD binary except that when it's time to delete a file it looks up the unlink system call in the Linux syscall table and jumps to that instead. Some of the arguments or calling conventions are different so most of the calls have a shim to rearrange stuff for the FreeBSD equivalents before calling into the same code the FreeBSD binaries are using. This is the same thing that FreeBSD does to support ABI versions for old versions of itself, to support SysVR4 binaries[2], and it's the same trick that SmartOS's Linux zones use as well[3] What this means is that if your binary isn't usin

I don't understand. If bcantrill's description of VDSOs (buried in this presentation: ▶ this video ) is accurate, then they're just a way to make certain syscalls faster, when syscalls are already fast enough.

Funny you should mention that one in particular -- from our (SmartOS's) epoll(5) man page: While a best effort has been made to mimic the Linux semantics, there are some semantics that are too peculiar or ill-conceived to merit accommodation. In particular, the Linux epoll facility will -- by design -- continue to generate events for closed file descriptors where/when the underlying file description remains open. For example, if one were to fork(2) and subsequently close an actively epoll'd file descriptor in the parent, any events generated in the child on the implicitly duplicated file descriptor will continue to be delivered to the parent -- despite the fact that the parent itself no longer has any notion of the file description! This epoll facility refuses to honor these semantics; closing the EPOLL_CTL_ADD'd file descriptor will always result in no further events being generated for that event description. So while we do aspire to be bug-compatible, we're not about to compromise our principles over it. More details (or some of them, anyway) can be found in the talk on LX-branded zones that I gave at illumos Day at Surge 2014.[1][2] [1] link [2] ▶ this video

We have put (and are putting) a bunch of work into LX branded zones[1], and are at the point that they are working on an incredibly broad class of apps (64-bit and with on-the-metal performance). A concrete case in point: we recently ran the (amazing!) hundred language quine relay[2] in an LX branded 64-bit Ubuntu 14.04 zone on SmartOS.[3] As for ZFS support and Docker, it will be via sdc-docker[4], our (emerging) end-point for the Docker Remote API. The progress there has been swift and everything is being done in the open; expect to see something in production from us in this first calendar quarter. [1] ▶ this video [2] link [3] link [4] link

Of course you can get more specific - for example, the Solaris kernel/libc combination is so tightly coupled that it is described as a "welded unit" ▶ this video OpenBSD solved the Year 2038 problem by incompatibly changing the kernel and libc ABI in lockstep, which broke every single existing binary application using time_t - how's that for tight coupling... link

Assuming the last question is earnest: while I think "amateur" is a tad strong, I do think that Linux suffers enormously from NIH; it remains the first system (or first Unix system, anyway) for many of its contributors, who seem to know no other way -- and this shows up all over the system.[1] In terms of how this has changed over time: if anything, it used to be much, much worse. For example, Linus' attitude on kernel debugging circa 2000[2] is not merely amateurish, and not merely irresponsible, but actively dangerous. While his attitudes have soften over the years, there still remain striking differences of engineering principle between Linux and other systems -- of which systemd is merely (in my opinion) one particularly egregious manifestation. [1] ▶ this video [2] link