Run containers on bare metal already!

SmartOS's LX branded zones seem kind of cool to me, and may hit an interest point for you. Basically, SmartOS is an OpenSolaris variant where they use Solaris Zones for docker images, but they've implemented a Linux syscall table API so you can create Linux "compatible" containers from a Solaris kernel (and also full access to ZFS, etc). You can see a talk here[1] where Bryan Cantrill goes over the reasons you might want to use containers, SmartOS and how LX branded zones. Warning though, he's an extremely engaging speaker, so not only are you likely to watch all of that 40 minute talk, but you might feel like binging on some other talks of his, so here you go.[2] 1: ▶ this video 2: link

It's all just history now for all I know, but there was work in the past to make Linux containers work on a Solaris fork (SmartOS, specifically) by emulating the Linux syscall table and presenting that to the containers. Joyent did work on this (alas, and there's an excellent and entertaining talk from Bryan Cantrill[1] that goes over it. I imagine FreeBSD could do something similar if they aren't already. IIRC FreeBSD has a Linux emulation layer (but I don't know how much attention it still gets), and it's had containerization primitives longer than linux, so some amount of filling in the gaps in containerization features and syscall tables (if needed) could possibly yield an OCI compatibility later (for all I know all this already exists). The problem, and the reason if this doesn't exist why people probably weren't as interested in doing the work, is it would always be "mostly" compatible and working and there would be no guarantee that the underlying software wouldn't exhibit bugs or weird behavior from small behavior differences in the platform when emulating something else. Why open yourself up to the headache when you can just run Linux with containers or build what you want on FreeBSD with jails and their own native containerization primitives. 1: ▶ this video

I have run containers on bare metal and in Amazon. Performance can be noticeably slower in Amazon on machines with similar specifications. A detailed explanation why is in this informative video by Bryan Cantrill ▶ this video . It is a talk about how layering containers on top of virtual machines is a recipe for poor performance and wasted resource utilization. He equates it to developers “being given fire, then proceeding to put the fire out.”

Unikernels : hypervisor :: processes : traditional operating system Joyent CTO Bryan Cantrill gave a great talk at Velocity NYC 2015 about the history of containers (going back to March 18, 1982 when Bill Joy added the chroot system call) and the problems of running containers on operating systems in VMs on hypervisors: "Stop killing kittens and melthing the ice caps: Run containers on bare metal already!" ▶ this video

The second I read "shared block cache" my brain went to containers. If you want data colocated on the same filesystem, then put it on the same filesystem. VMs suck, nobody spins up a whole fabricated IBM-compatible PC and gaslights their executable because they want to. [1] They do it because their OS (a) doesn't have containers, (b) doesn't provide strong enough isolation between containers, or (c) the host kernel can't run their workload. (Different ISA, different syscalls, different executable format, etc.) Anyone who has ever tried to run heavyweight VMs atop a snapshotting volume already knows the idea of "shared blocks" is a fantasy; as soon as you do one large update inside the guest the delta between your volume clones and the base snapshot grows immensely. That's why Docker et al. has a concept of layers and you describe your desired state as a series of idempotent instructions applied to those layers. That's possible because Docker operates semantically on a filesystem; much harder to do at the level of a block device. Is the a block containing b"hello, world" part of a program's text section, or part of a user's document? You don't know, because the guest is asking you for an LBA, not a path, not modes, not an ACL, etc. - If you don't know that, the host kernel has no idea how the page should be mapped into memory. Furthermore storing the information to dedup common blocks is non-trivial: go look at the manpage for ZFS' deduplication and it is littered w/ warnings

> What most people call "containers" was always about virtualization (ie isolation), and the isolation primitive in the Linux kernel is namespaces. There is no isolation with cgroups in Linux, that is the crux of the matter: ▶ this video since containers in Solaris existed before cgroups and before the entire Linux hype, and you specifically adress my "misunderstanding" (of hype), you compel me to correct on terminology: containers are resource constraints, while technology like LXC and OpenVZ provide the lightweight virtualization and isolation , a very important distinction (full virtualization is achieved via XEN and KVM on GNU/Linux). Conceptually, as a resource constraint, containers are in that sense the same in Solaris as they are in Linux, with vastly different mechanism implementations, but neither provide isolation . Again, and I corrected you on this before (this happens to be my problem domain), what you think of as containers are lightweight virtual machines, as zones in Solaris and LXC / OpenVZ in Linux, and equating cgroups and namespaces with a lightweight virtual machine technology is conflating two different things. If you should have the inclination to point out my other "misunderstandings" of Linux, an operating system I very heavily use, develop on, and engineer for, I would be interested to learn of them. > So does Linux, I'm missing your point here. If they exist, I have not heard of them, read about them, or met them yet; at any rate, since Linux has so man

The biggest missed opportunity of ECS that AWS missed was completely hiding the complexity and concerns of managing EC2 instances. If you use ECS you have to deal with the complexity of both and manage those resources. The dream being sold by the container industry is that you don't really care about the machines you're containers are running on. ElasticBeanstalk gets this right because they hide that concern. ECS is akin to selling git on top of SVN. It doesn't really make sense. Bryan Cantrill gave a talk about the craziness of running containers on VMs at ▶ this video .

From a recent post, here's a talk[1] about containers where they use SmartOS (Solaris/IllumOS derivative) to containerize Linux by just implementing the syscall table. The future of hosting legacy apps might be providing legacy interfaces on modern OS's (IMO). 1: ▶ this video