MacOS Containers, Docker Desktop and Unikernels

Apple unleashed MacOS Containers at the WWDC the other week. For the longest time Apple did not have any container support and since this isn't an official release - macOS Tahoe 26 comes later this fall - only a handful of developers actually ended up trying it out.

You can now download the first release on the releases page here. Before that you needed to install an appropriate swift toolchain composed of swift, swiftly and the static linux sdk. The last part is weird considering that Apple has gone out of its way to make building static binaries on macos insanely difficult to impossible.

They also released a containerization library that you can use to hook into this system.

But how does all of this compare to existing container solutions on mac and what does it have to do with unikernels?

Apple Thinks Container Security Sucks

One thing that people immediately saw was that each container was isolated by a virtual machine and consequently each container got its own ip. This is really interesting as Apple today does not have a public cloud that developers can use so what they are really saying is that containers are so incredibly dangerous that they don't even want you to run them locally unless they are properly isolated.

Just to be clear here - it isn't just about protecting one container from another container. It is also about protecting the host from a guest container.

It should be noted that Apple is by far not the only company that has grave concerns about container security. Google implemented gVisor to wrap a sandbox around containers and AWS has lambda built on a version of firecracker which wraps containers in virtual machines.

A lot of people were asking on HN and lobste.rs why Apple didn't just integrate container support into MacOS natively vs booting them inside a linux vm. I think this is, at least, a three part answer.

First off there is no "create_container(2)" in linux. This is a very misunderstood concept that a lot of people don't really understand. Containers are a conglomeration of different apis designed for different purposes that are implemented in different ways across different runtimes.

Second off, Apple seems very apprehensive about allowing container related constructs into its kernel for very good reason as the attack surface is just way too large and is constantly changing. Your average container escape is a combination of a symlink pointing to something it shouldn't with a TOCTOU check gone awry and they happen all the time.

It is far easier for Apple to simply have a better 'docker desktop' wrapping these in a vm - after all those who use docker desktop wouldn't be using a linux desktop for their day to day to begin with.

Thirdly, macOS is very different from a linux server environment. You have mach-o binaries, sometimes fat, vs elfs. You have predominantly x86 vs arm64. You have APFS vs ext4. It just doesn't make sense to have "native" container support on macs even though a huge chunk of developers use them day to day.

MacOS Containers

If you downloaded the container runtime from the link posted earlier we can check them out.

To start the system:

container system start

To build:

container build --tag web-test --file Dockerfile .

A lot of people are noticing that macos containers boot fast - that is from their custom init system and kernel.

➜  web-test gtime -f "%E" container run --name golang --rm golang
vim-go
0:00.72
➜  web-test gtime -f "%E" container run --name golang --rm golang
vim-go
0:00.73

➜  web-test gtime -f "%E" ops run web-test
running local instance
booting /Users/eyberg/.ops/images/web-test ...
[0.002239] vtbln failed to attach: (status:failed to find free MSI-X slot)
[0.003496] en1: assigned 10.0.2.15
vim-go
0:00.33
➜  web-test gtime -f "%E" ops run web-test
running local instance
booting /Users/eyberg/.ops/images/web-test ...
[0.002256] vtbln failed to attach: (status:failed to find free MSI-X slot)
[0.003488] en1: assigned 10.0.2.15
vim-go
0:00.34

Whle some people have noted in the past that you can't get these fast boot times for unikernels in prod that totally depends on where you are deploying them to. They are correct that if you do a simple 'ops instance create' on AWS the spinup time on EC2 is not tens of milliseconds but with NanoVMs Inception you can achieve instant boot times and also enjoy pause/resume. If you haven't tried that out yet we'd highly encourage you to see for youreself. Boot times are a function of both the orchestration and the size of the disk image being loaded. You also need to be clear about what you're measuring - from the time the kernel starts executing? From the time the kernel has an ip address (that can take a much longer time with DHCP)? From the time the user program is executing?

Current Limitations of MacOS Containers

If you're brave and have an apple developer account you can upgrade to get the Tahoe developer beta and unlock a lot of capabilities. For instance right now you can't talk from one container to another on the same network.

You also have to do hacks to have container to host networking.

Similarly, there is only partial support for memory ballooning today with macos containers. What this means is that if you are provisioning workloads that need it, like ai applications you are probably going to need to restart quite a lot. OPS has had native memory ballooning for quite some time and it is how we can track memory utilization locally with ops desktop.

These limitations are probably one of the reasons why Apple hadn't crafted compose like functionality yet, however that hasn't stopped several folks that have already came out with compose alternatives.

Other Differences

There are some other differences as well. They wrote a new init system 'vminitd' in swift. It also makes use of Swift Static linux sdk to cross-compile linux binaries with musl. While musl is praised for being 'smaller' lots of folks have observed many differences that don't pair 1:1 to libc including runtime performance issues and the infamous dns issue.

So as you can see the container runtime by Apple is pretty different from WSL2 and pretty different from Docker Desktop. While it is not what some people were looking for I don't actually expect Apple to do something different here. I do have a feeeling this is going to heavily eat into docker desktop though.

At the end of the day we see this as a push to make containers behave more and more like unikernels. You haven't tried out unikernels through ops yet - what are you waiting for? Give it a whirl!