You might've seen the new streaming memory metrics in ops desktop recently. You might be wondering how that was made. It was done with a balloon driver.
One of those recent emails was pointing out that we were silently ignoring the IP_PMTUDISC_DO flag which is an option you can pass to setsockopt. This sets the "dont fragment" flag. This option instructs the sending host and routers down the line not to fragment (eg: split up) IP packets.
We initially put in basic io_uring support almost four years ago but have been slowly applying more and more support as newer applications utilize it. In particular we added support for send, accept and recv which amongst other things allows you to write things like webservers using io_uring.
We think databases in particular mesh very well with the general idea of unikernels and we are far from alone, whether it was the folks behind SycallaDB or DBOS or TUM. One of the things that you'll see in some databases and we think we'll see more of in the future is O_DIRECT.
There are a ton of research papers on unikernels going back forever. Now that Nanos has been around for a few years we've been popping up in them and so we pay close attention to whenever someone seems to suggest there might be a performance or security issue. One paper had indicated poor memory management performance. We figured out pretty quickly that it was because we did not have support for transparent huge pages.
A while back ago CDNs would market on their TTFB - that is the 'time to first byte'. It sounds like a really important number and the lower you can make it the more impressive it looks. At the end of the day though it doesn't really matter cause a single image can be tens of kilobytes or even megabytes in size. A single page can have 80 some odd requests and transfer multiple megabytes in resources. So while it is interesting it's a bit of a vanity metric.
We've made substantial changes to our networking stack. Some of those include numerous changes to our LWIP fork (which not even sure we can call it LWIP anymore as it's totally different). This latest change allows multiple transmit and receive (tx/rx) queues. In short this allows superior network performance when you have an instance with multiple vcpus.
We recently introduced support for the memfd_create syscall which allows one to create anonyous files. Anonymous files are different from something like a temporary file in that it lives in ram vs being stored on disk.
We recently enabled KASLR on by default. We currently don't even have a way of turning it off via manifest since it is turned on before we even detect the disk from which the manifest option might be located -- in other words you have to disable it via a custom build.
We brought AMD SEV into the Nanos unikernel. SEV (secure encrypted virtualization) is an extension of the existing AMD SME (secure memory encryption) extension. Essentially, memory pages (including code and data) are encrypted outside the purview of the hypervisor using dedicated hardware. This hardware encrypts data as it is written to DRAM and decrypted upon read.
We took google's UMCG (user mode concurrency groups) implementation (which is a part of google fibers - no not the isp) and added it as an optional klib to nanos. This allows you to make your own user-space schedulers. This opens a whole new world of ultra highly scalable software. I can see a lot of applications for this including high performance databases and making various language runtimes a lot smarter. We also think this fits in very well with the Nanos architecture.
We recently made a very large change in Nanos. We jettisoned all of the null terminated strings that we could. While it's something that's been on our long list of known issues we wanted to address we finally bit the bullet and started the transition.
Everyone seems to think you can wave a few magic wands and instantly get superior performance -- unfortunately reality is not that cut and dry. Performance engineering requires extensive engineering, measuring and re-engineering to achieve results. Don't listen to charalatans that tell you otherwise.
One of the first things that stands out to people when they first start playing around with unikernels is that the concept of an interactive server that you can pop into and start running all sorts of arbitrary programs (also known as 'commands') is supplanted by just one program - yours.
In Nov 2023 Amazon announced micro-second precision with their PTP clock. Within two weeks we had the first bit of support out the door. But we're getting ahead ourselves. PTP? PHC? Is this some kind of a new drug? What happened to NTP?
We've supported nginx running as a unikernel as-is w/no patches for a long time but it still uses 90s style worker processes, so naturally we cut patches to convert them to threads and instantly got a free performance boost.
VSOCK refers to the AF_VSOCK address family. You might be more familiar with the AF_INET socket family. At it's core, it enables a way to communicate talking from a guest vm to the host or vice-versa. You can utilize both SOCK_STREAM (connection based) and SOCK_DGRAM (connection-less).
This isn't the first time we took a look at running postgres as a unikernel. A few years ago one of our engineers took a look at how difficult it might be to port it. He made a lot of progress but after spending a handful of days getting rid of shared memory and converting the processes to threads decided to punt it down the road until someone really wanted it.
We recently added support for both OpenBSD's pledge and unveil syscalls. The first one can apply security policies for disallowing certain syscalls while the latter one can apply a restricted filesystem view. In this tutorial we'll show you how you can easily apply a sandbox to your prod workloads effortlessly.
For quite some time now ops has auto-generated firewall rules for various cloud providers such as GCP and AWS. In this tutorial we learn how to implement a network firewall in the Nanos unikernel through simple configuration.
With the addition of VirtFS support to the Nanos kernel, it is now possible to access files and folders on your local development machine from your on-premises Nanos instance. This allows creating/deleting/modifying files in your computer and having these changes synchronized to a running Nanos instance without re-creating an image or rebooting the instance.
There are about a billion ways to get logs out of your nanos instance. By default when ran locally via 'ops run' or 'ops pkg load' stdout/stderr go straight out through the serial port and that is fine for most use-cases *locally*.
Machine learning frameworks such as TensorFlow and heterogeneous computing libraries such as OpenCL can use GPUs as computing devices in addition or as an alternative to CPU cores. With a new klib implementing a GPU driver, you can run a Nanos unikernel in a VM instance equipped with a GPU and execute your compute-intensive workloads on the GPU.
It seems that all the cool companies are doing cross-region geo-routing for applications today. Many of them will label this as "edge". For someone like myself edge usually means something different like having actual devices at sea, in the air, on a cell tower, in a Tesla, etc but if you are coming from the CDN world that translates to having multiple POPs (point of presence) spread throughout the world where you can host typically static sites (like javascript, css, etc.) closer to the end-user.
In the Nanos kernel, we have recently added support for the virtio-rng device. This virtio device allows applications to use it as a source of randomness without relying on dedicated processor instructions that may not be available in all clouds or hypervisors.
Thanks to a new kernel library, Nanos can now communicate with Amazon CloudWatch, the monitoring service for AWS applications and resources. This article explains how to set up a Nanos unikernel application so that its memory utilization metrics can be monitored via Amazon CloudWatch.
This article does an overview on the use of closures in the Nanos kernel. Closures are common in languages like Ruby and Swift. The C language does not include them, thus, the Nanos kernel relies on its own implementation. We use closures not only to implement callbacks with saved variables but also to compose continuations for asynchronous operations. In this article, we present what a closure is and we show the dedicated API that the Nanos kernel provides to manipulate closures.
General Purpose Operating Systems use a filesystem (fs) to store data in a persistent way like on a hard disk. Generally speaking, data is stored in files which belong to a directory. Thus most filesystems are organized in a hierarchical way. The fs stores files in blocks into physical devices.
With Nanos now able to run on AWS Graviton instances, we have the option of deploying a Nanos unikernel on public cloud VMs with processors other than Intel or AMD. This article shows how to run Nanos on a Graviton2 (64-bit ARM) instance, and explains some technical details involved in running an operating system on an ARM-based server platform.
Nanos can now be built for the riscv64 (64-bit RISC-V) architecture. We discuss how to cross-compile and run Nanos and RISC-V programs with QEMU, and also go over some of the technical aspects of porting Nanos to RISC-V.
Red-black trees (rbtrees) are data-structures widely used in the kernel. For example, this data-structure is one of the building blocks of the Linux scheduler. In this article, we overview general concepts about trees and then we focus on what makes rbtrees very interesting.
This article is an overview of bitmaps, which is a data-structure heavily used in the kernel. NanoVMs provides a dedicated API to handle bitmaps. In the following, we present the API to instantiate bitmaps and how this API is adapted to deal with different use-cases. Also, we present the kernel’s data-structures that are represented by using bitmaps.
This article overviews EBPF (Extended Berkeley Packet Filter), which is a recent technology available in the Linux kernel to add user-defined functionalities in the kernel. This article aims at understanding what EBPF is and its use-cases.
In this tutorial we show you have to run a Nanos unikernel on the Bhyve hypervisor on top of FreeBSD.
The security advantages of unikernels make them a great fit for a public-facing dedicated network appliance such as a VPN gateway. With some special configuration, we can run a userspace WireGuard implementation on Nanos to make a gateway to a cloud private network.
Nanos has many different logging options. You can log to serial, you can log to files, you can ship things over syslog. We even made a syslog klib so you don't have to modify your code. Today we'll show you yet another way to log.
OPS recently gained native TFS dump support. TFS is the default filesystem for the Nanos unikernel. The new support allows you to run simple commands such as ls, cp, and tree directly on a Nanos image. You don't need FUSE and you can run this on a Mac too.
It is no secret we like performance. Hell, we wrote our own kernel for that very reason. We've also made a lot of tooling in the past to help us find hotspots and make it easier to profile Nanos. Today we'll introduce you to another tool.
Autoscaling has also been a common feature request from many users coming from a kubernetes environment. The cool thing about unikernels is that all of kubernetes complexity gets abstracted away since Nanos unikernels are deployed as virtual machines and the underlying cloud network and volume primitives are utilized versus duplicated with frameworks such as k8s.
Some applications are hard to cross-compile on Mac if you are deploying to Linux servers. If you are using unikernels you might be relying on docker or vagrant to do this, however that is not necessary anymore with the new cross-compilation support for OPS.
Many Nanos/OPS users come from a docker background and there is a hidden implication that they might be working on a Mac for dev purposes. This can pose a problem for those that want to create their own software on a mac to run as a Nanos unikernel as Nanos consumes ELF binaries just like Linux does.
Nanos now targets platforms based on the aarch64 (64-bit ARM) architecture. In this tutorial, we'll learn how to build and run ARM unikernels, both on a Raspberry Pi 4 and as a cross-build from a non-ARM host. No ARM hardware required!
Ruby and Python are both popular interpreted languages. For many years developers that wished to deploy their Django, Rails and other applications written with these frameworks have routinely relied on front-end proxies such as Nginx and Haproxy and usually stuck those in front of a load balancer such as an ELB.
One of our engineers just opened a PR for a FUSE driver for Nanos. What does this mean? It means you can now mount the filesystem natively on MacOS and Linux amongst other things.
Memory management errors are some of the most pernicious and most difficult bugs to troubleshoot. We can leverage the unique design of Nanos in order to create an small and efficient way of identifying these problems.
Mattermost is an open source, private cloud slack alternative written in Go and React. It has many integrations and can can connect to mysql or postgres for persistence. While there are many "hello world" tutorials out there for running the Nanos unikernel we thought this would be a nice little example to showcase two things.
A while ago we wanted a pluggable method to instrument various stats in Nanos and we wanted to be able to apply this functionality to ad-hoc applications without having to re-write the code. So we took a page from the library operating system playbook and produced an inital draft for klibs in Nanos.
The most obvious time when you would want to attach a volume to your unikernel instance is when you are working with a database. Your database image probably doesn't change that much but many databases can grow very large and it makes no sense to keep the same data volume in your base image.
Some people think you can't debug unikernels. Tell that to the team of kernel engineers that routinely have to debug unikernels, sometimes without source code, sometimes without access to the system in question.
How does one go about profiling or tracing a unikernel? How do you login to the host and run perf or produce flame graphs? Isn't this impossible in a unikernel?
Knowing the memory layout of an application and it's host operating system is vital to attacking a system. What if you could reboot an entire server in seconds and get a brand new memory layout each time? While not feasible with traditional linux vms and definitely not containers this is a uniquely cool unikernel feature.
In this short tutorial we learn how to build, deploy, and run node.js unikenels to AWS using the t2 instance type.
Learn to use machine learning classification with python3, sklearn and run them using unikernels.
InfluxDB is a popular time series database that fits well inside an edge or IoT pipeline where you might be already using something like MQTT. If you're already using unikernels for your edge devices it might be a good idea to turn your time series datastore into one as well.
MQTT is the quintessential protocol for edge and IoT devices to talk to the cloud for more than a few reasons. If you're a pumpjack in the middle of nowhere west texas 50 miles from town and you're in a field of said pumpjacks getting that data out can be a hassle.
A lot has been said about the security posture of unikernels. I know I've said a mouthful myself. ;) I originally wasn't going to write this article but I've now had to respond to the same question a number of times so my hand has been forced.
Forth has been called lots of things and it was something that I figured we could easily unikernelize so I dove into it. I think it's safe to say I've never written any production Forth nor do I see myself doing so but having said that - let's begin!
WASM is a new way of transpiling high level languages to a common binary format. Here we show how we can take WASM and run it as a unikernel.
Learn how to implement logging in your Go unikernels with a Free Account from PaperTrail.
After running my first c example in rumprun a few years ago the next language I wanted to try out was Go. Unfortunately there were no Go unikernels at the time so we sponsored one for rumprun. That was 3 years ago.
Javascript developers were some of the earliest adopters of things like containers, serverless and other infrastructure patterns so it's not surprising that unikernels interest them too.
