Mounting a S3 Bucket to Your Unikernel with the s3fs klib

Unikernels don't just offer better security and performance - because of how they are architected you can often get much cleaner abstractions for various things as well. In our private issue tracker we've had an issue for being able to provide native s3 mounts to s3 buckets for a while now. Typically on linux you install something like s3fs. However, linux being linux that's for the entire instance, whereas with the unikernel it's soley for the one program that would need it. In nanos all you have to do is provide a configuration at image build time to tell nanos where/how to mount the bucket. This is interesting because new images are created every single time you hit the deploy button which means that you can run the same application locally without a bucket (or with) and when shipping to a different environment like prod enable it seamlessly. This also removes a lot of s3 glue code that would otherwise muck up your application.

Just to paint the picture - this is a minimal configuration to mount a s3 bucket at a particular path of which your application is oblivious to the fact that it's now looking at a s3 bucket vs a normal fs path:

{
  "Klibs": ["s3fs", "tls"],
  "ManifestPassthrough": {
    "s3fs": {
      "region": "us-west-1",
      "access_key": "MY_ACCESS_KEY",
      "secret": "MY_SECRET",
      "buckets": [
        {
          "name": "my-bucket",
          "mount": "/mnt"
        }
      ]
    }
  }
}

The exciting thing here is that this configuration is applied at image creation time and can remain entirely out of your development cycle. Just to restate this - you can take any existing application and apply this config and instantly have a s3 backed mount - it is that transparent.

Data engineers should take note that this is an excellent tool in your tool belt to build a data lake with.

Let's Build a Unikernel Backed Data Lake

To showcase this feature at play we can build a minimal server that simply ships http requests to parquet files hosted on s3 and allows them to be read from another path:

package main

import (
	"encoding/json"
	"fmt"
	"io"
	"net"
	"net/http"
	"os"

	"github.com/parquet-go/parquet-go"
)

func hitsHandler(w http.ResponseWriter, r *http.Request) {

	file, err := os.Open("tmp/hits.parquet")
	if err != nil {
		fmt.Println(err)
	}
	defer file.Close()

	reader := parquet.NewGenericReader[WebsiteHit](file)
	defer reader.Close()

	buffer := make([]WebsiteHit, 10)
	hits := []WebsiteHit{}

	for {
		rowsRead, err := reader.Read(buffer)
		if err != nil && err != io.EOF {
			fmt.Println(err)
		}

		for i := 0; i < rowsRead; i++ {
			hits = append(hits, buffer[i])
		}

		if err == io.EOF {
			break
		}
	}

	jhits, err := json.Marshal(hits)
	if err != nil {
		fmt.Println(err)
	}

	fmt.Fprintf(w, string(jhits))
}

func hitHandler(w http.ResponseWriter, r *http.Request) {
	userAgent := r.UserAgent()
	host, _, err := net.SplitHostPort(r.RemoteAddr)
	if err != nil {
		fmt.Println(err)
	}

	err = flush(host, userAgent)
	if err != nil {
		fmt.Println(err)
	}

	fmt.Fprintf(w, "hello!")
}

type WebsiteHit struct {
	ID        int64  `parquet:"id"`
	IP        string `parquet:"ip"`
	Path      string `parquet:"path"`
	UserAgent string `parquet:"user_agent"`
}

func flush(host string, userAgent string) error {
	outputFile, err := os.Create("tmp/hits.parquet")
	if err != nil {
		return err
	}
	defer outputFile.Close()

	writer := parquet.NewGenericWriter[WebsiteHit](outputFile)

	rows := []WebsiteHit{
		{ID: 1, IP: host, Path: "/some-path", UserAgent: userAgent},
		{ID: 2, IP: host, Path: "/my-page", UserAgent: userAgent},
	}

	_, err = writer.Write(rows)
	if err != nil {
		return err
	}

	return writer.Close()
}

func main() {
	fmt.Println("data lake 3000")
	http.HandleFunc("/", hitHandler)
	http.HandleFunc("/hits", hitsHandler)

	http.ListenAndServe("0.0.0.0:8080", nil)
}

Now, this is obviously a vastly over-simplified example but it's just meant to give you ideas. You could make several changes to this. For instance you could flush the changes ever N minutes or hours and then pull up the hits that you want to see. You could store it by instance-id - a single go unikernel can take a lot of traffic but if you're building a data lake around website hits chances are you have enough traffic that it's being load balanced. Again - this is just an example to get your thinking juices going.

Limitations

As we have focused on getting the initial klib out the door as of right now there are a handful of limitations - namely:

• You can not write to objects at a non-zero offset.
• Each write replaces the entire contents of an object.
• Anything over 4kb needs O_DIRECT set.
• It is not possible to use writev/pwritev when using direct i/o.
• Symlinks aren't supported.
• Renaming using RENAME_EXCHANGE is not supported.
• Truncating to a zero length is not supported.
• All mount points need to have an existing directory.

Many of these limitations are not real limitations - more that we just haven't added support for it yet.

As part of the changes the internal client that houses allocate_http_parser now allows a headers_only bool to be set for HEAD requests.

Like I said earlier this is one of those features where unikernels can really shine outside of the realm of increased performance and security and show how they can offer much cleaner abstractions than ones that might be used today. If you build something with this new feature let us know as we're always interested in hearing how folks are scaling out and re-architecting their infra with unikernels.