Extended Attributes

When making a package for opencloud I noticed that we were missing the extended attribute family of syscalls. OpenCloud is designed not to have to rely on a separate database and stores metdata directly into files with extended attributes, falling back on external metadata files if necessary.

What are Extended Attributes

Extended attributes are key/value pairs that can be stored with individual files and directories. They can be set via setxattr, lsetxattr, fsetxattr, and retrieved via getxattr, lgetxattr, or fgetxattr.

A key name can be 255 bytes and a value can be up to 64k. Attribute names are a part of namespaces. In linux you have to use one of the namespaces: security, system, trusted, and user.

Use Cases of Extended Attributes

There are quite a few different uses cases that extended attributes have found over the years. For example Darwin uses this for the quarantine flags - you might've had to remove the attribute in the past few years for a downloaded binary on osx:

xattr -d com.apple.quarantine

SELinux also uses extended attributes in the security namespace.

Attributes can come in handy for new files as well. Sometimes metadata is attached - such as if you downloaded a file it could set the url of where you downloaded it from. You could also set things like mime-types or checksums - the checksum feature is fairly useful. Attributes can also used for things like retention and lifecycle policies.

Extended Attributes Example

A simple example of this in go using the xattr pkg might look like the following:

package main

import (
    "fmt"
    "os"

    "github.com/pkg/xattr"
)

func main() {
    fpath := "test.txt"
    attrName := "user.extra"
    attrValue := []byte("bob was here")

    file, err := os.Create(fpath)
    if err != nil {
        fmt.Println(err)
    }
    file.Close()

    err = xattr.Set(fpath, attrName, attrValue)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("set %s on %s\n", attrName, fpath)

    getValue, err := xattr.Get(fpath, attrName)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Printf("we got back: %s\n", string(getValue))
}

You can then retrieve the attributes via getfattr like so:

eyberg@venus:~/xa/rxa$ getfattr -d -- test.txt
# file: test.txt
user.extra="bob was here"

Zero Length Values

We ended up adding support for zero length buffers for our buffer implementation. setxattr for instance, will call allocate_string which internally calls allocate_buffer, however as the man page clearly states:

Why? You might wish to set a zero length value for a boolean type (for instance if the attribute is set than it's true), or where you only need to know that an attribute exists vs having a value attached. This obviously can help save space.

So next time when you are looking at pushing out a single binary and you are debating whether or not you need a database or not - remember sqlite isn't the only thing you can reach for. Sometimes you can store the data you need right on the files you are working with to begin with.