Slide into the HOV Lane with NAPI ID and TCP Keep-Alive Options

We recently made some changes to allow more support for various TCP KEEP_ALIVE options and for NAPI ids.

Let's chat TCP KEEP_ALIVE first. You probably know about HTTP KEEP_ALIVE which allows you to issue multiple http requests on the same connection without having to go through another handshake. There is a similar, albeit totally different capability, that operates at a lower layer of the stack on TCP called... wait for it... TCP KEEP_ALIVE.

To enable this you simply toggle TCP_KEEPALIVE on with setsockopt:

int keepalive = 1;
setsockopt(sockfd, SOL_SOCKET, SO_KEEPALIVE, &keepalive, sizeof(keepalive));

When this is enabled it sends periodic ack's to keep the connection alive. We've had support for it for a while but just recently added more of it's tunables.

TCP_KEEPIDLE is the wait before sending an initial probe. The default value is 2 hours. While this is great for normal desktop like usage there are variety of situations where you might want a much lower value to bet set such as in a router, gateway, or firewall.

int idle_time = 60; 
setsockopt(sockfd, IPPROTO_TCP, TCP_KEEPIDLE, &idle_time, sizeof(idle_time));

TCP_KEEPINTVL is used to set the wait period in between new probes sent to check for liveliness. The default is 75.

int interval = 3; 
setsockopt(sock, IPPROTO_TCP, TCP_KEEPINTVL, &interval, sizeof(interval))

TCP_KEEPCNT adjust the number of probes that are to be sent before the connection should be dropped.

int max_probes = 3; 
setsockopt(sock, IPPROTO_TCP, TCP_KEEPCNT, &max_probes, sizeof(max_probes)

The default, at least on my system is 9:

sudo sysctl -a | grep net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_probes = 9

What is NAPI

Now let's discuss NAPI ids.

NAPI, or 'new api', is the event handling framework used in linux. It'll batch multiple packets for each interrupt.

For heavy workloads one can use the NAPI_ID to effectively pin a thread to a receive queue. From there instead of using interrupts it can "busy poll" the queue for new incoming data. So a firewall or a HFT setup might use this.

NAPI_IDs are a part of Application Device Queue technology.

What is the (Application Device Queue) ADQ

Application Device Queues is a technology that has dedicated hardware network queues that applications can use. Using napi ids is a part of this. If you want to use this you'll need something like ena or gvnic *and* something that supports multiple queues.

To get the napi id you would call getsockopt:

int get_napi_id(int fd) {
    int napi_id = -1;
    socklen_t len = sizeof(napi_id);
    
    if (getsockopt(fd, SOL_SOCKET, SO_INCOMING_NAPI_ID, &napi_id, &len) == -1) {
        perror("getsockopt SO_INCOMING_NAPI_ID failed");
        return -1;
    }
    
    return napi_id;
}

Wrap

All of these options are .. optional and there's a good chance that none of our readers are ever going to use these but perhaps you're designing the next great firewall using the nanos unikernel and if that's you - perhaps this helps.