Don't Fragment My UDP Packets

Intro

One of the many perks our customers have over mere users is that they get to email me directly with bug reports and have the privilege of a personal response to the tune of something like "yes we need to add support - looking."

One of those recent emails was pointing out that we were silently ignoring the IP_PMTUDISC_DO flag which is an option you can pass to setsockopt. This sets the "dont fragment" flag. This option instructs the sending host and routers down the line not to fragment (eg: split up) IP packets. This setting is typically off by default.

Why would you need to fragment in the first place? Essentially there is a limit on the maximum size that each router is willing to deal with when your packets are going from host a to host b (and all the hops in between). The MTU (maximum transmission unit) size is the limit that each hop a packet takes fits into and if it does not than the packet needs to be fragmented (or in this case, when over the limit - rejected). A very typical size here will be 1500 bytes - but that is far from a 'given' - for instance on GCP it is 1460 bytes.

Why is this? Many reasons. If you are transiting across something like a VPN that encapsulates your packets your payloads might need to be smaller. If you set the DF bit and you run into a situation where fragmentation is necessary a ICMP Fragmentation Needed response should be sent back.

Another problem could arise if you are, for example, creating a new Zoom or Twitch like product. UDP is used quite often for things like voice/video so fragmentation and re-assembly aren't something you want as you can get delay and jitter. RTP which rides on top of UDP has its own fragmentation solution because of this.

Another characteristic arrives with UDP in that UDP has no guarantee of delivery and doesn't do sequencing either so if the DF flag is set you are explicitly stating that the packets either arrive or don't (cause there aren't fragments). I can see some of my ISP buddies starting to squint here.

Fragmentation also has a very long history of being an enabler of DDOS attacks such as the infamous teardrop attack. This is important to note as many IOT devices are still routinely abused in this manner because they use networking stacks that are not as robust or updated as ones found in servers. We've had to fix quite a lot of various security issues in the networking stack that we forked.

Unfortunately, IPV6, which we'll touch on later, also opens the door to various fragmentation related abuse such as device fingerprinting or out-right attacks such as IPS/firewall evasion, spoofing, ddos, mitm and drum roll remote code execution.

Before we go further though - let's look at a simple example of setting the DF flag:

package main

import (
    "fmt"
    "net"

    "golang.org/x/sys/unix"
)

func main() {
    conn, err := net.ListenUDP("udp", nil)
    if err != nil {
        panic(err)
    }
    defer conn.Close()
    rc, err := conn.SyscallConn()
    if err != nil {
        panic(err)
    }
    err = rc.Control(func(fd uintptr) {
        err := unix.SetsockoptInt(int(fd), unix.IPPROTO_IP, unix.IP_MTU_DISCOVER, unix.IP_PMTUDISC_DO)
        if err != nil {
            panic(err)
        }
    })
    if err != nil {
        panic(err)
    }
    bs := make([]byte, 7000)
    fmt.Println(conn.WriteToUDP(bs, &net.UDPAddr{
        IP:   net.IPv4(8, 8, 8, 8),
        Port: 53,
    }))
}

We try to ship 7000 bytes out via UDP. This will do as it is told unless the DF flag is set.

If IP_MTU_DISCOVER is set then IP_PMTUDISC_DO can be set as well to set the DF flag for outgoing UDP packets on IPV4.

Path MTU Discovery

So why isn't this flag just called 'Dont Fragment'? There is something called 'path mtu discovery' (eg: IP_PMTUDISC_DO) and essentially it is the process of discovering what the largest MTU one can use on a path without being fragmented. What happens is the DF bit is set and packets are sent, each one increasing in size, until you start getting ICMP failure messages back. Of course this isn't a perfect solution because ICMP gets blocked all over the place for security reasons. Funnily enough this is actually one reason why ipv6 adoption is still lagging - it is hard to mandate something when people are already intentionally blocking it.

Of course what do engineers do when they run into issues with existing standards? They create new standards! The Packetization Layer Path MTU Discovery or PLPMTUD is one possible answer to this.

Things have changed over the years as we continue to transition into ipv6. One of the big differences in IPV6 is that fragmentation can only take place at the source versus somewhere along the route. That is to say ipv6 acts as if DF is always set. So instead of having option fields, ipv6 has extension headers one of which is a fragment extension header which looks like so:

--------------------------------------------------------------
| next header | reserved | fragment offset | result | M flag |
--------------------------------------------------------------
|                       identification                       |
--------------------------------------------------------------

Parts of ipv6 are fragmentable such as the authentication header while others such as the routing header are not. Note: Despite popular opinion saying otherwise, ipsec is actually not mandatory in ipv6. One must also be careful with the fraggle rock header - j/k - was making sure you're a human. IPV6 also mandates a minimum MTU of 1280 bytes which helps with this.

Earlier (and by earlier I mean the early 90s) setting the DF flag was perceived as something that could potentially produce too much overhead, especially for a lot of the protocols such as SMTP or DNS. Keep in mind we didn't really have 56k modems until 1998. That is a far cry from the 100+ http requests and 50mb page loads for "modern" webpages today or all the Zoom/Netflix from 2020.

Since it is still going to be a while before everyone is predominantly using IPV6 - you might want to say - don't fragment my UDP packets.