Foo over UDP

Tunneling protocols are increasingly important in modern networking setups. By tying distant networks together, they enable the creation of virtual private networks, access to otherwise-firewalled ports, and more. Tunneling can happen at multiple levels in the networking stack; SSH tunnels are implemented over TCP, while protocols like GRE and IPIP work directly at the IP level. Increasingly, though, there is interest in implementing tunneling inside the UDP protocol. The "foo over UDP" (FOU) patch set from Tom Herbert, which has been pulled into the net-next tree for 3.18, implements UDP-level tunneling in a generic manner.

Why UDP? Just about any network interface out there has hardware support for UDP at this point, handling details like checksumming. UDP adds just enough information (port numbers, in particular) to make the routing of encapsulated packets easy. UDP can also be made to work with protocols like Receive Side Scaling (RSS) and the Equal-cost multipath routing protocol (ECMP) to improve performance in highly connected settings. The advantages of UDP tunneling are enough that some developers think it's going to become nearly ubiquitous in the coming years.

Packet encapsulation and tunneling over UDP is a relatively straightforward concept to understand. Suppose a simple TCP packet is presented to the tunneling interface:

This packet has the usual IP and TCP headers, followed by the data the user wishes to send. The encapsulation process does something like this:

At this point, the packet looks like a UDP packet that happens to have a TCP packet buried within it. The system can now transmit it to the destination as an ordinary UDP packet; at the receiving end, the extra headers will be stripped off and the original packet will be fed into the network stack.

Configuring a FOU tunnel will typically be a two-step process. The transmit and receive sides have been separated, a feature which, among other things, allows asymmetric setups should anybody want them. On the receive side, configuration is really just a matter of setting up a UDP port to be the recipient of encapsulated packets. The new "fou" subcommand is intended for this purpose:

    ip fou add port 5555 ipproto 4

This command sets aside port 5555, saying that packets arriving there will have protocol 4, which is IP encapsulation. Packets received on that port will have the encapsulation removed; they will then be fed back into the network stack for delivery to the real destination.

Things are a little more complicated on the transmit side, since the destination address must be provided and transmission needs to work with existing encapsulation protocols. A typical command might look like:

    ip link add name tun1 type ipip \
       remote 192.168.1.1 local 192.168.1.2 ttl 225 \
       encap fou encap-sport auto encap-dport 5555

This command will set up a new virtual interface (tun1) configured for IPIP encapsulation. The source port for packets is left to the network stack to decide, but the destination port will be 5555. Of course, one has to get the encapsulation protocol to actually use this interface. At the moment, support for doing that has been added to the IPIP, SIT (an IPv4-to-IPv6 tunneling protocol) and GRE (used for virtual private networks) protocols.

Some numbers posted in the patch set show significant performance increases for the SIT and IPIP protocols; performance with GRE was roughly equivalent to the no-FOU case. So this feature has a clear potential to speed things up by taking advantage of existing optimizations around UDP transmission and receipt. The nice thing is that no special hardware support is required; current hardware can handle UDP just fine. So it is a simple solution that will work on existing systems — and it should be available in the 3.18 kernel release.

Index entries for this article
Kernel	Networking

---

to post comments

Foo over UDP

Posted Oct 2, 2014 0:48 UTC (Thu) by luto (subscriber, #39314) [Link] (6 responses)

I don't understand. If I type:

# ip fou add port 5555 ipproto 4

then incoming UDP packets destined for port 5555 will have the UDP header stripped and the contents will be interpret as... what?

From the diagram, it looks like it's just an IP header in the UDP packet. But what does ipproto 4 have to do with it?

Foo over UDP

Posted Oct 2, 2014 3:54 UTC (Thu) by Fowl (subscriber, #65667) [Link] (4 responses)

So that it knows to treat them as IPv4 packets? Equivalent to the EtherType field in the Ethernet header, I'd imagine.

Foo over UDP

Posted Oct 2, 2014 4:59 UTC (Thu) by luto (subscriber, #39314) [Link] (3 responses)

Is it?

The number 4 could refer to IP protocol 4, which is ipip (in the same numbering system in which TCP is 6) or to IPv4.

If it's IPIP, then does 4 mean that the UDP payload is to be treated as an IPIP payload? If so, then would 6 mean TCP on UDP on IP (as opposed to TCP on IP on UDP on IP)?

If it's the 4 in IPv4, then 6 would mean IPv6 on UDP on IPv4.

I found this bit of the article to be unclear. Looking at the patch description, I'm pretty sure it's the former. I think that the TX/RX split is a bit odd, though.

Foo over UDP

Posted Oct 2, 2014 15:47 UTC (Thu) by luto (subscriber, #39314) [Link] (2 responses)

And, to answer my own question for real:

On receive, the UDP header will be stripped and the payload will be processed as though it the IP sub-protocol specified in the fou configuration. If that's protocol 4 (IPIP), then the next header will be another IP header, which will be matched against an ip tunnel config.

So the RX and TX paths aren't quite as split as the article made me think.

Foo over UDP

Posted Oct 2, 2014 20:55 UTC (Thu) by jhoblitt (subscriber, #77733) [Link] (1 responses)

Why wouldn't the nested IP header have a valid protocol field?

Foo over UDP

Posted Oct 2, 2014 21:02 UTC (Thu) by luto (subscriber, #39314) [Link]

It does, but I think you're confused for the same reason I was.

That "4" means that this is an IPIP tunnel (so the next header is IP) as opposed to, say, a GRE tunnel with a different format.

Foo over UDP

Posted Jun 25, 2019 10:25 UTC (Tue) by elic307 (guest, #132352) [Link]

As far as I understand, fou allows for generic encapsulation; it can be IP or Ethernet.
The receiver does not know what is encapsulated so you have to tell it explicitly. When you say ipproto 4 you let it know that IPv4 is encapsulated.

Foo over UDP

Posted Oct 2, 2014 3:23 UTC (Thu) by raven667 (subscriber, #5198) [Link] (4 responses)

This seems fundamentally a NAT problem, maybe the designers of IP should have given up and put port/service numbers in the IP header instead of the individual protocols, effectively making everything encapsulated in UDP, and then removing ports from TCP. It seems that in modern networking we either have stuff tunneled in UDP (VXLAN, IP, etc.) or over TCP on port 80/443 (VPNs, every kind of RPC is now JSON or XML over HTTP), every other port and IP protocol doesn't exist anymore and can't work because of NAT and overzealous firewalls.

Foo over UDP

Posted Oct 2, 2014 17:47 UTC (Thu) by drag (guest, #31333) [Link]

Tunneling is interesting in itself to work around some of the natural deficiencies in IP networking.

For example with mobile networking I can setup a laptop or smartphone to use a 'tinc'-based VPN. This allows for more of a 'mesh' style VPN networking rather then a traditional 'hub and spoke' style networking. Tunnels can be setup to work on a best-effort basis. So if I am on my private network it connects to the vpn to internal addresses, when I am on a public network it connects to the public points I have setup. It can use udp or tcp, etc etc.

What this gets me (at least in theory) is then a continuous, persistent, private network connection regardless of were I am at. That way I, although it isn't perfect, can keep a persistent network connection for things like ssh and whatnot. I don't end up using it a whole lot this way, but it's pretty handy.

Tunneling can actual alleviate a whole host of issues associated with TCP/IP style networking. Just like with virtual machines separating the logical from the physical has it's own benefits.

Foo over UDP

Posted Oct 2, 2014 19:35 UTC (Thu) by josh (subscriber, #17465) [Link] (2 responses)

With IPv6, I wonder to what extent we need port numbers. Just give each machine a range of addresses large enough to give a unique one to each of its services, and let each address correspond to exactly one service.

Foo over UDP

Posted Oct 3, 2014 16:30 UTC (Fri) by mbunkus (subscriber, #87248) [Link] (1 responses)

You can open ports as non-root, but you cannot assign IPv6 addresses. It would be very bad if each daemon required network admin capabilities.

Additionally selecting the correct source address for outgoing connections becomes interesting in multiple-address situations. For example, if you have a DNS master server running on a machine with three IPv6 addresses then you most likely have some slave servers somewhere else which are configured to accept notifications from certain IPv6 addresses only. Therefore you have to tell your DNS server which source address to use for outgoing packates (yes, this comes from my own experience). It increases administrative work by a considerable amount.

Foo over UDP

Posted Oct 3, 2014 21:28 UTC (Fri) by josh (subscriber, #17465) [Link]

You could run each of those services in containers to which you assigned the corresponding service address.

OT: SSH over UDP

Posted Oct 2, 2014 9:13 UTC (Thu) by debacle (subscriber, #7114) [Link] (2 responses)

Btw. is it possible to run SSH (or a SSH tunnel) over UDP? Just curious.

OT: SSH over UDP

Posted Oct 2, 2014 9:33 UTC (Thu) by fpletz (guest, #88477) [Link] (1 responses)

You are probably looking for mosh: http://mosh.mit.edu/

OT: SSH over UDP

Posted Oct 9, 2014 14:44 UTC (Thu) by nye (subscriber, #51576) [Link]

>You are probably looking for mosh: http://mosh.mit.edu/

I can't recommend this highly enough. It gives a great experience even over unreliable (eg. mobile or crappy NAT) links where ssh is largely unusable.

Foo over UDP

Posted Oct 3, 2014 14:32 UTC (Fri) by Baylink (guest, #755) [Link] (8 responses)

"they allow access to otherwise firewalled ports" our fearless editor says in the lede to this article. Says it like its a good thing, you understand...

Does that strike anybody else as a bit beyond the pale given the tenor of the front page in this week's edition? We firewalled those ports for a reason...

Access to firewalled ports

Posted Oct 3, 2014 15:24 UTC (Fri) by corbet (editor, #1) [Link] (6 responses)

Here at LWN, we use tunnels all the time to get at stuff that we do not wish to expose to the world as a whole. That's what VPNs do too. It's not an abusive use at all.

Next time you find yourself in a hotel that thinks that IMAP, say, is a hostile protocol, you might appreciate this feature more as well.

Access to firewalled ports

Posted Oct 4, 2014 11:26 UTC (Sat) by gerdesj (subscriber, #5446) [Link]

For a similar reason I generally have my OpenVPN servers listen on 443/tcp. Client traffic looks like a https session from a web browser.

Hotels etc generally allow web traffic through unmolested. When operating on a site with a mandatory web proxy then simply use the proxy as well - if necessary via cntlm.

The main problem with tcp in tcp tunnelling is exponential standoffs making latency potentially horrendous but the general increase in bandwidth available nowadays makes this less of a problem than in the past. Maybe I ought to look into 53/udp instead 8)

Access to firewalled ports

Posted Dec 11, 2014 2:40 UTC (Thu) by Baylink (guest, #755) [Link] (4 responses)

Certainly, Jon.

But "using a VPN" is a different thing entirely from "tunneling a protocol through a port number not assigned for that purpose so that people charged with maintaining security cannot execute their responsibilities", surely?

Access to firewalled ports

Posted Dec 11, 2014 17:03 UTC (Thu) by nix (subscriber, #2304) [Link] (3 responses)

It's only a matter of viewpoint. My mother's ISP has a transparent proxy/firewall combo in the way that blocks everything but HTTP and HTTPS, specifically calls out vaguely stated 'security' as their reason, and is completely unresponsive to requests to unblock anything else. To them, my use of httptunnel to get my SSH session (and thus much else) through their stupid block is precisely "tunneling a protocol through a port number not assigned for that purpose so that people charged with maintaining security cannot execute their responsibilities". To me, it's "getting my job done while I'm at my parents' house" and "unbreaking the Internet".

Access to firewalled ports

Posted Dec 11, 2014 19:35 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (2 responses)

Would you mind outing the offending ISP so that those with the luxury may avoid it in the future? :)

Access to firewalled ports

Posted Dec 14, 2014 17:02 UTC (Sun) by nix (subscriber, #2304) [Link] (1 responses)

It's Tooway and all its resellers, a UK satellite broadband provider. They are very expensive (hundreds of quid for installation, and as much for 5GiB/month as most people would expect to pay for 100+), but if you use them it's because BT in their infinite wisdom have decided you can't get ADSL in your locality (in my parent's case, the exchange is full, and if anyone asks for ADSL they are told to pay the £250,000+ upgrade costs first) and nothing else is available. So the only people using this are a captive market, and Tooway knows it.

They actually work quite well as a service, speed-of-light-induced latency being what it is: the only real problem with them is that ToS and filtering, blocking virtually everything and thus making me feel like I'm breaking some rule whenever I dare log in to work from my parents' house, let alone set up a tunnel so I can do a git pull or some other dark and dire activity like that.

As for where this is called out in their terms of service... well, I can't even *find* the terms of service any more, they're so well hidden: when last I found them several years ago they marked almost everything as FORBIDDEN including things like 'terminal emulation': only HTTP and HTTPS were marked PERMITTED. Note: the things calling themselves terms of service on Tooway's website are for the website, not for the service. For all I know the TOS has changed, in which case this comment is mostly of historical interest. :)

I thought maybe the prohibition was due to bandwidth considerations, given that this is satellite broadband, but they allow iPlayer... so I guess not. Maybe they're caching the hell out of something and don't like things that break it, but even so, the only thing they'd save on is bandwidth to and from their Italian downlink site, not to/from orbit, and to be honest if they can afford to contract with Eutelsat for satellite service they can damn well afford decent bandwidth to their downlink site!

It's not like they can rely on everyone viewing the same thing in iPlayer at the same time: they'll be using limited downlink bandwidth from orbit for every independent viewer. This is not multicast! And terminal service does not strain it, ffs. Certainly not more than all those ack packets for an iPlayer stream (and yes, the acks do flow back, or at least *something* flows back while you're watching iPlayer, so they don't have a special optimization for that case).

Access to firewalled ports

Posted Dec 26, 2014 15:17 UTC (Fri) by Baylink (guest, #755) [Link]

The argument could be made -- and would and should be made -- that this isn't "internet service" and either

1) buyers should have beworn, or
2) they should get sued for not warning people that what they were advertising wasn't what they were selling,

but I understand your point of view (which, I suspect, is roughly that "retail customers can't be expected to understand the difference" and "I have shit to *do*". :-)

We weren't really talking about this particular edge case, though, I don't think.

Foo over UDP

Posted Oct 3, 2014 17:20 UTC (Fri) by noxxi (subscriber, #4994) [Link]

> We firewalled those ports for a reason...

Unfortunatly simple firewalling of ports is not enough today. Because most ports except http(80) and https(443) are closed today everybody is tunneling through these ports already so if you want to have security today you need to look at the application layer.

FOU just encasulates layer 3 traffic into layer 4 to pass through existing systems. This is similar to Websockets which encapsulate layer 4 capabilities into layer 7. At the end you have the same or worse security problems then before, but need more bandwith and beefier hardware to compensate for the overhead.

Foo over UDP

Posted Oct 4, 2014 7:59 UTC (Sat) by tomgj (guest, #50537) [Link] (1 responses)

When the article says that "support has been added to the IPIP, SIT [...] and GRE [...] protocols", I am not sure whether this means that support for UDP encapsulation is in the definition / specification of those protocols, or if it means that support has been added to the implementations of those protocols within Linux. Can anyone clarify this?

Foo over UDP

Posted Oct 4, 2014 12:36 UTC (Sat) by corbet (editor, #1) [Link]

The latter, mainly; the implementations have been augmented with FOU support. There is an RFC for GRE over UDP, at least, and the Linux code follows it.

Foo over UDP

Posted Oct 4, 2014 17:11 UTC (Sat) by malor (guest, #2973) [Link] (4 responses)

>SSH tunnels are implemented over TCP, while protocols like GRE and IPIP work directly at the IP level. Increasingly, though, there is interest in implementing tunneling inside the UDP protocol.

The article doesn't hit on this, but one major reason to tunnel over UDP instead of TCP is when you're on a poor quality network.

When you're on fast, reliable bandwidth, TCP-over-TCP is fine. But as soon as you start getting packet loss, you end up with a real mess: you have two levels of TCP trying to handle retransmits and flow control. Your connection will go completely to heck with just a little bit of loss, as the inner layer fights with the outer layer, both timing out and demanding repeats of the same packets. This can cascade into total failure very, very quickly.

Tunneling over UDP avoids that whole mess; it's much, much more reliable on a poor network. This is the primary reason why I use OpenVPN in UDP mode whenever I can, rather than SSH tunneling. It's much faster on your typical crappy hotel or convention network; you can usually maintain a somewhat useful connection even on low quality bandwidth, where SSH tunneling can easily croak and die if the connection is even a little spotty.

This tunneling method would be more interesting to me, personally, if it supported good solid encryption. Plaintext tunneling, post-Snowden, strikes me as a bad idea. Yes, you can run encryption at the TCP layer, but you're giving away more data than you need to. Among other things, packet inspection engines could easily see the inner layer, and throttle or otherwise interfere with the connection based on its content, and of course more of your habits can end up in the NSA's databases.

Foo over UDP

Posted Oct 7, 2014 0:12 UTC (Tue) by jonabbey (guest, #2736) [Link] (3 responses)

To ask a silly question, how are you guaranteed to receive the UDP packets sent back to you? Why wouldn't firewalls be filtering those out if you were in a hotel, etc.?

Foo over UDP

Posted Oct 7, 2014 0:23 UTC (Tue) by malor (guest, #2973) [Link] (2 responses)

Most stateful firewalls will set up an implicit accept rule allowing UDP traffic back in for a couple of minutes after you send a packet out. (in iptables, this entry goes into the ESTABLISHED table.) As long as a packet shows up at least once every couple of minutes, the connection will normally stay alive.

Details, of course, are dependent on what OS is being used on the router, its NAT engine, and its configuration. But usually, most of the time, replies are allowed back in.

I don't think I've ever been in a network that blocked OpenVPN. If you use a password, rather than an SSL key exchange, the connection just looks like random bytes. If you're using SSL certs, packet inspection can see the connection setup and key exchange, so they can tell it's a VPN. Inspection engines could block either scenario, and firewall rules could block the UDP traffic, but I've never seen either done in a guest-oriented network.

Foo over UDP

Posted Oct 10, 2014 11:15 UTC (Fri) by mathstuf (subscriber, #69389) [Link] (1 responses)

Is there a way to hide the SSL key exchange with a password? I guess a two-factor setup would be possible as well. I'll have to look into that.

Foo over UDP

Posted Oct 12, 2014 12:38 UTC (Sun) by malor (guest, #2973) [Link]

I don't remember seeing anything about that in the documentation, but I imagine you could probably hack the code to make it work how you wanted.

How easy that would be, though, I dunno.

Foo over UDP

Posted Nov 11, 2015 16:15 UTC (Wed) by tysonite (guest, #105305) [Link] (3 responses)

Am I right that I can't encapsulate SCTP packets in today's implementation?
For instance, to meet requirements as stated here: https://tools.ietf.org/html/rfc6951

Foo over UDP

Posted Nov 11, 2015 18:13 UTC (Wed) by flussence (guest, #85566) [Link] (2 responses)

I thought the existing SCTP implementation in the kernel already supported UDP tunnelling without this?

Foo over UDP

Posted Nov 11, 2015 19:56 UTC (Wed) by tysonite (guest, #105305) [Link] (1 responses)

I am not aware of how it is possible. Could you elaborate please?

Foo over UDP

Posted Nov 12, 2015 20:56 UTC (Thu) by flussence (guest, #85566) [Link]

Well I went looking for the documentation and came back empty-handed. My bad.

lksctp-tools and the kernel code make mention of tunnelling udp/tcp inside sctp, but not vice-versa. I may have been misremembering based on that.