Tunnels Within Tunnels: One Weird Trick for Free IP Transit

2023-11-16 update: John Elliott published a post with router configuration details

Imagine you took your homelab hobby to the next level by acquiring an ASN. You don't have much money, but you've still managed to connect to an IXP and peer with other ASs over the exchange fabric. You'd like to connect your network back to sites scattered across a metro region, often with only consumer internet connections because other types of service are too expensive.

You may want to network the sites together and may also want the sites' internet-bound traffic to exit through your own AS (there are a number of reasons you might want to do this --see the last section). Hence, you decide to set up VPN tunnels between your sites and your network equipment at the IXP.

Unfortunately, the large regional ISPs servicing your sites will not peer with you because you are small fry, so the tunneled traffic doesn't reach your router at the IXP via settlement-free peering relationships, instead it goes through a paid upstream transit provider. You don't want to increase the traffic through your transit provider more than you have to because it's expensive and limited. Is there a solution?

Recently, while tinkering with VPN tunnels with other volunteers from the Seattle Community Network (eg. John Elliott), I had a sudden flash of inspiration. An absurd idea emerges...

Enter the Hurricane Electric Free IPv6 Tunnel Broker

The Hurricane Electric Free IPv6 Tunnel Broker is a free service that HE provides out of the goodness of their hearts. If your ISP's IPv6 service is lacking or nonexistent, no problem! HE will generously lend you a /64 IPv6 address block and provide you with a tunnel endpoint address where you can send IPv6 using RFC-compliant 6to4 technology!

For costing nothing, HE's IPv6 service is amazing¹. Here in Seattle on a Centurylink connection HE's tunnel broker generally outperforms Centurylink's own 6rd service. And unlike 6rd, the HE IPv6 block is static, making it far more useful for running servers at home.

Besides providing free IPv6 forwarding, Hurricane Electric also freely peers with other networks at exchange points, no matter the size.

Hmm... Could we use HE's network to bypass an upstream transit provider?

It turns out yes, and the performance is much better than any of us expected.

A disclaimer: Don't abuse this. There are also some details about why this is kind of technically icky². But we had to see whether it worked (and how badly).

Read on...

The Initial Configuration (via Paid Upstream Transit Provider)

The inital tunneling config is pretty straightforward. We're running a VPN tunnel endpoint on one of our servers at the IXP. Internet-bound traffic is NATted with an address from our public IPv4 block.

Routers at the remote sites connect to the VPN endpoint at the IXP. Internet-bound traffic is routed to the IXP via the tunnel, unless the tunnel is down because we messed something up, in which case traffic bypasses the tunnel and directly egresses via the consumer ISP connection as a fail-over.

We're using WireGuard for these tunnels. So far WireGuard has been simple to configure (though there are a few quirks), fast, and ...fast. MikroTik's RouterOS 7, released in the last year, supports WireGuard, allowing us to finally use it on our MikroTik hardware. Other volunteers, John Elliott and Esther Jang, did the bulk of the work figuring out the WireGuard and firewall configurations.

John used iperf to achieve a respectable 860 Mbps over the tunnel between his home 940 Mbps GPON connection to the VPN server.

These are two traceroutes from different consumer ISPs showing that the path taken to our test server includes a hop through an upstream transit provider (AS1299):

$ mtr -w -r -z -c 5 23.146.168.1
HOST: foo                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
1. AS???    rb5000.home.arpa                      0.0%     5    2.1   1.0   0.7   2.1   0.6
2. AS209    tukw-dsl-gw69.tukw.qwest.net          0.0%     5    3.2   3.0   2.9   3.2   0.2
3. AS209    63-226-198-33.tukw.qwest.net          0.0%     5    3.0   3.3   2.1   5.1   1.1
4. AS???    ???                                  100.0     5    0.0   0.0   0.0   0.0   0.0
5. AS3356   4.69.219.210                          0.0%     5    3.5   3.6   3.3   3.9   0.3
6. AS1299   sea-b1-link.ip.twelve99.net           0.0%     5    3.5   3.8   3.5   4.3   0.3
7. AS1299   doof-ic-375811.ip.twelve99-cust.net   0.0%     5    3.5   3.4   3.1   3.6   0.2
8. AS54429  seattlecommunitynetwork.org           0.0%     5    3.8   3.6   3.5   3.8   0.1

A traceroute from a Centurylink residential connection to our router at the SIX transits 1299.

$ mtr -w -r -z -c 5 23.146.168.1
HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    router.asus.com                                    0.0%     5    1.2   3.2   1.2   9.2   3.4
  2. AS???    100.92.123.67                                      0.0%     5    8.9  11.6   8.9  16.6   3.0
  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   13.1  11.8   9.6  14.1   2.0
  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5    9.9  11.4   8.6  15.4   2.6
  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5    9.7  17.4   9.7  29.9   7.8
  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net         0.0%     5   10.2  15.0  10.2  31.1   9.0
  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   12.5  10.9  10.1  12.5   1.0
  8. AS7922   50.208.233.194                                    40.0%     5   32.0  25.1  20.8  32.0   6.0
  9. AS???    ???                                               100.0     5    0.0   0.0   0.0   0.0   0.0
 10. AS1299   doof-ic-375811.ip.twelve99-cust.net                0.0%     5   15.1  14.7  10.4  25.1   6.1
 11. AS54429  seattlecommunitynetwork.org                        0.0%     5    9.8  15.2   9.8  35.5  11.3

A traceroute from a Comcast residential connection³ to our router at the SIX hops through 1299 too.

This cutesy diagram shows roughly the same information as the traceroutes but with a larger font and some arrows.

The Path to the Hurricane Electric IPv6 Tunnel Broker

Let's do a few more traceroutes to see the path the traffic takes. We'll start with an IPv6 traceroute over the HE IPv6 tunnel broker to Seattle Community Network's router (2602:fd94::1) at the Seattle Internet Exchange:

[asdf@rb5] > /tool/traceroute 2602:fd94::1
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
#  ADDRESS            LOSS  SENT  LAST     AVG  BEST  WORST  STD-DEV
1  2001:470:a:f00::1  0%     101  4ms      4.4  2.8   31.9   2.9
2                     100%   101  timeout
3  2602:fd94::1       0%     100  3.2ms    4    2.1   67.7   6.5

(First hop IP address slightly redacted)

This traceroute doesn't show hostnames or AS numbers, but that's ok. What's notable is that this traffic seems to hop right into our network, after a mystery hop at hop #2 that I suspect is some HE-internal stuff we can't see. It appears traffic enters HE's network via their IPv6 tunnel broker then passes through the peering connection between HE and our router at the IXP, rather than going through a transit provider. This makes sense because the peering link should have fewer hops, seeing as the traffic is already inside HE's network.

That last traceroute was run inside the tunnel so it doesn't show any of the hops that happen before the packets reach HE. To see the path taken by the tunnel itself, in other words the outside view, we can run traceroute to the IPv4 address of our HE tunnel broker endpoint 216.218.226.238.

Here's an example of an outside-the-tunnel traceroute to the HE tunnel broker endpoint taken from a Comcast residential connection. Pay attention to the AS numbers, you can see that Comcast (AS7922) connects to Hurricane Electric (AS6939) without an intermediate hop. Looks like they peer with each other.

$ mtr -w -r -z -c 5 216.218.226.238
HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    router.asus.com                                    0.0%     5    2.3   3.0   1.2   7.5   2.6
  2. AS???    100.92.123.67                                      0.0%     5   10.2  11.1   8.7  14.4   2.1
  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   16.0  11.5   9.8  16.0   2.6
  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5   12.1  10.9   9.0  12.2   1.4
  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5   11.9  13.7   9.8  20.1   3.9
  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net        60.0%     5   13.6  12.1  10.5  13.6   2.1
  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   21.9  14.0  10.2  21.9   5.0
  8. AS7922   50.208.233.194                                    20.0%     5   28.1  20.7  10.5  31.1  10.4
  9. AS6939   tserv1.sea1.he.net                                 0.0%     5   11.7  10.4   9.8  11.7   0.8

An mtr traceroute from a Comcast residential connection to the Hurricane Electric Free IPv6 Tunnel Broker

Cramming Tunnels Inside Tunnels

The last piece is getting the user traffic to take the path through HE shown above. Briefly, we

• Set up HE free IPv6 tunnels at user sites.
• Send site traffic over an IPv6-addressed VPN to our network.

In other words, we put a VPN tunnel inside HE's 6to4 tunnel. Our inner VPN tunnel must be destined to an IPv6 address so that it goes inside the HE 6to4 tunnel. However, the traffic inside our innermost tunnel can still use IPv4.

Traffic takes a path that looks something like this:

The "???" network represents hops between a residential ISP's network and HE. In our case on Centurylink and Comcast there's a direct connection⁴ to HE (AS6939), but other ISPs might go through a transit provider. Also, AS1299 and AS395823 are no longer part of the traffic's path so they're grayed out.

At this point, we've discovered a path from other networks back to our network at the SIX that doesn't involve a transit provider. This traffic doesn't cost us anything (because we peer with HE) and it might not cost HE anything either (because they're probably peering freely with big ISPs).

Performance

With WireGuard as the innermost tunnel, the packets look something like this mess:

Surprisingly, this gross hack only increases overhead by 100 bytes (okay, that is a lot, but I was expecting more). Here's the breakdown:

• The 6to4 tunnel contributes 60 bytes of overhead from a 20-byte IPv4 and a 40-byte IPv6 header.

• WireGuard adds another 40 bytes via its 8-byte UDP header and 32 bytes of protocol-specific data (4-byte type + 4-byte key index + 8-byte nonce + 16-byte authentication tag)

I'm not counting overhead from the outer PPPoE connection and the TCP/IP headers in the user data because these headers are present regardless.

The performance of putting a tunnel inside another tunnel will probably not be as good as not putting a tunnel inside another tunnel. But how bad is it?

John, who did the majority of the implementation, set out to investigate just how bad this idea actually was.

He initially squeezed a measly 230 Mbps over a connection advertised as being capable of 940 Mbps, but with some careful tuning of the MTU and iperf parameters, pushed 680 Mbps of UDP traffic through this double tunnel setup (see the middle pane in the screenshot below).

Top pane: pushing the maximum 940 Mbps of traffic outbound from John's home Centurylink GPON service. Middle Pane: 680 Mbps of UDP traffic received at our router at the SIX! Bottom pane: Graph of traffic through Bond0.510 --our router's connection with the SIX fabric, and where we peer with HE, indicating this traffic is going over a peering link rather than a transit link.

Image credit: John Elliott

His MikroTik RB5009 sat around 70% CPU during this abuse, indicating the hardware is sufficient to make line rate (at least for large packet sizes).

Why Tunnel at All?

Even straightforward VPN tunnels cut into performance. One must wonder why tunnel at all?

One might use VPN tunnels for financial reasons. Virtual layer 2 circuits, eg. via metro ethernet or similar technologies, are expensive (let's not even mention running one's own fiber or fixed wireless). Consumer-grade internet connections are cheap by comparison.

However, this assumes that one would want to network sites that already have internet connections together at all. One reason is to monitor equipment, allow secure remote management, and provide network services (dns, ntp, etc.) to both equipment and user devices.

I mentioned in the beginning that it could be desirable for traffic from these sites to exit from a different ASN, opposed to the consumer ISP connection. Among other things, this allows the ASN owner to receive and handle DMCA complaints themselves (look for a link here to our 2021 City of Seattle report that describes the issue). You could also use split tunneling to shed load from the VPN by routing popular streaming services outside of the tunnel, for example.

Acknowledgments

I owe thanks to Seattle Community Network for providing the infrastructure necessary to explore this interesting hypothetical.

Thanks to John Elliott for doing the hands-on-keyboard work configuring the network gear, running performance tests, and providing the traceroutes and screenshots.

Thanks to Esther Jang for pre-work in scoping the problem, helping set up the Seattle Community Network infrastructure on which these experiments were run, and editing and giving feedback on this article.

Thanks to doof.net for providing hosting and support to Seattle Community Network.

Also thanks to the numerous other volunteers and donors to Seattle Community Network whose contributions of money, rackspace, bandwidth, physical space, etc. make the project possible.

Footnotes