Using Erlang Records in Elixir

Introduction

This is a post about effectively using Erlang records in Elixir code.

Lately I've been working on an Elixir project that uses a few Erlang libraries that represent data using the record data type.

Here's how Erlang's documentation describes records:

A record is a data structure for storing a fixed number of elements. It has named fields and is similar to a struct in C. [...] record expressions are translated to tuple expressions during compilation

Records exist in Elixir too but are mostly there for compatibility with Erlang (for a variety of reasons you're better off using a struct or a map than a record in pure Elixir code). The official Elixir docs about records are brief and left me unsure about how to actually use them, so after some experimenting I've made this brief guide for reference. I hope you find it useful!

Prerequisites

This guide assumes basic familiarity with Elixir's syntax, pattern matching, and tooling. While no prior Erlang experience is required, you should be comfortable working with Elixir's basic data types like tuples and maps.

Overview of Records in Elixir

Broadly, Elixir's Record module lets you do two things:

1. Extract Definitions

Extract definitions of records from existing Erlang header files into an Elixir module as tuples. The functions extract/2 and extract_all/1 are used for this.

2. Generate Helpers

Generate helpers for creating, accessing, and pattern matching record data types from tuples describing records. The macros defrecord/3 and defrecordp/3 are used for this.

It's crucial to note that you need to do both steps (extract and define) to actually use the records defined in Erlang files inside Elixir code. Extracting record definitions from Erlang headers results in tuples describing the records, but does not generate the helpers for creating, accessing, and pattern matching on records.

Simply put, you need to plumb the output of extract/2 or extract_all/1 into defrecord/2 (or defrecordp/3) to use Erlang records in Elixir code.

Examples of Extracting and Defining Records

As an example, the code below extracts the :gtp record type from the library gtp_packet.hrl and puts a tuple defining the record into gtp_def. Then, :gtp and gtp_def are passed to Record.defrecord/3 to generate code for using the gtp record type:

  gtp_def = Record.extract(:gtp, from_lib: "gtplib/include/gtp_packet.hrl")
  Record.defrecord(:gtp, gtp_def) # generates the macros gtp/0, gtp/1, gtp/2

Extracting and defining records can get really tedious if you find yourself working with many different record types. Thankfully, Elixir also provides a way to extract records in bulk from Erlang headers. Unfortunately, there's no way to define record helpers in bulk. However, the Enum module and a simple lambda function comes to the rescue!

  # Define helpers dynamically based on the extracted records
  Enum.each(Record.extract_all(from_lib: "gtplib/include/gtp_packet.hrl"), fn {name, fields} ->
    Record.defrecord(name, fields)
  end)

Each tuple from extract_all/2 of {name, fields}, where name is simply the name of the record and fields is a tuple describing the record's fields is passed into defrecord/3 just as in the manually-done first example we saw earlier. I think it's kind of strange there's no extract_and_define function in Elixir that does this all for you, but whatever, this way gives you a little more control over things (for example, you could modify the names, such as putting a prefix on them, to avoid a collision if you're importing records from several different Erlang libraries into the same Elixir module).

Using Records in Elixir

After you've managed to generate code for working with records using defrecord/3 you can finally use them! If you called defrecord(:foo, foo_defs) (where foo_defs was returned from extract/2) you'd end up with three new macros all named foo. Let's imagine the record type foo contains the fields bar and baz. In fact, you can do all this in iex as long as you wrap in inside a module definition:

iex> defmodule Foo do
...> Record.defrecord(:foo, [bar: :undefined, baz: :undefined])
...> end

Creating Records

Now let's use foo/0 to create a foo record with default values (spoiler: it's boring, but it's a start):

iex> Foo.foo()
{:foo, :undefined, :undefined}

Notice the result is a tuple starting with :foo.

To make things a little more interesting we can set values in the foo we're creating:

iex> Foo.foo(bar: :asdf, baz: 1234)
{:foo, :asdf, 1234}

Inspecting Records

Let's say we have a record like {:foo, :asdf, 1234} and that we're forgetful and can't remember the order of bar and baz inside the record. We can use the foo/1 macro generated by defrecord/2 earlier to see the keys and the values laid out clearly:

iex> Foo.foo({:foo, :asdf, 1234})
[bar: :asdf, baz: 1234]

Accessing a Field within Records

You can access individual fields within a record using square brackets, similarly to a Map. To do this, you pass the tuple representation of a record into the generated function with 1-arity, here foo/1.

iex> my_foo = Foo.foo(bar: :asdf, baz: 1234)
{:foo, :asdf, 1234}
iex> Foo.foo(my_foo)[:bar]
:asdf
iex> Foo.foo(my_foo)[:baz]
1234

The generated 2-arity function foo/2 allows for the same with slightly different syntax:

iex> Foo.foo(my_foo, :bar)
:asdf
iex> Foo.foo(my_foo, :baz)
1234

Now we can clearly see that bar is set to asdf and baz is set to 1234. This technique can come in really handy for working with records in the repl if you want to inspect them.

Pattern Matching on Records

However, we can go further and even pattern match on fields within records with similar syntax to pattern matching on Maps. Taking the same foo example record as before, bar and baz can be pattern matched into their own variables:

iex> my_foo = Foo.foo(bar: :asdf, baz: 1234)
{:foo, :asdf, 1234}
iex> Foo.foo(bar: bar, baz: baz) = my_foo
{:foo, :asdf, 1234}
iex> bar
:asdf
iex> baz
1234

Pattern matching is very useful if an Erlang library hands you a record from which you want to concisely extract multiple values. Pattern matching can also be done in function heads to select a specific implementation of a function based on the values of its inputs. For more details on pattern matching see the Elixir docs.

Updating Records

Finally, you can update fields in an existing record (leaving other fields untouched) using the 2-arity version of the generated macro, again foo/2 here:

iex> my_foo = Foo.foo(bar: :asdf, baz: 1234)
{:foo, :asdf, 1234}
iex> Foo.foo(my_foo, bar: :qwerty)
{:foo, :qwerty, 1234}
iex> Foo.foo(my_foo, bar: :qwerty, baz: 5678)
{:foo, :qwerty, 5678}

Summary

I've tried to show in detail with plenty of examples how to use the functionality of Elixir's Record module. This includes extracting record definitions from Erlang header files using extract/2 and extract_all/2, generating macros for working with records via defrecord/3 and defrecordp/3, and examples of how to read, write, and update record data types in the previous few subsections.

Hopefully this guide fills in the gaps in the official documentation and makes at least one other person's life a little easier.

Tunnels Within Tunnels: One Weird Trick for Free IP Transit

2023-11-16 update: John Elliott published a post with router configuration details

Imagine you took your homelab hobby to the next level by acquiring an ASN. You don't have much money, but you've still managed to connect to an IXP and peer with other ASs over the exchange fabric. You'd like to connect your network back to sites scattered across a metro region, often with only consumer internet connections because other types of service are too expensive.

You may want to network the sites together and may also want the sites' internet-bound traffic to exit through your own AS (there are a number of reasons you might want to do this --see the last section). Hence, you decide to set up VPN tunnels between your sites and your network equipment at the IXP.

Unfortunately, the large regional ISPs servicing your sites will not peer with you because you are small fry, so the tunneled traffic doesn't reach your router at the IXP via settlement-free peering relationships, instead it goes through a paid upstream transit provider. You don't want to increase the traffic through your transit provider more than you have to because it's expensive and limited. Is there a solution?

Recently, while tinkering with VPN tunnels with other volunteers from the Seattle Community Network (eg. John Elliott), I had a sudden flash of inspiration. An absurd idea emerges...

Enter the Hurricane Electric Free IPv6 Tunnel Broker

The Hurricane Electric Free IPv6 Tunnel Broker is a free service that HE provides out of the goodness of their hearts. If your ISP's IPv6 service is lacking or nonexistent, no problem! HE will generously lend you a /64 IPv6 address block and provide you with a tunnel endpoint address where you can send IPv6 using RFC-compliant 6to4 technology!

For costing nothing, HE's IPv6 service is amazing1. Here in Seattle on a Centurylink connection HE's tunnel broker generally outperforms Centurylink's own 6rd service. And unlike 6rd, the HE IPv6 block is static, making it far more useful for running servers at home.

Besides providing free IPv6 forwarding, Hurricane Electric also freely peers with other networks at exchange points, no matter the size.

Hmm... Could we use HE's network to bypass an upstream transit provider?

It turns out yes, and the performance is much better than any of us expected.

A disclaimer: Don't abuse this. There are also some details about why this is kind of technically icky2. But we had to see whether it worked (and how badly).

Read on...

The Initial Configuration (via Paid Upstream Transit Provider)

The inital tunneling config is pretty straightforward. We're running a VPN tunnel endpoint on one of our servers at the IXP. Internet-bound traffic is NATted with an address from our public IPv4 block.

Routers at the remote sites connect to the VPN endpoint at the IXP. Internet-bound traffic is routed to the IXP via the tunnel, unless the tunnel is down because we messed something up, in which case traffic bypasses the tunnel and directly egresses via the consumer ISP connection as a fail-over.

We're using WireGuard for these tunnels. So far WireGuard has been simple to configure (though there are a few quirks), fast, and ...fast. MikroTik's RouterOS 7, released in the last year, supports WireGuard, allowing us to finally use it on our MikroTik hardware. Other volunteers, John Elliott and Esther Jang, did the bulk of the work figuring out the WireGuard and firewall configurations.

John used iperf to achieve a respectable 860 Mbps over the tunnel between his home 940 Mbps GPON connection to the VPN server.

These are two traceroutes from different consumer ISPs showing that the path taken to our test server includes a hop through an upstream transit provider (AS1299):

$ mtr -w -r -z -c 5 23.146.168.1
HOST: foo                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
1. AS???    rb5000.home.arpa                      0.0%     5    2.1   1.0   0.7   2.1   0.6
2. AS209    tukw-dsl-gw69.tukw.qwest.net          0.0%     5    3.2   3.0   2.9   3.2   0.2
3. AS209    63-226-198-33.tukw.qwest.net          0.0%     5    3.0   3.3   2.1   5.1   1.1
4. AS???    ???                                  100.0     5    0.0   0.0   0.0   0.0   0.0
5. AS3356   4.69.219.210                          0.0%     5    3.5   3.6   3.3   3.9   0.3
6. AS1299   sea-b1-link.ip.twelve99.net           0.0%     5    3.5   3.8   3.5   4.3   0.3
7. AS1299   doof-ic-375811.ip.twelve99-cust.net   0.0%     5    3.5   3.4   3.1   3.6   0.2
8. AS54429  seattlecommunitynetwork.org           0.0%     5    3.8   3.6   3.5   3.8   0.1

A traceroute from a Centurylink residential connection to our router at the SIX transits 1299.

$ mtr -w -r -z -c 5 23.146.168.1
HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    router.asus.com                                    0.0%     5    1.2   3.2   1.2   9.2   3.4
  2. AS???    100.92.123.67                                      0.0%     5    8.9  11.6   8.9  16.6   3.0
  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   13.1  11.8   9.6  14.1   2.0
  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5    9.9  11.4   8.6  15.4   2.6
  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5    9.7  17.4   9.7  29.9   7.8
  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net         0.0%     5   10.2  15.0  10.2  31.1   9.0
  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   12.5  10.9  10.1  12.5   1.0
  8. AS7922   50.208.233.194                                    40.0%     5   32.0  25.1  20.8  32.0   6.0
  9. AS???    ???                                               100.0     5    0.0   0.0   0.0   0.0   0.0
 10. AS1299   doof-ic-375811.ip.twelve99-cust.net                0.0%     5   15.1  14.7  10.4  25.1   6.1
 11. AS54429  seattlecommunitynetwork.org                        0.0%     5    9.8  15.2   9.8  35.5  11.3

A traceroute from a Comcast residential connection3 to our router at the SIX hops through 1299 too.

This cutesy diagram shows roughly the same information as the traceroutes but with a larger font and some arrows.

usersitecenturylinkorsomethingas1299(arelion)as395823(doof)as54429(us)

The Path to the Hurricane Electric IPv6 Tunnel Broker

Let's do a few more traceroutes to see the path the traffic takes. We'll start with an IPv6 traceroute over the HE IPv6 tunnel broker to Seattle Community Network's router (2602:fd94::1) at the Seattle Internet Exchange:

[asdf@rb5] > /tool/traceroute 2602:fd94::1
Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
#  ADDRESS            LOSS  SENT  LAST     AVG  BEST  WORST  STD-DEV
1  2001:470:a:f00::1  0%     101  4ms      4.4  2.8   31.9   2.9
2                     100%   101  timeout
3  2602:fd94::1       0%     100  3.2ms    4    2.1   67.7   6.5

(First hop IP address slightly redacted)

This traceroute doesn't show hostnames or AS numbers, but that's ok. What's notable is that this traffic seems to hop right into our network, after a mystery hop at hop #2 that I suspect is some HE-internal stuff we can't see. It appears traffic enters HE's network via their IPv6 tunnel broker then passes through the peering connection between HE and our router at the IXP, rather than going through a transit provider. This makes sense because the peering link should have fewer hops, seeing as the traffic is already inside HE's network.

That last traceroute was run inside the tunnel so it doesn't show any of the hops that happen before the packets reach HE. To see the path taken by the tunnel itself, in other words the outside view, we can run traceroute to the IPv4 address of our HE tunnel broker endpoint 216.218.226.238.

Here's an example of an outside-the-tunnel traceroute to the HE tunnel broker endpoint taken from a Comcast residential connection. Pay attention to the AS numbers, you can see that Comcast (AS7922) connects to Hurricane Electric (AS6939) without an intermediate hop. Looks like they peer with each other.

$ mtr -w -r -z -c 5 216.218.226.238
HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    router.asus.com                                    0.0%     5    2.3   3.0   1.2   7.5   2.6
  2. AS???    100.92.123.67                                      0.0%     5   10.2  11.1   8.7  14.4   2.1
  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   16.0  11.5   9.8  16.0   2.6
  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5   12.1  10.9   9.0  12.2   1.4
  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5   11.9  13.7   9.8  20.1   3.9
  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net        60.0%     5   13.6  12.1  10.5  13.6   2.1
  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   21.9  14.0  10.2  21.9   5.0
  8. AS7922   50.208.233.194                                    20.0%     5   28.1  20.7  10.5  31.1  10.4
  9. AS6939   tserv1.sea1.he.net                                 0.0%     5   11.7  10.4   9.8  11.7   0.8

An mtr traceroute from a Comcast residential connection to the Hurricane Electric Free IPv6 Tunnel Broker

Cramming Tunnels Inside Tunnels

The last piece is getting the user traffic to take the path through HE shown above. Briefly, we

  • Set up HE free IPv6 tunnels at user sites.
  • Send site traffic over an IPv6-addressed VPN to our network.

In other words, we put a VPN tunnel inside HE's 6to4 tunnel. Our inner VPN tunnel must be destined to an IPv6 address so that it goes inside the HE 6to4 tunnel. However, the traffic inside our innermost tunnel can still use IPv4.

Traffic takes a path that looks something like this:

usersitecenturylinkorsomethingas1299(arelion)as395823(doof)as54429(us)as6939(hurricaneelectric)???

The "???" network represents hops between a residential ISP's network and HE. In our case on Centurylink and Comcast there's a direct connection4 to HE (AS6939), but other ISPs might go through a transit provider. Also, AS1299 and AS395823 are no longer part of the traffic's path so they're grayed out.

At this point, we've discovered a path from other networks back to our network at the SIX that doesn't involve a transit provider. This traffic doesn't cost us anything (because we peer with HE) and it might not cost HE anything either (because they're probably peering freely with big ISPs).

Performance

With WireGuard as the innermost tunnel, the packets look something like this mess:

a diagram of user packets tunnelled via Wireguard via Hurricane Electric IPv6 Tunnel Broker via Centurylink PPPoEeyJ2ZXJzaW9uIjoiMSIsImVuY29kaW5nIjoiYnN0cmluZyIsImNvbXByZXNzZWQiOnRydWUsImVuY29kZWQiOiJ4nO1dW1NcdTAwMWK7sn7Pr6DYXHUwMDBm55yqPbN1v6w3ICFcdOGWcE1O7aKMPYDvxjZcdTAwMThYlf++NVx1MDAwNjwjzdhSXHUwMDAzJiTbWatSwe5m1FJ3f909Lenvd0tLy8PbXrL819JyclOttOq1fmW0/M/08+ukP6h3O+YrMv550L3qV8eUXHUwMDE3w2Fv8Ne//pVxxNVu+54raSXtpDNcdTAwMWNcdTAwMTi6/zc/Ly39Pf4795x+Ulx1MDAxZFY6561kzDD+KntcdTAwMTRl2P10u9tcdTAwMTk/VjElXHUwMDExxVpNXGLqg/fmccOkZr49q7RcdTAwMDZJ9k360fJqv7JWu904PTq7XFxZu/x6c9ZtVqrZU8/qrdbe8LY1XHUwMDFl1aBrJMm+XHUwMDFiXGb73WZyVK9cci9cdTAwMWUnIPf5NK5+9+r8opNcZlx1MDAwNlx1MDAxNk+3V6nWh7fmM4zQ5NP7KfhrKfvkJqXAgk0+SFlcdTAwMTiVMXdcdTAwMWW/1m11++nj/4GT9L9sXHUwMDAwp5Vq89yMolOb0Fxm+5XOoFfpm0XJ6EaPguHs8Vx1MDAxN0n9/GKYTrSItfUne3wynmssXHUwMDA0XHUwMDEzVGmZXHI1fWjvc2287P/OZrhfaSefU5bOVauVn6ZO7WGaXHUwMDFl1SNTXHUwMDEw+vDJz0yqlP6Dq1h55bJcdTAwMTRsmNxkoubU4Xg9+XLcxtUjcnlyd7GxwvYjsrs8ofv58K9s+Fe9WmX4ILFWWFGdaiCdfN+qd5qubK1utZkp5LucIFx1MDAwNUOwxpm3XHUwMDAxOtVcdTAwMDZcYlx1MDAxMlx1MDAxY0spSLBccpRcdTAwMGL9tm2AIFx1MDAxMkujZEJiyYmmtkFwNUeDwJTE3DxZcME0w1xmiaJ9MF4wXGIsXGLlhDPxfIOwviho/ksqZzaqbme4V79cdTAwMWJcdTAwMWKfsD5dr7TrrfGqTT5cdTAwMWVrrZnGXuW21a3kXHUwMDE2Pv1ipVU/T1V4uWpcdTAwMDac9C3tXHUwMDFl1lxyWkxcYtr1Wi1cdTAwMGZcdTAwMDFV87RKvZP0P4e47m6/fl7vVFr701x1MDAwNmOkTz49Llx1MDAxOI5cdM9pxCBJv02FJTPtcyZQccrdT1x1MDAxZo1cdTAwMTRrLTVnKptKn5Fer3VuOz++XHUwMDFmjLa3vzTY6NPlzYhtvnEjRTKT71x1MDAxZajmaZcqQ/1cZqeYXHUwMDBmpyjHWlx1MDAxMkyzof9cdTAwMGU4dUjPb1x1MDAxMTtcdTAwMWOtSPLlZO2y3vh+drD3XHUwMDA2cUqQqSZAMFx1MDAxMZRwXGawgXKp37hccmhcdTAwMWNrwozqXHUwMDExKpFA3LaIuVwiXHUwMDE1XHUwMDE3MeZcYiFscEdJpcOQylx1MDAxMGPNXHUwMDA1Vs+3iN9cdTAwMDWohtXenEDK47ZdkLJcdTAwMDYyd4BcIlx1MDAwNLmfTjIpQZDxiyzcOPeS7e1cdTAwMDHrRntXPzbJsDnCx3XceNvGqbWbSGnn2S+ITjhG+T+ZcufAqphFSVwipaTyXHUwMDA1bPFcdTAwMTXR6dtp9Fx1MDAwNV/yg0uJefLhU5vii87qXHUwMDFiRCeSK1x1MDAxNDj6T7lCXHUwMDE0SVx1MDAxMp5FlVx1MDAwYv2m9d9QoFi+JlwiiXylYla2RFx1MDAxMNZYIPFflC3Ve9dsTijk8c0uXG7ZI5k7XGbR3CS7eZI0KipcZlx1MDAxNlx1MDAwMYLE96e7vd56o9Ho7+1KdrKL3rdv3rZcdTAwMWRKlf38kCfNzVx1MDAwMk2qY6dEXHUwMDE56OSqe649Miy51i9cdTAwMTJcdTAwMTO+XCJcdTAwMGXxjY/y4OZsZf94XVxmLnon36W8XHUwMDFlvEFcdTAwMWOiuVx0L2RJiGKlXHUwMDE0XG5cdTAwMDeicqnftFx1MDAwMSjEY1wieZapvFwiJGFBY5bmZ0IzRVhcdTAwMGXxZ+FcdTAwMTMzK8JcdTAwMTl+garB71x1MDAwMk+jej85v6r051XO8/htXHUwMDE3o0qG81x1MDAxY6C6N51m9fv3m/1ke9Rpk8/HvPtcdTAwMDNdNq9z8k5Hs7FcdTAwMWJHytZbKnJ6+6hvskTBclxuXzSQl1L4X2v4c8WLXHUwMDA3qyQmWOAkfStQ4mQx0e6nkyhDXHUwMDE4XHUwMDA3xDRcdTAwMGWK9l9cYpt0beN9/yQ6Pz5cdTAwMWZcXNCjjVx1MDAwYvxtu3v0XHUwMDBisGnWaFxuim9cdTAwMDPYWOcxi1x1MDAwNVeUXHUwMDEy4zslkjNcdTAwMWP3pCqlYmFcdTAwMTgoo1x1MDAxY3HFy1x1MDAwMpCcw13Ywyy8mLxcXMJMXG6NXHUwMDE1KVP83Ko4is9ccotgKuhdoav31jheXHUwMDEwnFx1MDAxZfHmqpYrhYFcdTAwMTHr6bBkYcU0sPJcdTAwMDCFXHUwMDBiVpYsXHUwMDA1mJphlzt9XTmILn/ojcuv31x1MDAwZprRwfutL8NgQOLMscdSQDJptp1cdTAwMTFkTDmAmla9Wtjn0/FKKSm1wZ1CXHUwMDE1Nlx1MDAxZN+sl4eUSsVMpPx6cHV9e9Xo7P6obKmNr8OGWF2tXHUwMDFjjNZ/XHUwMDE5XFyVjyZcdTAwMDCuuKazXHUwMDBiXsGFqoXeh+CSxFgjJmWphuOp1VeMiDKMNCjrfW1gqveuxdtFJlx1MDAwZmKUlPrE06Dp/da342/75PDL4O68ff3pR/d6pdlcYoYmxjNcdKZDky6xwVx1MDAwNVx1MDAxND2M6UWhiCNjqpSWQ1x1MDAxMZueOVx1MDAxMVx1MDAwM2BSaZxcdTAwMDXxc4einY1et9c8QtVop3FHt5un7Vx1MDAwZjeNX1x1MDAwNkXlo1x0gFwiJvhcdTAwMDKKXHUwMDFlPn1ccihSTGohMSqrXHJcdTAwMTDKpml4Wlx1MDAxNsTkSfr9XG5AxN4uXHUwMDEwefBcdTAwMDH4zmmGXHUwMDA1XjW/ne91WtvN5lx03zjb/2Rysv5tMFx1MDAxMFFcdTAwMTZStMOo7DXJomr3XCLQw1x1MDAxNDFcdFxyL0SDKZMsVDQmXHUwMDFkKoRJgp5cdTAwMTQhPlx1MDAxMXjW23hrd/Sh0cZtPNrD1ZWj/ZMvv1xmeMpHXHUwMDEzXHUwMDAwPJTTmKcvPFx1MDAxZTvSQkBIk1gjVPKCZlx1MDAwMUhPXHUwMDA0JGpCLqowLiBP+l2xkJe9XHUwMDExx0ohJYP2uLw2XCL1er1u8nYhyYNcdTAwMTSFjnFbXHUwMDFhXGImra6J3u5av9Zu3NDT/u6qqu/fdorGedY3uvC4m2r8zX1kmM3K7dhcdGZrPXmNz0p6OXnG92iCNMZcZnNNXHUwMDA0p1xcaZnbKLEwyYJJXHUwMDEyXHUwMDEz6CnFRbEtLP1STC1XSKY0l7nc6ddbZK9bd7Eu+9dS5pnHP0z+/e9/llJHOE+OYeRcZkYuYOTSR25cdTAwMGad+MiJRe5cdTAwMWS7Te5cdTAwMWQ7tciVj5zlyYl3mbhF7lx1MDAxNVVY5NRHLmHkylwi5z5ybZF7J1x1MDAxMiMgvaVcdTAwMDbErzXWwlx1MDAxMu9KYXuptJeeXHUwMDAz6a3Zp15VIFxiSE+A9Fx1MDAxY0ZPMZBcdTAwMWU4XHUwMDFlyoD0lnpSr0ujXHUwMDFh9vtcdTAwMTlw/IxcdTAwMDLpgfPPgPrDXHUwMDE0jJ5cdTAwMDP1jVx1MDAwM+XlQHmF7Vx1MDAxZrz2JYD2K1x1MDAwNIxeXHUwMDAyxyNcdJBcdTAwMWU6XHUwMDFlXHSk1zB6XHUwMDA1lFdcdTAwMDHnX1x1MDAwMeXVwPFoXGakXHUwMDA3rpdcdTAwMDb6f1xylVx1MDAxN2jv2vZvXn+YXHUwMDBiwUNcdTAwMTlsj+iNrDCiUFx1MDAwNlx1MDAxYlx1MDAwM7whXHUwMDA3RrZTXHRcYjqsVaD+KFx1MDAwMtnL4Fx1MDAwZiOQtVx1MDAwZcy7bthcdTAwMGWEmDfow3YkxPyzZIdCzC+Dw+CPbWxcdTAwMDbu1yWHIWCWnjckuNB+1XBcdTAwMThcdTAwMDJcIljyvIXzXHUwMDFikMPgXHLxXYZcdTAwMDDls2yaXHUwMDA1rDSD2oNt01x1MDAwMVx1MDAwYiegXGa2TVx1MDAwN6y0elx1MDAxZUOAamggg5NcdTAwMWVcdTAwMDQw2PG7X/lcYoUyQL23k4JcdTAwMDQw2Cvtt1x1MDAwN6KgXGb2OvhccohcIihcdTAwMDOGMlx1MDAxMChcdTAwMDOFMjAoXHUwMDAzhzJcYiiDhDJAXHUwMDE3jkFcdTAwMTeOQVx1MDAxN45BXHUwMDE3jkGnlUFnyUlcbv3u3snyXHUwMDAyXHUwMDE4XHUwMDE4lEFcdTAwMDBcdTAwMTlcdTAwMDQ0blx1MDAxNVx1MDAxOMrAoVxmXHUwMDEyylx1MDAwMCxeYDubXGZhYFBcdTAwMDao0Fx1MDAxMppwKOjCKahcZlxuKoOCroOGyqChXHUwMDBiZ+eJIVxmUOXTQKFcdFx1MDAwMspAnMQvgFx1MDAwMSg0QUDlI1x1MDAxOLhwXHUwMDA0Q4XGQOUjXHUwMDE4KjRcdTAwMDauNCFQoVx00FVcdTAwMTJcdTAwMDI0UUKgQlx1MDAxM6i2XHUwMDEyqGpQ6CxR6CxRqD1Ai+HEXHSwXHUwMDAyXHUwMDE4oK8j7Hq7v35FKLDg6DJ48zhih3D+V0LEXHUwMDBl4fyvtIhcdTAwMWTC+V/IXHUwMDEx+y1cdTAwMDDxrzSzVlx1MDAxYftnyVx1MDAwZVx1MDAxMnGA0DaDN1N0XHUwMDE5XHUwMDAyZslm8L9ydVx1MDAxOLxxq8tcdTAwMTCwXHUwMDBlNkPAOthcZn57mM7wzmFcXO71k8HgyvxltzhcZurtq1ZlmOw+fG2+XHUwMDFk9q+yXHUwMDA2gFZlMFxc67bb9eEwqe2m3Vx1MDAwMFYzgDWCyfPvn/7z/pu/XHUwMDFmf9V9S8v307Nvoz1+ez6oyUNGVoQ+2rxcbm1poVLHWpqMWlx1MDAxM6mMrdpcci5cZvNYUYqQlFQgnu0/m2xcdTAwMTkt2TMtit2XPNaMXGIuXHUwMDA1libW4lmp81x1MDAwZmx2mSze8vnR7kGzeVT7tvN5rX483L+7XHUwMDFi7pPlglx1MDAxZT29b5pwTmhub3Rua2lxv8CkT41cdG7+z+/F/sO6Yiw4iPypp+Wro1x1MDAwMFfqPMDrSu3CSuSvx9qYXHUwMDE5+cvcttuKuFx1MDAxNz/sXHUwMDFhQySAL54j4fW99pvnSHhnidvTKrzrwInN4J0ll8FcdTAwMGLLM1x1MDAxOF5cdFx1MDAwZfIjyJ4/XHUwMDBiXHUwMDBlblx1MDAwNoefrr+K6uijXHUwMDFhttrseJ81P35cboZcdTAwMDPFY8k4Nb6ecUWzKs5cdTAwMDNcdTAwMWOI2ES9XHUwMDA0U04xY7jY/Vh2JHmWTCzwYI54QIVMXHUwMDBmsS6eWDRcdTAwMWXb9I1iVGGpXHUwMDE1/XPhwPZDfjiwyP3FV9tTe7HDdiher2uR+/2VjWReXHUwMDFmLWx6WGdi5M8hbJDxI6tccnv+tMmm9+dl2NFcdTAwMDW/MjhcZlx1MDAwMc1cdTAwMGVTXHUwMDE5Xlx0MfIjyJ4/XHUwMDBiMZDe2ddcdNrcXFxlqL623/jRuGiQYMTQJJbKYEG6oVxcSY2dky0wjSVhXG5rgrSWNHvRPMFcZlqSQ1x1MDAxNPdcdTAwMGUvIGNcdTAwMGUpRFxu41SS0q0uuZ6A4k1cdTAwMTaaUYQ1f0ubL+eXQsCo/UVcdFx1MDAxMDVcdTAwMDVRXHUwMDBiXHUwMDEwtVx1MDAwNFFj4Fx1MDAxNDpplZeew+jtKnhcdTAwMDA9XHUwMDA20lMgvVx1MDAwNNIrIL1cdTAwMDN2XsVcdTAwMDGmwXZDwnPoX1x06PJcdTAwMDOYPH5cdTAwMTbONe92Vf/bOVx1MDAxNcd7ndWdjc5pb3u7WsS5ko2ZxM6DiC7uw8RaxjQ9019gRDnmolx1MDAwNNVcdTAwMTY7MWH4pJTkjDBdXHUwMDA2T9lL4eJcdTAwMTk1aVwiRFju5pBfj06TXHUwMDBiXHKSVrW7VO1fneWX7MnbMVvJmX1wrb1cdTAwMTlz2O3lXHUwMDE1yL9cdTAwMTPTXHUwMDEypXBcdTAwMDdC2dghmy/F/mirXHUwMDE14Vx1MDAxMX1P2/uDzUq/0ezjoNOhsoaV+42XuWMhJuf4i5imm6dcdTAwMTWT6blcdTAwMTDZhC3M78mHRHFigkNeemqnLnyYnYImkED6SYegzdv8xLD7XCInc7yq3dmDXHUwMDA2XHUwMDE53IEkV5u7/fftteujj/v1o91delx1MDAxZbzbWdmpXHUwMDFjz41+gnolVzvlpnJhZiFmxrikgklaes7G1PNvTMbN0oNF6Vuys5fd3Wy/pveSW1x0h3+7skXufXdcdTAwMTBZ4bq/lFx1MDAxNWFgk0HkbFX1j1xiXHUwMDAzeywjaP+6+4TnMLxzXHUwMDE455RcdTAwMTRYI5g8f1ZWsKOPbkSiRvxcdTAwMDM+k5XNndN1cVx1MDAxZIX7SPtEfJ5zflx1MDAxM1x1MDAxZlmSXHSQhY9cdTAwMDRWqjDDXHUwMDFhXHUwMDEzUfaye/qVq1x1MDAxOGmkXHUwMDE5ybXN/mk+kj2dXHUwMDFhVu332r5F7XVetjdcdTAwMDWW1f1cdTAwMDc0OMdcdTAwMTD46/y2s1x1MDAwZdhRYtFcdTAwMDfs05lG/0p+MT9cdTAwMDBcdTAwMTLiXHUwMDE2+99cdTAwMGV3259vXHUwMDBlaqfH31x1MDAwZmrnq1xun1x1MDAwZlx1MDAwZoPdonTdYsmr4lx1MDAxMreI6dQ7cFx1MDAxNm4yxE1qprjCqLQnaPoxOUSaXHUwMDFjz0DZb1x1MDAxNEk+w+9cdTAwMTlcdTAwMTVjXHUwMDE4S1x1MDAxM3EjrsYn3MGcXHUwMDE1jlV6YfTkl3grt1x1MDAxNjujcZ45oFMnzy1ZrPJP9zoqy1x1MDAwZmpcdTAwMTXnXHUwMDFmrfwnXHUwMDAw2W2sWMWW5MCQvGjdL8f/Sm40P6BJv9EsN3p+OVx1MDAxYcottj48wVfdreZqi3/4kISUvFx1MDAxODGLpTjFgppcdTAwMTSPSLtcdTAwMDBmzDVcdTAwMTYmsNFUSYpcdTAwMTgv1qNcdTAwMTlcdTAwMTGxMT06vjOW5jZlLuphT/WuXHUwMDAya00pKz0zfUY5zGTpnEtO51x1MDAxMIOmf7JFfEI9rJZcZoZ/LX368D+DpeFVp5O0lk7Tqe//djUyvyCQulmlvy5cIr2xc7GSrN983Ti62Dm7qoVcdTAwMDY/3GmiLqublZTN8Fwi1oGdXHSolaZElp9cYjjNXHUwMDE4XHUwMDA1Slx1MDAwZqelQVebv5FI51x1MDAxOc1ufny2z4mAwbG/QGV3I/hHXHUwMDAzTFxup9O/VjSQXHUwMDFiXHUwMDAwXHUwMDBl6bS6a41GlFWvuse1k0azpjc+XHUwMDFjXHUwMDFml9xcdTAwMWE0za/YMVx1MDAwMM+94MyuXGYqOpZFrVx0mkShNFx1MDAxYlwipUlcdTAwMTSZmkRJgrSiVL+lt84v6lpcYojaPlx1MDAxZsZHbVuyv7fILlx1MDAwN/nLNcBcdTAwMDM7XHUwMDFkev/vt09cdTAwMDHzl4Om0r+S58pcdTAwMGYgqFx1MDAxY3SxS9dOXHUwMDA3jY6+OdtcdTAwMWag7ubaXHUwMDE2v1x1MDAwYj43ueC5WNFzlTiubJvmwnEthUREXGJjXCKJ1mVcdMp0x8WEJml77lu60ulcdTAwMTfGRFx1MDAxNrXXkm1HXHUwMDA0fKXpr2NH0HeC0Fx1MDAxN3ZcdTAwMTa9f2PadPpXclxc+Vx1MDAwMbCQpr/19zcnm1x1MDAxZipcdTAwMWZ3hp/1+t7Op/NcdTAwMWFaXHUwMDBmu5FcdTAwMGXz2Fx1MDAwMDoz6ZxQlDO7XHUwMDA1kKviiz6OcKzyLYAlLm1Rclx1MDAwMb73k5jy9Fx1MDAxZF6JSyv2XHUwMDA1ZjWXdCeUVPN476dlvjP+yTWX7lV/aXJV9lLSqY1d3m9ad/FcdFx1MDAwM6m9fLxcdTAwMWX1XHUwMDBmfvCv0ernr3drtZXL5spNLzTSwIjjmDNjtoRcdTAwMGLOc9WB+7ijWImR8fiCZCw4JlRl5zbkXG4zSMVKYqNRgkmmhCxe5rCw4VlcdTAwMWJcdTAwMTNcdTAwMTU2XHUwMDEzy8s3mYhCrDIp1UgtmUbzKNWkXHUwMDE3WKq8cr5QWIJjwYxcdTAwMTZxZZTPRFSwhIlPUcT0XHUwMDBmi1x1MDAwNULE4IpcdTAwMTZUXHUwMDEyIf07nWf8MkxjlWKU4kpjrJD//MRZv43oXHUwMDE4UyaoJoxcbk1FwFx1MDAxMf4zZomJXHUwMDE4JqhcdTAwMWRQSVx1MDAxNVx1MDAxM5TeXk6pScU5XHUwMDA3XHUwMDE2wDSPkUSaKMRcdTAwMTVcdTAwMTXKv6HcXHTnXHUwMDEwi4nBK4aMtmPMkHfzqMvvOpqX43+t8Cw3IFx1MDAxY1x1MDAxNJ9VOndr+nR/q3PGL1f2No/WjmWXh7t75jRcdTAwMWGwYmN4WU0st0994cNDUkujzVx1MDAxMitSflxy6fStXHUwMDE4jFx1MDAxOJcr2Ju6x/5le1RhLVV2yTrgXHUwMDA2XHUwMDFly8ChLaFcdTAwMDH56HSG1/JcdTAwMTj5XHUwMDExXHUwMDA0nXe0tb23drK7sq1cdTAwMWF9+mVcdTAwMTOvXa7I05JtXFxTXHUwMDAzRPtWvbJaVGnH5sJjQFuRsFScXHUwMDE0L8ybWYxKrzFkWopFNeqBIE9ccjsuwustgK/n5t2yabdI+mv0U+lfyXNZLZUhjuukebTzsTnEODnm9WF0e/Dj6HPJfbjFUpSWPMbC5EpcdTAwMDQxpPNZ7LhcdTAwMTeIo9jALKVCUqZJMVx1MDAwMKLahNwmXHUwMDFlxlx1MDAxYWkstVjcXHUwMDEx+lx1MDAwMlvjKCNKsmKnwTjqke6nk7txtclcZqSax0tCo1x1MDAwMOhcdTAwMDV6gXpJf+lqYP4yS3h2Vq/+plx1MDAxNanpYkBqUT3Vfz9I1N7t6fVcdTAwMTFqN3DvsD8qucq3PNSQxH7rxUhJpFFWJOaLUFx1MDAwM1x1MDAxOGpgk3VcdTAwMTNSaoxq6ikm6eXxVMu5VJjeRqhcdTAwMDHdQFx1MDAwNzxcdTAwMDHCofdnMzY9sLpcdTAwMTJB365cdTAwMDWcZ+UywNqpXHUwMDAz6J3fXHUwMDBmPO/LXHUwMDFmNFx0IL1cdTAwMDLSu8dZXHUwMDAxo7JcdTAwMTBcdTAwMDZcdTAwMDZlXHUwMDEwQFx1MDAwNlx1MDAwMpWBQGUgUFx1MDAxOZyzX/xcZlx1MDAxNCpcdTAwMDOFXHUwMDBliUJVg1x1MDAxMShcdTAwMDNUWVx1MDAxOXRIXHUwMDFjOiRcdTAwMGVcdTAwMWSSQFBcdTAwMDbokFx1MDAwNNhcdTAwMDWAZYBOq4TKIKEySOiQXHUwMDE0dEhcbjpLXHUwMDFhKoNzkWJcYlx1MDAwN4Zqk3MpWlx1MDAxMFx1MDAwN9jlu1x1MDAwN2tcdTAwMDVwgCUn4NklYMndXHUwMDAztvxcdTAwMWNcdTAwMTQsOVx1MDAwNctBofCFKVx1MDAxNCycS7BCjrl0nHlcYodcdTAwMDMwXHUwMDAxPZiO5P4uSVx1MDAwN1x1MDAwMEJu+7Y5/Fx1MDAxN2Bg55jqgMvkXFxcdTAwMGW/XHUwMDFjLodfXHUwMDBlXHUwMDA3yVx1MDAwMm5wdDlcdTAwMDLkcDhcdTAwMDLksNc84PbNXHUwMDE5XHUwMDFj71x1MDAxY855VenyQ8hcdTAwMDYwq05cdTAwMTd9XHUwMDFjXHUwMDFkNsngZLB13qqcND7XXHUwMDBmVuonoVm/yf8mP99n/SWn5qCSNpNF1lx1MDAwZsz6Tb5vXHUwMDEy+Nzdv7msP1x1MDAwYlmKR0JgXCJcdTAwMTFiufdcdTAwMTJ/WNZcdTAwMGY879omXHUwMDA3ZsCw/lh/yu9cdTAwMDAwOONcdTAwMDe+r1xyKVx1MDAxMThcZv4qh4AyOHFKwCk70DjFeY9cdTAwMWPEXHUwMDAxjbdcInCkXHUwMDE5gePGXGJcdTAwMWM3RuC4MVx1MDAwMseNXHUwMDExOG6MKFhycKRcdTAwMTlRsOTgXHUwMDFhQMTAkoPLXHUwMDA2XHUwMDExXHUwMDAzS1x1MDAwZa40ROBSQ8TBkoOrXHUwMDEzXHUwMDExXHUwMDA3j1xuXFyfiMBcdTAwMDWKXGJcXKGIwCWKXGJcXKOIJFhycFkjXHUwMDAy1zVcIlx0llx1MDAxY1xcXG6JXHUwMDE0tDZcdTAwMTgpaIEzUtByYuRcdTAwMTRcXFx1MDAwMs6/U1xuyqFcdTAwMWTJ/VDrcvixVlx1MDAxMzCHM1f+XHUwMDE1tC+HXHLIWiPt5LlcdTAwMDFz5XBcdTAwMDT0niFooltkXHSIXHUwMDFhXHUwMDFjXHUwMDE2XHUwMDE20LOGoNlxkSVEfGh+XHUwMDFj5fK6J7OEzJjNwkNmbDrLO4d1bn1++TFkI5iViCtNmvj04jC6vdzVa1u717x5vVx1MDAxOdIwI7TdXHUwMDE2TEuuqlwiWsVcdTAwMDJcdTAwMGIhTNaG00tFJlx1MDAxNIuumCd3xZjUmlwiXnpg+/SdXHUwMDFlSqebtfCbOn7ssSkm29V0kVRqv+HBONNcdTAwMDVcdTAwMDBcdTAwMWQkvS02uu3r5KC6J1x1MDAwNpsnV5s721x1MDAxYsGXylx1MDAxOX1za2LFTlx1MDAxOFJ6dVxcplx1MDAxMlx1MDAwYlx1MDAwM1xmMMDU9jiXpbslZ1x1MDAxY/+HqZZMMfZf0lxiXHUwMDAzrSpBXHUwMDFiW/yoP+9GXHUwMDE4aFx1MDAxOVxy2jdcdTAwMDMtokFLYlxuSO9cdTAwMTbEwFxmwFx1MDAwM1xcQ1x1MDAxOKC3yrpcdTAwMDU3P1x1MDAwM7RuXGIut4GrbeBiXHUwMDFiuNZcdTAwMDYutYErbeCyXHUwMDE5uFx1MDAwNlx1MDAwNi5ogetZ4OJcdTAwMTS8c1x1MDAwNipcdTAwMDO4llx1MDAwNe+cXHUwMDAxN5FAZ0lBn1x1MDAwMG5cItHgNlx1MDAxNXhHXHUwMDFmuFlcdTAwMDPBm07g7Vx1MDAxZOBRuYbt95euZVx1MDAwN/hwMEy4tlxyfqlcdTAwMWWAXW6RN4BcdTAwMDNcbsDYLY5cdTAwMDZwgEHbbeJcbuBcdTAwMDDPlWuyflx1MDAwZVxy1ivXyv1AjKAlW+JYrb84SpySz1x1MDAxMzj83c1cYtrSQ1x1MDAxMLSlZ1x1MDAxNsc7h3NOpStrXGLZXHUwMDAwZlWuPlx1MDAxZpBEvP92KL7sqfVcdTAwMWKxc9n9dNRcbs6XXHS296jSTCdzu9pLekhcdTAwMTZXL0GvXlwiUlx1MDAxM0XLz8ufvnNEKk1cdTAwMTnT5I9NmJ+xSVx1MDAxNbYnXHUwMDAytiNinlx1MDAxYmCB22W9U/i8Iy2BXHUwMDFiLfwpuFx1MDAxZJJcdTAwMDXQXHUwMDAzd1x1MDAwN9vZqJ+eXHUwMDAwx0+A4yHA8diJZVx1MDAwMD1w/Fx1MDAxNDh+XG5cdTAwMWM/XHUwMDAzjp9cdTAwMDH1gVx1MDAwMcfPgfPDgfJcbuB4JFBeXHUwMDA1pNdAeTX090PtXHUwMDExXHUwMDAxXHUwMDE1wklcZkNcdTAwMTiAS4bBPlx1MDAwYupUnL1cdTAwMGIhXGZQt1xi9UNcdTAwMTjqiDDUXHUwMDEzOXltyGnIXHUwMDA0yGDXk0JcdTAwMThcdTAwMTiUXHUwMDAxKoPjL6BZc1xiXHUwMDAz8KxWbDdGgVx1MDAxOUK2PtpcZlx1MDAwMSBcdTAwMGU+yNpmXGKoWtt3K0Fl8HeuuFxm4CH9itO1cyNcYrqEsom1Uud3X798PDyg3YpqdDfkTkinXHUwMDAzRtS+pZeW3EDJXGKKVXpXXGZVWrDSQy5cdTAwMTetXHUwMDBlsMxRmsSRKl56XHUwMDFmpZx6/Fx1MDAwN1x1MDAxNlJcYoTe1v0jj61cdTAwMGXj4zLMzDWT4dL/Jp1q/7ZnnvB/v13Hg1eOKY1cdTAwMGbvXHUwMDFlXHUwMDFjw3Kl19tcdTAwMWKa2TXf3pvp8nU9XHUwMDE5rVx1MDAxNpX8XHUwMDFmZ+M/y+9cdTAwMWXMOlXwsSv5++e7n/9cdTAwMDHV+b4tIn0=payloadtcpipv4wireguardudpipv6ipv4pppoetelco cruft6to4dest: HE's tunnel brokerdest: our wireguard endpointdest: per user trafficwireguard headeruser packet (encrypted)

Surprisingly, this gross hack only increases overhead by 100 bytes (okay, that is a lot, but I was expecting more). Here's the breakdown:

I'm not counting overhead from the outer PPPoE connection and the TCP/IP headers in the user data because these headers are present regardless.

The performance of putting a tunnel inside another tunnel will probably not be as good as not putting a tunnel inside another tunnel. But how bad is it?

John, who did the majority of the implementation, set out to investigate just how bad this idea actually was.

He initially squeezed a measly 230 Mbps over a connection advertised as being capable of 940 Mbps, but with some careful tuning of the MTU and iperf parameters, pushed 680 Mbps of UDP traffic through this double tunnel setup (see the middle pane in the screenshot below).

Top pane: pushing the maximum 940 Mbps of traffic outbound from John's home Centurylink GPON service. Middle Pane: 680 Mbps of UDP traffic received at our router at the SIX! Bottom pane: Graph of traffic through Bond0.510 --our router's connection with the SIX fabric, and where we peer with HE, indicating this traffic is going over a peering link rather than a transit link.

Image credit: John Elliott

His MikroTik RB5009 sat around 70% CPU during this abuse, indicating the hardware is sufficient to make line rate (at least for large packet sizes).

Why Tunnel at All?

Even straightforward VPN tunnels cut into performance. One must wonder why tunnel at all?

One might use VPN tunnels for financial reasons. Virtual layer 2 circuits, eg. via metro ethernet or similar technologies, are expensive (let's not even mention running one's own fiber or fixed wireless). Consumer-grade internet connections are cheap by comparison.

However, this assumes that one would want to network sites that already have internet connections together at all. One reason is to monitor equipment, allow secure remote management, and provide network services (dns, ntp, etc.) to both equipment and user devices.

I mentioned in the beginning that it could be desirable for traffic from these sites to exit from a different ASN, opposed to the consumer ISP connection. Among other things, this allows the ASN owner to receive and handle DMCA complaints themselves (look for a link here to our 2021 City of Seattle report that describes the issue). You could also use split tunneling to shed load from the VPN by routing popular streaming services outside of the tunnel, for example.

Acknowledgments

I owe thanks to Seattle Community Network for providing the infrastructure necessary to explore this interesting hypothetical.

Thanks to John Elliott for doing the hands-on-keyboard work configuring the network gear, running performance tests, and providing the traceroutes and screenshots.

Thanks to Esther Jang for pre-work in scoping the problem, helping set up the Seattle Community Network infrastructure on which these experiments were run, and editing and giving feedback on this article.

Thanks to doof.net for providing hosting and support to Seattle Community Network.

Also thanks to the numerous other volunteers and donors to Seattle Community Network whose contributions of money, rackspace, bandwidth, physical space, etc. make the project possible.


Footnotes

1

One downside I've heard about HE's IPv6 tunnel broker is that some treat it with suspicion. For example, Netflix blocks it because they can't tell which country you're connecting from, and Wikipedia doesn't allow edits from these IP blocks.

2

Namely, the reduced MTU, and the additional latency caused by the additional processing steps.

3

Side-note: hop 2, 100.92.123.67, is within the 100.64.0.0/10 address space reserved for Carrier-Grade NAT on provider networks in RFC 6598. I just thought that was interesting.

4

Centurylink hops through Level3, which became part of Centurylink, (which is now Lumen) on its way to HE. Presumably Lumen is not charging themselves for transit. Comcast connects directly to HE.