𝙷𝚈𝙿𝙾𝙳𝚈𝙽𝙴

Visualizing Computer Hardware Topology

2025-11-17T00:00:00+00:00

There are times when you need a clear understanding of how a computer's hardware goes together. In my case, I'm working on a project using XDP</a> to rapidly process packets in the kernel, bypassing the Linux networking stack.</p>

For my purposes in testing and development, it's important to know which network cards are connected to which CPUs. For best performance, packet data should not cross the interconnect from one NUMA node</a> to another, but should instead be handled all within the same NUMA node.</p>

In many cases lshw</code> does the needful, but its output is rather verbose and one can lose sight of the forest for the trees:</p>

char@underbed:~$ sudo lshw | head -n 40
</span>underbed
</span>    description: Rack Mount Chassis
</span>    product: PowerEdge R630 (SKU=NotProvided;ModelName=PowerEdge R630)
</span>    vendor: Dell Inc.
</span>    serial: 5QP0B42
</span>    width: 64 bits
</span>    capabilities: smbios-2.8 dmi-2.8 smp vsyscall32
</span>    configuration: boot=normal chassis=rackmount sku=SKU=NotProvided;ModelName=PowerEdge R630 uuid=4c4c4544-0051-5010-8030-b5c04f423432
</span>  *-core
</span>       description: Motherboard
</span>       product: 0CNCJW
</span>       vendor: Dell Inc.
</span>       physical id: 0
</span>       version: A05
</span>       serial: .5QP0B42.CN7475151G0196.
</span>     *-firmware
</span>          description: BIOS
</span>          vendor: Dell Inc.
</span>          physical id: 0
</span>          version: 2.19.0
</span>          date: 12/12/2023
</span>          size: 64KiB
</span>          capacity: 16MiB
</span>          capabilities: isa pci pnp upgrade shadowing cdboot bootselect edd int13floppytoshiba int13floppy360 int13floppy1200 int13floppy720 int9keyboard int14serial int10video acpi usb biosbootspecification netboot uefi
</span>     *-cpu:0
</span>          description: CPU
</span>          product: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
</span>          vendor: Intel Corp.
</span>          physical id: 400
</span>          bus info: cpu@0
</span>          version: 6.63.2
</span>          slot: CPU1
</span>          size: 1974MHz
</span>          capacity: 4GHz
</span>          width: 64 bits
</span>          clock: 3705MHz
</span>          capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts vnmi md_clear flush_l1d ibpb_exit_to_user cpufreq
</span>          configuration: cores=8 enabledcores=8 microcode=73 threads=16
</span>        *-cache:0
</span>             description: L1 cache
</span>...
</span></code></pre>
There's another useful tool called lstopo</code> that is part of the
hwloc</code> package in many Linux distributions. lstopo</code> creates a
visualization of the system's layout that filters out less important
information and even elides repeating objects to reveal that forest
which lshw</code> loses sight of. Here's an example output from the same
system as seen above:</p>
</p>
Now one can clearly see what PCIe hardware is homed to which NUMA
node. Node 0 on the left has sda</code>, my disk drive along with 2 NICs.
PCI 03:00 is a 2-port Intel 82599 10GbE NIC that I got for $15 from
eBay and PCI 01:00 is a 4-port NIC that came with the server. On Node
1, there's another 2-port 82599 on PCI 81:00.</p>
Along with the peripherals, you can also clearly see each NUMA node
has shared RAM, an L3 cache common to all cores, and individual L2 and
L1 caches per core.</p>
Pretty neat!</p>


Getting 3GPP Specifications
2025-10-27T00:00:00+00:00
Presently, the specifications for mobile telecom (i.e. 4G LTE, 5G,
etc.) are developed and freely published by groups working within the
3GPP (3rd Generation Partnership
Project)</a>. The 3GPP is sort of
analogous to the IETF (Internet Engineering Task Force), the
organization tasked with developing and promulgating internet
standards.</p>
As a newbie in the telecom world (and the internet world too, to be
clear), I find the 3GPP's specifications challenging to locate and
comprehend so I've written some notes that are hopefully useful to
myself in the future as well as others.</p>
Getting Specification PDFs</h1>
When you already know which specification you're looking for, head
over to the 3GPP specification search
portal</a> and enter the
title or specification number in the search box.</p>
Getting from the results page to a copy of the spec is not
straightforward.</p>
First, click the little glasses-on-a-sheet-of-paper icon on the right
side (circled in red in the screenshot).</p>
</p>
Clicking the glasses icon opens a new window where you must choose the
version of the specification you're interested in. Select the
Versions</em> tab and then click the green ETSI</em> button to the right of
the version you're interested in.</p>
</p>
Yet another window will open, this one titled "Work Programme". Click
the icon circled in red under the Download Standard</em> heading to
download a PDF of the specification (the button on the right downloads
a Word document, but I don't recommend it because the Word files are
huge and slow to navigate compared to the PDFs).</p>
</p>
Finally, you'll have your specification PDF. That was a lot of work so
you might want to save it to avoid repeating the process.</p>
</p>
Tips for Finding the Right Specification</h1>
Figuring out which specification contains the information you need is
a bit more challenging. A good approach might involve reading
Wikipedia or other summaries and following their citations.</p>
Series Organization</h2>
The 3GPP organizes specifications into different series which are
identified by the first number the dot in the specification's name.
For example, TS 29.244 is part of series 29. Sometimes the document
name includes a leading 1, eg. TS 129.244. I haven't found an
explanation as to why. Drop the leading 1, if present.</p>
These notes are mostly about the technical specification (TS)
documents, not the technical report (TR) documents. This is because I
haven't read any TR documents yet.</p>
Note: Series numbers < 21 are only about pre-LTE GSM 2G stuff, so you
can ignore them if that's not relevant.</p>
I am nowhere close to being an expert on the different series, but
here are my feelings on a few:</p>

Series 21, Requirements: These are lists of other specifications
relating to a given topic. Essentially, they are an index that is
the jumping off point into the specifications you'll need. For
example, 21.201 lists all the specifications necessary for the
Evolved Packet System (EPS).</li>
Series 22, Service Aspects: Very high-level descriptions of what</em> a
mobile network should do from a user or service's perspective. These
documents are often described as "stage 1" descriptions. Some of the
contents seem pretty obvious if you've used a cell network before. I
have not read any of these yet, probably to my detriment.</li>
Series 23, Technical Realization: These are described as "stage 2"
documents and seem to focus on how</em> a network should achieve the
stuff outlined in series 22. Series 23 covers the messaging
exchanged between different entities in the network ("procedures"
in 3GPP jargon). These documents include diagrams showing the
ordering of messages and can be really useful for understanding how
things are supposed to work at a medium-high level. Out of the 23
series, I most often refer to TS 23.401 (E-UTRAN Access).</li>
Series 24, 29, Signaling Protocols: So-called "stage 3"
descriptions. These are down-to-bits specifications for
communications protocols. They answer the "how" and sometimes "what"
but rarely "why" questions. Specs for GTPv2-C (TS 29.274) and PFCP
(TS 29.244) live in series 29 which covers internal network
signaling. I haven't ready any specs in series 24, which covers
UE-network signaling (I mostly work on core network stuff). These
documents explain the semantics of a protocol and define its
messages. The protocols defined here are mentioned in the series 23
specifications. Be careful because the series 23 terms describing
the contents of a given message type can occasionally differ
slightly.</li>
Series 36, LTE Radio Aspects: I don't refer to these often, as I
don't work on the radio part. I've skimmed parts of TS 36.300
(E-UTRAN Overall Description) a couple of times.</li>
</ul>
Pro-Tips</h1>

When learning a new specification it's often easiest to start by
reading an early (in many cases for LTE, this is release 8) version
of the specification. Earlier versions have less extra distracting
fluff.</li>
These specs can be really, really dense and require a lot of context
from other specs to comprehend. I've found that it sometimes takes
months to understand parts of a spec. It is hard, but keep at it.
Don't be afraid to put the spec down and come back later if
something doesn't make sense.</li>
Keep notes in the margins of the document explaining difficult parts
of the spec or summarizing walls of text. You'll be happy you did
when you have to come back to these parts.</li>
First, cross-out the parts of the spec that do not apply to your
situation, if applicable. This will save tremendous time in
aggregate as you can quickly skip over irrelevant parts each time
you refer to the spec.</li>
Russell DeLong has posted some great lectures of LTE concepts on
his YouTube channel. I recommend watching them as an introduction to
any of the topics they
cover.</a></li>
</ul>


Using Erlang Records in Elixir
2024-11-13T00:00:00+00:00
Introduction</h1>
This is a post about effectively using Erlang records in Elixir code.</p>
Lately I've been working on an Elixir project that uses a few Erlang
libraries that represent data using the record data type.</p>
Here's how Erlang's documentation describes records:</a></p>

A record is a data structure for storing a fixed number of elements.
It has named fields and is similar to a struct in C. [...] record
expressions are translated to tuple expressions during compilation</p>
</blockquote>
Records exist in Elixir too but are mostly there for compatibility
with Erlang (for a variety of reasons you're better off using a struct
or a map than a record in pure Elixir code). The official Elixir docs
about records</a> are brief
and left me unsure about how to actually use them, so after some
experimenting I've made this brief guide for reference. I hope you
find it useful!</p>
Prerequisites</h1>
This guide assumes basic familiarity with Elixir's syntax, pattern
matching, and tooling. While no prior Erlang experience is required,
you should be comfortable working with Elixir's basic data types like
tuples and maps.</p>
Overview of Records in Elixir</h1>
Broadly, Elixir's Record module lets you do two things:</p>
1. Extract Definitions</strong></p>
Extract definitions of records from existing Erlang header files into
an Elixir module as tuples. The functions extract/2</code> and
extract_all/1</code> are used for this.</p>
2. Generate Helpers</strong></p>
Generate helpers for creating, accessing, and pattern matching record
data types from tuples describing records. The macros defrecord/3</code>
and defrecordp/3</code> are used for this.</p>
It's crucial to note that you need to do both steps (extract and
define) to actually use the records defined in Erlang files inside
Elixir code. Extracting record definitions from Erlang headers results
in tuples describing the records, but does not generate the helpers
for creating, accessing, and pattern matching on records.</p>
Simply put, you need to plumb the output of extract/2</code> or
extract_all/1</code> into defrecord/2</code> (or defrecordp/3</code>) to use Erlang
records in Elixir code.</p>
Examples of Extracting and Defining Records</h1>
As an example, the code below extracts the :gtp</code> record type from the
library gtp_packet.hrl</code> and puts a tuple defining the record into
gtp_def</code>. Then, :gtp</code> and gtp_def</code> are passed to
Record.defrecord/3</code> to generate code for using the gtp</code> record type:</p>
  gtp_def = </span>Record</span>.extract(</span>:gtp</span>, </span>from_lib: </span>"</span>gtplib/include/gtp_packet.hrl</span>")
</span>  </span>Record</span>.defrecord(</span>:gtp</span>, gtp_def) </span># generates the macros gtp/0, gtp/1, gtp/2
</span></code></pre>
Extracting and defining records can get really tedious if you find
yourself working with many different record types. Thankfully, Elixir
also provides a way to extract records in bulk from Erlang headers.
Unfortunately, there's no way to define record helpers in bulk.
However, the Enum</code> module and a simple lambda function comes to the
rescue!</p>
  # Define helpers dynamically based on the extracted records
</span>  </span>Enum</span>.each(</span>Record</span>.extract_all(</span>from_lib: </span>"</span>gtplib/include/gtp_packet.hrl</span>"), </span>fn </span>{name, fields} ->
</span>    </span>Record</span>.defrecord(name, fields)
</span>  </span>end</span>)
</span></code></pre>
Each tuple from extract_all/2</code> of {name, fields}</code>, where name</code> is
simply the name of the record and fields</code> is a tuple describing the
record's fields is passed into defrecord/3</code> just as in the
manually-done first example we saw earlier. I think it's kind of
strange there's no extract_and_define</code> function in Elixir that does
this all for you, but whatever, this way gives you a little more
control over things (for example, you could modify the names, such as
putting a prefix on them, to avoid a collision if you're importing
records from several different Erlang libraries into the same Elixir
module).</p>
Using Records in Elixir</h1>
After you've managed to generate code for working with records using
defrecord/3</code> you can finally use them! If you called
defrecord(:foo, foo_defs)</code> (where foo_defs</code> was returned from extract/2</code>)
you'd end up with three new macros all named foo</code>. Let's imagine the record
type foo</code> contains the fields bar</code> and baz</code>. In fact, you can do all this
in iex</code> as long as you wrap in inside a module definition:</p>
iex> </span>defmodule </span>Foo </span>do
</span>...> </span>Record</span>.defrecord(</span>:foo</span>, [</span>bar: </span>:undefined</span>, </span>baz: </span>:undefined</span>])
</span>...> </span>end
</span></code></pre>
Creating Records</h2>
Now let's use foo/0</code> to create a foo</code> record with default values
(spoiler: it's boring, but it's a start):</p>
iex> </span>Foo</span>.foo()
</span>{</span>:foo</span>, </span>:undefined</span>, </span>:undefined</span>}
</span></code></pre>
Notice the result is a tuple starting with :foo</code>.</p>
To make things a little more interesting we can set values in the
foo</code> we're creating:</p>
iex> </span>Foo</span>.foo(</span>bar: </span>:asdf</span>, </span>baz: 1234</span>)
</span>{</span>:foo</span>, </span>:asdf</span>, </span>1234</span>}
</span></code></pre>
Inspecting Records</h1>
Let's say we have a record like {:foo, :asdf, 1234}</code> and that we're
forgetful and can't remember the order of bar</code> and baz</code> inside the
record. We can use the foo/1</code> macro generated by defrecord/2</code>
earlier to see the keys and the values laid out clearly:</p>
iex> </span>Foo</span>.foo({</span>:foo</span>, </span>:asdf</span>, </span>1234</span>})
</span>[</span>bar: </span>:asdf</span>, </span>baz: 1234</span>]
</span></code></pre>
Accessing a Field within Records</h2>
You can access individual fields within a record using square
brackets, similarly to a Map. To do this, you pass the tuple
representation of a record into the generated function with 1-arity,
here foo/1</code>.</p>
iex> my_foo = </span>Foo</span>.foo(</span>bar: </span>:asdf</span>, </span>baz: 1234</span>)
</span>{</span>:foo</span>, </span>:asdf</span>, </span>1234</span>}
</span>iex> </span>Foo</span>.foo(my_foo)[</span>:bar</span>]
</span>:asdf
</span>iex> </span>Foo</span>.foo(my_foo)[</span>:baz</span>]
</span>1234
</span></code></pre>
The generated 2-arity function foo/2</code> allows for the same with
slightly different syntax:</p>
iex> </span>Foo</span>.foo(my_foo, </span>:bar</span>)
</span>:asdf
</span>iex> </span>Foo</span>.foo(my_foo, </span>:baz</span>)
</span>1234
</span></code></pre>
Now we can clearly see that bar</code> is set to asdf</code> and baz</code> is set to
1234</code>. This technique can come in really handy for working with
records in the repl if you want to inspect them.</p>
Pattern Matching on Records</h2>
However, we can go further and even pattern match on fields within
records with similar syntax to pattern matching on Maps. Taking the
same foo</code> example record as before, bar</code> and baz</code> can be pattern
matched into their own variables:</p>
iex> my_foo = </span>Foo</span>.foo(</span>bar: </span>:asdf</span>, </span>baz: 1234</span>)
</span>{</span>:foo</span>, </span>:asdf</span>, </span>1234</span>}
</span>iex> </span>Foo</span>.foo(</span>bar:</span> bar, </span>baz:</span> baz) = my_foo
</span>{</span>:foo</span>, </span>:asdf</span>, </span>1234</span>}
</span>iex> bar
</span>:asdf
</span>iex> baz
</span>1234
</span></code></pre>
Pattern matching is very useful if an Erlang library hands you a
record from which you want to concisely extract multiple values.
Pattern matching can also be done in function heads to select a
specific implementation of a function based on the values of its
inputs. For more details on pattern matching see the Elixir docs.</p>
Updating Records</h2>
Finally, you can update fields in an existing record (leaving other
fields untouched) using the 2-arity version of the generated macro,
again foo/2</code> here:</p>
iex> my_foo = </span>Foo</span>.foo(</span>bar: </span>:asdf</span>, </span>baz: 1234</span>)
</span>{</span>:foo</span>, </span>:asdf</span>, </span>1234</span>}
</span>iex> </span>Foo</span>.foo(my_foo, </span>bar: </span>:qwerty</span>)
</span>{</span>:foo</span>, </span>:qwerty</span>, </span>1234</span>}
</span>iex> </span>Foo</span>.foo(my_foo, </span>bar: </span>:qwerty</span>, </span>baz: 5678</span>)
</span>{</span>:foo</span>, </span>:qwerty</span>, </span>5678</span>}
</span></code></pre>
Summary</h1>
I've tried to show in detail with plenty of examples how to use the
functionality of Elixir's Record</code> module. This includes extracting
record definitions from Erlang header files using extract/2</code> and
extract_all/2</code>, generating macros for working with records via
defrecord/3</code> and defrecordp/3</code>, and examples of how to read, write,
and update record data types in the previous few subsections.</p>
Hopefully this guide fills in the gaps in the official documentation
and makes at least one other person's life a little easier.</p>


Tunnels Within Tunnels: One Weird Trick for Free IP Transit
2023-10-28T00:31:47-07:00
2023-11-16 update: John Elliott published a post with router configuration
details</a></em></p>
Imagine you took your homelab hobby to the next level by acquiring an
ASN. You don't have much money, but you've still managed to connect to
an IXP and peer with other ASs over the exchange fabric. You'd like to
connect your network back to sites scattered across a metro region,
often with only consumer internet connections because other types of
service are too expensive.</p>
You may want to network the sites together and may also want the
sites' internet-bound traffic to exit through your own AS (there are a
number of reasons you might want to do this --see the last section).
Hence, you decide to set up VPN tunnels between your sites and your
network equipment at the IXP.</p>
Unfortunately, the large regional ISPs servicing your sites will not
peer with you because you are small fry, so the tunneled traffic
doesn't reach your router at the IXP via settlement-free peering
relationships, instead it goes through a paid upstream transit
provider</a>.
You don't want to increase the traffic through your transit provider
more than you have to because it's expensive and limited. Is there a
solution?</p>
Recently, while tinkering with VPN tunnels with other volunteers from
the Seattle Community Network</a>
(eg. John Elliott</a>), I had a sudden flash of
inspiration. An absurd</em> idea emerges...</p>
Enter the Hurricane Electric Free IPv6 Tunnel Broker</h2>
The Hurricane Electric Free IPv6 Tunnel
Broker</a> is a free service that HE provides
out of the goodness of their hearts. If your ISP's IPv6 service is
lacking or nonexistent, no problem! HE will generously lend you a /64
IPv6 address block and provide you with a tunnel endpoint address
where you can send IPv6 using RFC-compliant 6to4 technology!</p>
For costing nothing, HE's IPv6 service is
amazing</em>1</a></sup>. Here in Seattle on a Centurylink
connection HE's tunnel broker generally outperforms Centurylink's own
6rd service. And unlike 6rd, the HE IPv6 block is static, making it
far more useful for running servers at home.</p>
Besides providing free IPv6 forwarding, Hurricane Electric also freely
peers with other networks at exchange points, no matter the size.</p>
Hmm... Could we use HE's network to bypass an upstream transit
provider?</p>
It turns out yes, and the performance is much better than any of us
expected.</em></p>
A disclaimer: Don't abuse this. There are also some details about
why this is kind of technically icky2</a></sup>. But we had to
see whether it worked (and how badly).</strong></em></p>
Read on...</p>
The Initial Configuration (via Paid Upstream Transit Provider)</h2>
The inital tunneling config is pretty straightforward. We're running a
VPN tunnel endpoint on one of our servers at the IXP. Internet-bound
traffic is NATted with an address from our public IPv4 block.</p>
Routers at the remote sites connect to the VPN endpoint at the IXP.
Internet-bound traffic is routed to the IXP via the tunnel, unless the
tunnel is down because we messed something up, in which case traffic
bypasses the tunnel and directly egresses via the consumer ISP
connection as a fail-over.</p>
We're using WireGuard</a> for these tunnels.
So far WireGuard has been simple to configure (though there are a few
quirks), fast, and ...fast. MikroTik's RouterOS 7, released in the
last year, supports WireGuard, allowing us to finally use it on our
MikroTik hardware. Other volunteers, John
Elliott</a> and Esther
Jang</a>, did the bulk of the
work figuring out the WireGuard and firewall configurations.</p>
John used iperf to achieve a respectable 860 Mbps over the tunnel
between his home 940 Mbps GPON connection to the VPN server.</p>
These are two traceroutes from different consumer ISPs showing that
the path taken to our test server includes a hop through an upstream
transit provider (AS1299):</p>
$ mtr -w -r -z -c 5 23.146.168.1
</span>HOST: foo                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
</span>1. AS???    rb5000.home.arpa                      0.0%     5    2.1   1.0   0.7   2.1   0.6
</span>2. AS209    tukw-dsl-gw69.tukw.qwest.net          0.0%     5    3.2   3.0   2.9   3.2   0.2
</span>3. AS209    63-226-198-33.tukw.qwest.net          0.0%     5    3.0   3.3   2.1   5.1   1.1
</span>4. AS???    ???                                  100.0     5    0.0   0.0   0.0   0.0   0.0
</span>5. AS3356   4.69.219.210                          0.0%     5    3.5   3.6   3.3   3.9   0.3
</span>6. AS1299   sea-b1-link.ip.twelve99.net           0.0%     5    3.5   3.8   3.5   4.3   0.3
</span>7. AS1299   doof-ic-375811.ip.twelve99-cust.net   0.0%     5    3.5   3.4   3.1   3.6   0.2
</span>8. AS54429  seattlecommunitynetwork.org           0.0%     5    3.8   3.6   3.5   3.8   0.1
</span></code></pre>
A traceroute from a Centurylink residential connection to our router
at the SIX transits 1299.</em></p>
$ mtr -w -r -z -c 5 23.146.168.1
</span>HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
</span>  1. AS???    router.asus.com                                    0.0%     5    1.2   3.2   1.2   9.2   3.4
</span>  2. AS???    100.92.123.67                                      0.0%     5    8.9  11.6   8.9  16.6   3.0
</span>  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   13.1  11.8   9.6  14.1   2.0
</span>  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5    9.9  11.4   8.6  15.4   2.6
</span>  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5    9.7  17.4   9.7  29.9   7.8
</span>  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net         0.0%     5   10.2  15.0  10.2  31.1   9.0
</span>  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   12.5  10.9  10.1  12.5   1.0
</span>  8. AS7922   50.208.233.194                                    40.0%     5   32.0  25.1  20.8  32.0   6.0
</span>  9. AS???    ???                                               100.0     5    0.0   0.0   0.0   0.0   0.0
</span> 10. AS1299   doof-ic-375811.ip.twelve99-cust.net                0.0%     5   15.1  14.7  10.4  25.1   6.1
</span> 11. AS54429  seattlecommunitynetwork.org                        0.0%     5    9.8  15.2   9.8  35.5  11.3
</span></code></pre>
A traceroute from a Comcast residential connection3</a></sup> to our router
at the SIX hops through 1299 too.</em></p>
This cutesy diagram shows roughly the same information as the
traceroutes but with a larger font and some arrows.</p>

    The Path to the Hurricane Electric IPv6 Tunnel Broker</h2>
Let's do a few more traceroutes to see the path the traffic takes.
We'll start with an IPv6 traceroute over the HE IPv6 tunnel broker to
Seattle Community Network's router (2602:fd94::1) at the Seattle
Internet Exchange:</p>
[asdf@rb5] > /tool/traceroute 2602:fd94::1
</span>Columns: ADDRESS, LOSS, SENT, LAST, AVG, BEST, WORST, STD-DEV
</span>#  ADDRESS            LOSS  SENT  LAST     AVG  BEST  WORST  STD-DEV
</span>1  2001:470:a:f00::1  0%     101  4ms      4.4  2.8   31.9   2.9
</span>2                     100%   101  timeout
</span>3  2602:fd94::1       0%     100  3.2ms    4    2.1   67.7   6.5
</span></code></pre>
(First hop IP address slightly redacted)</em></p>
This traceroute doesn't show hostnames or AS numbers, but that's ok.
What's notable is that this traffic seems to hop right into our
network, after a mystery hop at hop #2 that I suspect is some
HE-internal stuff we can't see. It appears traffic enters HE's network
via their IPv6 tunnel broker then passes through the peering
connection between HE and our router at the IXP, rather than going
through a transit provider. This makes sense because the peering link
should have fewer hops, seeing as the traffic is already inside HE's
network.</p>
That last traceroute was run inside the tunnel so it doesn't show any
of the hops that happen before the packets reach HE. To see the path
taken by the tunnel itself, in other words the outside view, we can
run traceroute to the IPv4 address of our HE tunnel broker endpoint
216.218.226.238.</p>
Here's an example of an outside-the-tunnel traceroute to the HE tunnel
broker endpoint taken from a Comcast residential connection. Pay
attention to the AS numbers, you can see that Comcast (AS7922)
connects to Hurricane Electric (AS6939) without an intermediate hop.
Looks like they peer with each other.</p>
$ mtr -w -r -z -c 5 216.218.226.238
</span>HOST: bar                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
</span>  1. AS???    router.asus.com                                    0.0%     5    2.3   3.0   1.2   7.5   2.6
</span>  2. AS???    100.92.123.67                                      0.0%     5   10.2  11.1   8.7  14.4   2.1
</span>  3. AS7922   po-323-406-rur302.seattle.wa.seattle.comcast.net   0.0%     5   16.0  11.5   9.8  16.0   2.6
</span>  4. AS7922   po-300-xar02.seattle.wa.seattle.comcast.net        0.0%     5   12.1  10.9   9.0  12.2   1.4
</span>  5. AS7922   be-304-arsc1.seattle.wa.seattle.comcast.net        0.0%     5   11.9  13.7   9.8  20.1   3.9
</span>  6. AS7922   be-36131-cs03.seattle.wa.ibone.comcast.net        60.0%     5   13.6  12.1  10.5  13.6   2.1
</span>  7. AS7922   be-2313-pe13.seattle.wa.ibone.comcast.net          0.0%     5   21.9  14.0  10.2  21.9   5.0
</span>  8. AS7922   50.208.233.194                                    20.0%     5   28.1  20.7  10.5  31.1  10.4
</span>  9. AS6939   tserv1.sea1.he.net                                 0.0%     5   11.7  10.4   9.8  11.7   0.8
</span></code></pre>
An mtr</code> traceroute from a Comcast residential connection to the
Hurricane Electric Free IPv6 Tunnel Broker</em></p>
Cramming Tunnels Inside Tunnels</h2>
The last piece is getting the user traffic to take the path through HE
shown above. Briefly, we</p>

Set up HE free IPv6 tunnels at user sites.</li>
Send site traffic over an IPv6-addressed</strong> VPN to our network.</li>
</ul>
In other words, we put a VPN tunnel inside</em> HE's 6to4 tunnel. Our
inner VPN tunnel must be destined to an IPv6 address so that it goes
inside the HE 6to4 tunnel. However, the traffic inside our innermost
tunnel can still use IPv4.</p>
Traffic takes a path that looks something like this:</p>

    The "???" network represents hops between a residential ISP's network
and HE. In our case on Centurylink and Comcast there's a direct
connection^{4</a></sup> to HE (AS6939), but other ISPs might go through a
transit provider. Also, AS1299 and AS395823 are no longer part of the
traffic's path so they're grayed out.</p>}
At this point, we've discovered a path from other networks back to our
network at the SIX that doesn't involve a transit provider. This
traffic doesn't cost us anything (because we peer with HE) and it
might not</em> cost HE anything either (because they're probably peering
freely with big ISPs).</p>
Performance</h2>
With WireGuard as the innermost tunnel, the packets look something
like this mess:</p>

𝙷𝚈𝙿𝙾𝙳𝚈𝙽𝙴

Visualizing Computer Hardware Topology

Getting 3GPP Specifications

Using Erlang Records in Elixir

Introduction</h1> This is a post about effectively using Erlang records in Elixir code.</p> Lately I've been working on an Elixir project that uses a few Erlang libraries that represent data using the record data type.</p>

Tunnels Within Tunnels: One Weird Trick for Free IP Transit

The Path to the Hurricane Electric IPv6 Tunnel Broker</h2> Let's do a few more traceroutes to see the path the traffic takes. We'll start with an IPv6 traceroute over the HE IPv6 tunnel broker to Seattle Community Network's router (2602:fd94::1) at the Seattle Internet Exchange:</p>

Cramming Tunnels Inside Tunnels</h2> The last piece is getting the user traffic to take the path through HE shown above. Briefly, we</p> Set up HE free IPv6 tunnels at user sites.</li>