burble.dn42 / Additional Info / Maintenance Log

Maintenance Log

A log of changes to the burble.dn42 network.


10th April 2021

The b.recursive-servers.dn42 DNS resovler is running an experimental build of pdns-recursor to test a fix of this issue.
Please let me know if you spot any strange problems.

burble.dn42 websites are now using a TLS certificate issued by the DN42 ACME service.

A number of significant changes have been implemented for the global route collector

  • Downstream peerings have been stopped, in favour of parsing the MRT dumps
  • The collector has moved from de-fra1 to fr-rbx1, where bandwidth is no issue
  • A special routing policy has been implemented for the collector to encourage traffic to go directly to fr-rbx1 and not transit through burble.dn42 nodes. See also the Routing Policy page.
  • Internal rate limits on BGP sessions have been relaxed

3rd April 2021

The collector is now using a TLS certificate issued by the DN42 ACME service. The collector is behind an anycasted reverse proxy, so a normal ACME challenge will not work. Instead, the certificate is managed using dnscontrol to respond to an ACME DNS challenge.

DNSSEC has been enabled on all edge nodes.

2nd April 2021

There was a major DNS outage today as a minor change took out the entire service.

What should have been a trivial config change actually upgraded the container from Alpine 3.11 to Alpine 3.13 and caused a number of the DNS applications to stop working due to incompatibilities.

The lack of working DNS meant it was more complicated to bootstrap the service back again, leading to a long delay in restoring service.

27th March 2021

Fixed a bug in bird that was preventing MRT dumps from the collector working. Hopefully the dumps can now be successfully parsed: https://mrt.collector.dn42

25th March 2021

Bird 2.0.8 has been deployed across the network. Please let me know if you see problems.

burble.dn42 uses a custom bird build that includes additional debugging. The source code for the build is available on git.burble.dn42.

Advanced Notice

  • us-nyc1 will be decommissioned before 15/04/21
  • us-chi1 will be decommissioned before 14/05/21

23rd February 2021

Updated IPv6 address for hk-hkg1

10th January 2021

Upgraded the looking glass to use bird-lg-go.

The main benefit of the go version is that it executes queries in parallel, greatly improving response times with a large number of nodes.

6th January 2021

hk-hkg1 is now open for IPv4 peering; see the node information for details.

IPv6 connectivity is expected ~February.

4th January 2021

Happy New Year DN42.

New Website

The new year brings a new website for burble.dn42 built using Hugo and statically delivered from each core node for speed. As always, the source for the website is available in the gitea repo.

Anycast MTU

The MTU for anycast services has been reduced to 1280 after a problem was seen with IPv6 path MTU discovery.

The problem was due to an asymmetric path, where a request to the wiki went to one node but the return path was via a different node. The other node also hosted a wiki instance, which meant that pmtud ICMP messages on the return path were being picked up by the wrong node. To fix this, the MTU has been clamped to the minimum allowable size of 1280.

Interestingly, Cloudflare also recognised the same type of issue and wrote up what they did in their blog.

The following services were impacted by the changes.

  • DNS Services
  • NGINX Reverse Proxy (and therefore also all websites, including the Wiki mirrors)
  • WHOIS Service

New Nodes

es-mad1 in Madrid, Spain has already been delivered and is now open for peerings.

The new node in Hong Kong, hk-hkg1 has also been delivered and I’m now just waiting for IPv6 to be available before it too will also be ready for peering.


Historical changes from previous years