Feb 1 2024

Appreciation of automated IX Quarantine LAN testing

Something that bgp.tools (my company) does a great deal is joining internet exchanges. In 2023 the bgp.tools route collector joined around 40 internet exchanges, and while I don’t think I will do another repeat of 2023’s level of expansion, I’ve already set up my first exchange for 2024.

That exchange is IX.BR São Paulo, the first exchange that bgp.tools has joined in Latin America. While the grunt work of joining exchanges can be boring, IX.BR has an interesting take on the quarantine LAN procedure that I think is unique, interesting, and well done. So therefore I believe it is worth documenting since not everybody gets to see these processes when they are done well.

But first!

Why do we have the quarantine LAN?

Quarantine LANs on internet exchange points are very important, as almost all exchanges are layer 2 broadcast domains with all member routers on the same subnet, there are several things that any member can accidentally do to greatly impact other members ability to exchange traffic.

Ethernet itself is not a loop tolerant protocol and exchanges avoid the use of spanning tree, therefore an ethernet switch loop can be extremely detrimental to an IX. Since the resulting broadcast packet flood will a Denial of Service for member routers.

But there are several other quirks the internet exchange participants need to look out to avoid, such as sending special management frames like LLDP, or having significantly more dangerous features enabled like Proxy ARP. Proxy ARP is uniquely dangerous to an internet exchange because it may cause your router to claim that it owns IP addresses on the internet exchange LAN that it does not, causing other members traffic to be directed to the router with Proxy ARP enabled instead of the correct destination.

This is all the more important because exchanges like ix.br have a significant percentage of the country’s internet traffic going through them, in IX.BR São Paulo’s case it peaks (at the time of writing) at 22 Terabits daily!

The IX.BR Quarantine LAN testing

When you have been given your IP details and confirm that the link is established between you and the exchange, you are provided with a URL that takes you to the IX.BR quarantine LAN testing portal.

You can then request tests by yourself! This on its own is a significant quality of life enhancement for network operators as they are not always awake or doing things at the same time that the exchange is awake and answering support requests!

The automatic tests

I ran tcpdump on the device while it was being tested, so I could get a look at what it was specifically doing. Here is what I found it does:

• Check for basic ARP responses and ping responses from the IX Route Server IP addresses

• Check that large (1500 byte) pings are possible, and without fragmentation

• Check that you don’t have Proxy ARP enabled by ARP-ing for addresses that are not on the IX LAN at all

• Create 4096 (ish) devices on the LAN to test if your equipment can handle the full size of the LAN you are about to join

• Check that you don’t have DNS, NTP or SNMP open on your router IX interfaces

The report

At the end of the testing you’re given a report and a final verdict based on what the testing system observed.

In my case I failed the tests a couple of times because I wasn’t established to the BGP Route Servers ( I was not expecting that to be a requirement when I first started the testing ).

Thankfully because of the self-serve nature of this test I could correct that and run the tests again without having to have the support person handling my provisioning involved!

If you’re interested in the raw packet capture data then you can find them here: (Zip File)

Overall I am extremely impressed at this part of IX.BR, and I have not seen anything like this on another exchange! However I can understand that for ix.br (being one of the larger exchanges) that this was likely extremely necessary to continue scaling the exchange!

If you are on IX.BR São Paulo, and wish to peer with the bgp.tools route collector, you can go to https://bgp.tools/kb/setup-sessions to setup sessions.

Until next time!