Jan 13 2022

Ghost in the ethernet optic

A few months ago I stumbled on a tweet pointing out a kind of SFP optic that claimed to be smart, made by a Russian company Plumspace. After going through their Smart SFP product listings I was on-board and decided I needed to have some of them.

You see, Smart SFP’s are a bit of a terrifying concept. SFP’s are (until now apparently) actually quite simple devices that “simply” take input electrical signals and turn them into optical signalling, or carry them down a Direct Attach Cable (DAC)

The proposed smart SFP said, “Hey there is plenty of space in this thing! Why not also put a little FPGA, and an ARM core that can share the ethernet link, that way we can do more things!”

Or in a simple diagram:

After some discussion over email with Plumspace, I ordered 3 (2 going to my similarly interested friends) at around the cost of 150 USD per optic. After some shipping+brexit drama I had a smart-sfp of my own sitting in front of me.

I want to point out that while this whole post may read like a sponsored post, it’s not! I just find the concept of these devices so immensely curious and cursed I feel the need to document their existance and feature set.

The first thing that comes to mind is that these are a lot longer than your standard SFP.

Due to stocking issues, I could only buy the single mode variants of the product. But that is fine since you can run single mode optics with multimode fibre over a short distance (I only had multimode fibre on hand).

I was eager to get this up and running so I just stuck it in my spare SFP port on my workstation. This also was handy as I could isolate the optic from the rest of my home network just in case it came with some nuclear default settings.

It turned out that isolation was a good call, since the default IP address of the “host” end of the system inside would have conflicted with my default gateway (of 192.168.2.1).

After some wrangling with the IP addresses:

[12:00:25] ben@metropolis:~$ ssh admin@192.168.2.1
admin@192.168.2.1's password: 
*************************************************************************
                         Welcome to Smart SFP!
*************************************************************************
admin@smart-sfp:~$ 

The system itself has around 4GB of non-replaceable eMMC storage, where around 2GB of it is usable (I assume the other partitions are for the factory restore image that can be activated using the DIP switches on the SFP itself) (dmesg output here)

root@smart-sfp:~# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0rpmb  179:24   0  512K  0 disk 
mmcblk0boot0 179:8    0   16M  1 disk 
mmcblk0boot1 179:16   0   16M  1 disk 
mmcblk0      179:0    0  3.6G  0 disk 
|-mmcblk0p2  179:2    0  1.4G  0 part /
|-mmcblk0p10 259:2    0  360M  0 part 
|-mmcblk0p9  259:1    0  360M  0 part 
|-mmcblk0p7  179:7    0  100M  0 part 
|-mmcblk0p5  179:5    0  600M  0 part /var
|-mmcblk0p3  179:3    0    1K  0 part 
|-mmcblk0p1  179:1    0    1M  0 part 
|-mmcblk0p8  259:0    0  100M  0 part 
`-mmcblk0p6  179:6    0  600M  0 part /home

CPU wise it’s a single core ARMv7, with 512M of RAM running Debian Jessie. Quite comparable to a first-generation Raspberry Pi. While the optical transceiver is a gigabit optic, the actual Linux system inside the SFP appears to be limited to 125 Mbits/sec. Something that is quite reasonable given the CPU power on-hand.

Thermals are worth paying attention to, in around 10 mins the chassis got to around 40C:

I later observed chassis temps of around 55-60C, enough to be an issue if you want to touch it without gloves

IPv6 support does work, however you will need to disable some sysctls that ship by default to disable all IPv6:

root@smart-sfp:/etc/sysctl.d# cat 98-disable-ipv6.conf 
# Disable IPv6 to prevent
# excess traffic while tests are running.

net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.default.autoconf = 0

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.all.autoconf = 0

So now that we casually have a sort of ethernet parasite device, what can we do with it?

BGP Route Server

A lot of smaller internet exchanges could find it useful to have a route server attached physically to a switch if they are rack space constrained.

Bird 1.6 and 2 installs and runs just fine on the SFP. With 512M of ram you would be able to hold a modest number of routes (you are unlikely to be able to fit a full table of 1M+ routes in 512M however).

DHCP/NTP/TFTP Server for featureless switches

We are spoilt in 2022 where a lot of switches are now hybrid layer 3 routers and have handy features.

With that in mind, a lot of older and cheap switches still do not have these features, but this can all easily be reimplemented using one of these SFP’s. Including just having a full PXE boot setup running from the SFP itself. Handy in a pinch or with remote sites to take over things like DHCP, DNS relay, or local NTP.

OOB for management switches

A long time ago I was looking into hacking cheaper layer 2 switches to have a out of band VPN appliance and management switch. Since it proved to be possible but not fully practical, this smart SFP offers us a new take on how to do it.

The kernel on my smart SFP is too old for mainline kernel wireguard (4.14.22-stcmtk-0.2.33), and the kernel does not come with headers making DKMS not possible.

There might be a newer version of the kernel and OS distribution. Software upgrades can be done either with a special script that writes into the boot partition, and then a factory reset with the micro–DIP switches, or using a special tool that uses the extra pins on the back of the SFP:

So just like in the original ethernet switch hacking post. We are back to tun/tap interfaces and the golang version of wireguard.

root@test-debian11:~# iperf3 -R -c 10.66.66.2
Connecting to host 10.66.66.2, port 5201
Reverse mode, remote host 10.66.66.2 is sending
[  5] local 10.66.66.1 port 35332 connected to 10.66.66.2 port 5201
...                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.21  sec  14.2 MBytes  11.6 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  12.2 MBytes  10.2 Mbits/sec                  receiver

iperf Done.
root@test-debian11:~# iperf3 -c 10.66.66.2
Connecting to host 10.66.66.2, port 5201
[  5] local 10.66.66.1 port 35336 connected to 10.66.66.2 port 5201
...     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.11 MBytes  3.45 Mbits/sec  1062             sender
[  5]   0.00-10.00  sec  4.07 MBytes  3.41 Mbits/sec                  receiver

iperf Done.

As we can see, you are going to win no awards for speed here. But it could still easily be competitive with a lot of console/out-of-band solutions out there!

Flexing the FPGA to filter and hijack

The smart-sfp can be configured to capture or hijack traffic in a bunch of ways. The most “classic” (in my eyes) is a “CPU Filter” that looks at two TCP/UDP destination ports (of your choosing), and if packets with those ports traverse the optic, they will also be copied to the internal Linux machine. This has a notable downside in that you don’t capture both sides of flows since the check is only done on the destination:

This makes it useful for monitoring / hijacking DNS, or trying to reset TCP connections that you don’t want. But not for full flow monitoring/hijacking.

There is also the “Loan IP” feature that allows you to hijack some of an IP address’s ports when traffic for it flows through the optical side of the SFP. This feature is actually designed so that the smart-sfp can live passively off the switches management IP, rather than needing it’s own IP address for SSH etc. But such a feature could also be used to create a fake 169.254.169.254 (AWS/Cloud metadata IP address endpoint) and serve requests from it. I’ve personally struggled to get this to work correctly and since failure is punished with a full factory reset, therefore I decided it was not worth pursuing too much!

Finally. In a more premium software package for the smart-sfp you can configure ERSPAN sessions with filters. This can be used to selectively mirror traffic to a remote destination!

smart-sfp(root)(config-erspan[profile1])# show filter 
Number  Dir  BSO  Protocol  IP src  Port src  IP dst  Port dst  VLAN id  Erspan   Cap       Enabled
0       rx   off  off       off     off       off     off       off      erspan0  capture0  false  
1       rx   off  off       off     off       off     off       off      erspan0  capture0  false  

When a packet going across the optic matches one of the defined filters, the SFP will wrap it in a GRE packet and send it onwards to somewhere else:

At the end of the day, I’m not sure what this optic is going to end up being used for. Since I have a selection of machines to run tasks on in my home network it’s not got a huge amount of use here.

I was wondering if it was practical to use on an internet exchange as a tiny node for anycast DNS or similar… but I don’t really have any good options for internet exchanges to test that with. It could not really be used in my existing IP network since the forwarding performance inside the CPU is too low to tunnel data back somewhere.

So, while this is an incredibly interesting bit of hardware, it’s still waiting for its day in the field.

---

If you want to stay up to date with the blog you can use the RSS feed or you can follow me on Twitter

Until next time!