This story, like another in the past, started as an eBay purchase that I would soon regret. I was scrolling through my favorite eBay supplier when I found a listing titled
Job Lot Of Roughly 350 Various Network Cards. This of course piqued my interest. Some close photo inspection identified a whole load of cards in the pile that were worth the price of the whole lot. So I naturally bought it.
What I had not mentally planned for was that “Roughly 350” is a lot of anything. This soon became apparent however due to a pallet truck turning up at my flat in London.
After dealing with the slightly embarrassing logistical challenge of getting a pallet up into my flat itself, I began to sort the cards in to their types, and found myself with a lot of two types, 4x 1G Intel i20 cards, and a lot of 8G Fibre Channel cards from QLogic (now Marvell) and Emulex (Now Broadcom). I had an unreasonable amount of these Fibre Channel PCIe cards.
Fibre Channel (FC) is used for connecting systems to remote storage efficiently over a fabric different to the common ethernet network. To make it slightly confusing, to the untrained eye they could be confused with 10G ethernet since both take SFP+ optical modules.
After doing some research, it does appear that buried in the history of FC, there was support for carrying IP traffic over FC presumably so that systems could be connected with only FC.
It seems that this functionality never actually really hit full support. IPoFC support used to exist in the QLogic driver and then got removed, here is a public attempt to restore this function to the QLogic driver with some degree of success, just without stability or modern device support.
Surely there is a way to get some use out of these cards? Since I had so many of them, I figured it would not hurt to try and bring new life into them.
So I grabbed two test machines, installed a single port QLogic card into both of them, and connected them up with some OM3 cable.
FC Host Bus Adapters (HBA) by default are looking for a target (typically a Storage Area Network (SAN)) to establish a session with. So while connecting two machines together will bring a link up, they will not be able to do anything with each other by default.
For this one side needs to be in target mode and provide storage. For this reason the SCST project has a driver for the QLogic series of FC cards that can make them act as a storage over FC target.
After compiling this module and getting it working, I looked into how to use SCST and found, to my delight, that there is a user space module. That module allows the creation of devices entirely in userspace, which can then be exposed to another system over FC.
root@testtop:~# rmmod qla2xxx # Unload the mainline driver root@testtop:~# modprobe qla2x00tgt # Load the target mode SCST version root@testtop:~# # Apply basic config to allow the other side use our "disk" root@testtop:~# scstadmin -config lol.conf Collecting current configuration: done. -> Checking configuration file 'lol.conf' for errors. -> Done -> Applying configuration. -> Setting target attribute 'rel_tgt_id' to value '1' for driver/target 'qla2x00t/50:01:43:80:26:68:8e:5c': done. -> Adding new group 'net' to driver/target 'qla2x00t/50:01:43:80:26:68:8e:5c': done. -> Adding new initiator '50:01:43:80:21:de:b5:d6' to driver/target/group 'qla2x00t/50:01:43:80:26:68:8e:5c/net': done. -> Adding new initiator '21:00:00:24:ff:0e:0e:75' to driver/target/group 'qla2x00t/50:01:43:80:26:68:8e:5c/net': done. -> Enabling driver/target 'qla2x00t/50:01:43:80:26:68:8e:5c': done. -> Done, 5 change(s) made. All done. root@testtop:~# ./scst-driver & # Start the driver root@testtop:~# # Attach the virtual disk to the other client. root@testtop:~# scstadmin -add_lun 0 -driver qla2x00t \ > -target 50:01:43:80:26:68:8e:5c \ > -group net -device net3 -attributes read_only=0 Collecting current configuration: done. -> Making requested changes. -> Adding device 'net3' at LUN 0 to driver/target/group 'qla2x00t/50:01:43:80:26:68:8e:5c/net': done. -> Done.
This is great because it allows me to create a virtual device using a just systems programming techniques. The downside is that, while there is documentation for this, the amount of examples of code that uses this in the wild is effectively one, and it’s the fileio example included in the SCST codebase. This code example, however, would prove invaluable as it’s high speed (can read up to 6gbps), and is clean enough to be used as a reference.
As I got deeper into proof of concepts I realised I could not simply send arbitrary packets back and forth over FC, meaning I would have to implement a SCSI compliant device for the purpose of getting network traffic back and forth.
Regardless, I got cracking on with it and decided to see if I could write a virtual device handler in Go. What I was not expecting was that this means you have to basically write a device nearly from the ground up. This meant I would have to get familiar with the protocol that disk drives speak: SCSI. The slightly annoying bit is that being a disk is actually surprisingly complex. I now understand why bugs in drive firmware exist, since the manuals that give a medium view into the SCSI command set are over 500 pages long.
One of the most fundamental SCSI commands that exists is INQUIRY, it describes the model and type of device attached. It’s also one of (apart from the SCSI version of a ping) the first commands linux will send to a SCSI device, and it will make classification decisions based on the output of the INQUIRY.
You can see the output of this early stage command by looking into your dmesg, and looking for something that looks like:
scsi 4:0:0:0: Direct-Access ATA KINGSTON SA400S3 71B1 PQ: 0 ANSI: 5
These are the results of the kernel issuing the INQUIRY command, and outputting what class type your device responded back with (In the above case a “Direct-access block device”). The device type provided will determine how the kernel handles you. Picking a block storage device type will result in the kernel automatically sending various information gathering SCSI commands that can be quite hard to implement correctly. Thankfully there are a range of other devices you can pick from.
|Value||Device type||Command set||Specification|
|00h||Direct-access block device (e.g. magnetic disk)||SBC Direct Access Commands||SCSI Block Commands (SBC)|
|01h||Sequential-access device (e.g. magnetic tape)||SSC Sequential Access Commands||SCSI Stream Commands (SSC)|
|02h||Printer device||SSC Printer Commands||SCSI Stream Commands (SSC)|
|03h||Processor device||SPC Processor Commands||SCSI Primary Commands (SPC)|
|04h||Write-once device||SBC Write Once Commands||SCSI Block Commands (SBC)|
|05h||CD/DVD-ROM device||MMC CD-ROM Commands||SCSI Multimedia Commands (MMC)|
|06h||Scanner device||SGC Scanner Commands||SCSI Graphics Commands (SGC)|
|07h||Optical memory device (e.g. some optical disks)||SBC Optical Media Commands||SCSI Block Commands (SBC)|
|08h||Medium changer (e.g. jukeboxes)||SMC Medium Changer Commands||SCSI Medium Changer Commands (SMC)|
|09h||Communications device||SSC Communications Commands||SCSI Stream Commands (SSC)|
|0Ah–0Bh||Defined by ASC IT8 (Graphic arts pre-press devices)||ASC IT8 Prepress Commands|
|0Ch||Storage array controller device (e.g. RAID)||SCC Array Controller Commands||SCSI Controller Commands (SCC)|
|0Dh||Enclosure services device||SES Enclosure Services Commands||SCSI Enclosure Services (SES)|
|0Eh||Simplified direct-access device (e.g. magnetic disk)||RBC Reduced Block Commands||Reduced Block Commands (RBC)|
|0Fh||Optical card reader/writer device||OCRW Optical Card Commands||SCSI Specification for Optical Card Reader/Writer (OCRW)|
|10h||Reserved for bridging expanders|
|11h||Object-based Storage Device||OSD Object-based Storage Commands||Object-based Storage Commands (OSD)|
|14h||Host managed zoned block device||Zoned Block Commands (ZBC)|
After some playing around, I found that identifying as a scanner resulted in the fewest questions being asked by the linux kernel. Just one or two INQUIRY’s and then a device would be made under
/dev/sg<n> for your use.
This resulted in a slightly amusing dmesg of:
scsi 6:0:0:0: Scanner BENJOJO Network Card lol 350 PQ: 0 ANSI: 6
Now that we have a minimal virtual SCSI device, we can get to work on getting it to move ethernet packets for us. Even if smartctl had some issues parsing it, it was “close enough”.
root@black-box:~# smartctl -a /dev/sg1 smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-9-amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: BENJOJO Product: Network Card lol Revision: 350 Compliance: SPC-4 User Capacity: 52,355,180,385,410,854 bytes [52.3 PB] Logical block size: 520093954 bytes scsiModePageOffset: response length too short, resp_len=7 offset=22 bd_len=18 scsiModePageOffset: response length too short, resp_len=7 offset=22 bd_len=18 >> Terminate command early due to bad response to IEC mode page A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
So I decided to build a basic interface that accepts SCSI writes as outbound packets, and allows the client to poll for incoming packets. Like in a previous project, I used TUN/TAP to make a virtual network card on both systems.
On the FC Client, we can use the generic SCSI interface in linux (sg) to manually fire READ and WRITE commands.
This leaves us with a basic polling mode that results in basic connectivity:
PING 184.108.40.206 (220.127.116.11) 56(84) bytes of data. 64 bytes from 18.104.22.168: icmp_seq=1 ttl=64 time=387 ms 64 bytes from 22.214.171.124: icmp_seq=2 ttl=64 time=194 ms 64 bytes from 126.96.36.199: icmp_seq=3 ttl=64 time=102 ms
But since it was polling the fake drive, it had high latency, and abysmally low throughput (less than 80kbits/s). We can quickly improve this by reducing the polling interval from the testing 100ms to 1ms.
This gave us a much better latency and throughput, we were nearly at 10mbit/s with just this.
However we are now at the limit for how fast we can reasonably poll for new packets. Now we need to ideally push packets to the other side when they are available. The easy way to do this is to hang a SCSI read request for a bit until there is a packet. However, due to the way SCST’s user space mode works ( only a single request can be processed at once ) we will have to find a smart way to somehow serve a long standing SCSI read while still being able to process writes… So I did the easy thing, and just made two SCSI devices.
root@black-box:~# dmesg | tail -n 5 [ 710.064439] qla2xxx [0000:01:00.0]-500a:6: LOOP UP detected (8 Gbps). [ 710.923684] scsi 6:0:0:0: Scanner BENJOJO Network Card lol 350 PQ: 0 ANSI: 6 [ 710.924371] scsi 6:0:0:0: Attached scsi generic sg1 type 6 [ 710.924979] scsi 6:0:0:1: Scanner BENJOJO Network Card lol 350 PQ: 0 ANSI: 6 [ 710.925502] scsi 6:0:0:1: Attached scsi generic sg2 type 6
One is designed for SCSI writes (to send packets to the other side), and the other for SCSI reads (to read packets from the other side).
This allows us to get low latency with full duplex communications:
root@black-box:~# ping 188.8.131.52 PING 184.108.40.206 (220.127.116.11) 56(84) bytes of data. 64 bytes from 18.104.22.168: icmp_seq=1 ttl=64 time=0.956 ms 64 bytes from 22.214.171.124: icmp_seq=2 ttl=64 time=1.11 ms 64 bytes from 126.96.36.199: icmp_seq=3 ttl=64 time=1.05 ms ^C --- 188.8.131.52 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 4ms rtt min/avg/max/mdev = 0.956/1.036/1.106/0.067 ms
With a throughput of around 100mbits:
root@black-box:~# iperf -c 184.108.40.206 ------------------------------------------------------------ Client connecting to 220.127.116.11, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 18.104.22.168 port 52766 connected with 22.214.171.124 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 105 MBytes 88.2 Mbits/sec
Or with jumbo frames (9000 MTU) enabled:
root@black-box:~# iperf -c 126.96.36.199 ------------------------------------------------------------ Client connecting to 188.8.131.52, TCP port 5001 TCP window size: 325 KByte (default) ------------------------------------------------------------ [ 3] local 184.108.40.206 port 52752 connected with 220.127.116.11 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 579 MBytes 485 Mbits/sec
At this point, while we have not reached anywhere near the technical limit of 6gbps, we have basically reached “good enough as an out of band network” levels of function. I sadly don’t have a FC switch to test if this can go over a fabric, but I don’t see any reason why it should not. But in a slightly more creative setup someone could use such a thing as a backup IP network for when systems go totally unreachable.
I suspect with some effort you could get this to be faster, however for me it’s personally not worth it. Much like the attempted IPoFC backport it might be impractical to get the throughput to be better than gigabit ethernet while also not consuming large amounts of CPU in the process, so the value of such a project is minimal. It’s also worth pointing out that I did look into modifying the qla2xx driver itself, but without decent documentation of the qlogic chipsets themselves it seemed to be reasonably difficult to modify it directly to be an ethernet card.
However, I did manage to make several performance jumps during the project with basic stuff like ensuring not to emit debug messages on the hot path, and slightly more obscure issues that I later on found involving the performance impacts of Go channels. I’m sure you could obtain faster links by chaining together many SCSI devices together, and then using the multi queue TUN/TAP feature to avoid blocking on a single TUN/TAP file descriptor.
The final full system flamegraph looked like this (click for interactive) half of CPU was used for jumbo frames 400mbit/s:
While the same systems with Intel X520 10 gigabit ethernet cards look like this (click for interactive) (CPU load was only a single core for a 10G flow):
Overall; not great, not terrible.
You can find this rough code over on my github, but it’s really best to be used as a data point for SCST’s user space module, since I cannot in good faith recommend people write virtual SCSI devices in a language like Go.
If you wish to see the internals, you can find them on my github: https://github.com/benjojo/IPoverFC
Since I mentioned at the start that I have a lot of these cards, if anyone is in the London area and is willing to take some of these off my hands, I will happily depart with them. I have roughly 2 tesco bags of Emulex FC 8Gbit cards and 1 tesco bag of QLogic. If that’s your thing feel free to email over at firstname.lastname@example.org!
If this was interesting to you, you may find other bits you like on the rest of the blog. If you want to stay up to date with my ramblings or projects you can use my blog’s RSS Feed or you can follow me on twitter.