Jan 7 2025

Building Ultra Long Range TOSLINK

This post is a textual version of a talk I gave at The 38th Chaos Computer Congress at the end of 2024. You can watch the talk that was recorded by the wonderful C3VOC team below if that’s your preferred medium:

Or watch using the C3VOC/media.ccc.de player

---

I’m a sucker for wanting to ultimately know how and why things work. My rule of thumb is most things are either really smart (in complexity, or simplicity), or incredibly stupid. However In my world optical data transmission is everywhere, and yet I had no idea how it really worked beyond what I can see from the command line interface.

Optics are a weird world

Optical data transmission is now everywhere in the datacenter, and my own stuff for bgp.tools is no exception. All connectivity in my world now comes out of Small Form-factor Pluggable (or, SFP) ports.

The cool thing about SFP’s is that they are for the most part (when they don’t contain a FPGA and ARM CPU to run Linux) very basic inside.

They are basically a set of laser drivers, and typically a small EEPROM+Microcontroller to tell the switch/router what it is and optionally some critical metrics like incoming/outgoing laser power.

At least 95% of all SFPs are made in the same set of factories under ODM agreements, so the majority difference between SFP vendors are the contents of their EEPROMs and the pricing markup the vendor charges for them.

SFP’s operate on a single “lane” of data, this means you can go from 100M/1G/10G and 25G. If you want to go faster, then we have to upgrade to optics with more than one lane. For this case, will look at the Quad Small Form-factor Pluggable (QSFP) form factor:

These basically build on the same ideas as the SFP, but adding 4 (or 8 in the case of the Double Density (DD)) lanes of 10/25/50/100G

These optics can be simple, just 4 “beams” of laser light that make up a higher speed link, or in the case of 100G coherent optics, quite complex, where there is a chip that merges 4 lanes of data into a single, higher speed one with a special encoding for longer range transmission. This typically involves an advanced processing chip inside each optic, typically using 7nm - 5nm processes to lower power consumptions and heat.

All of this is done so that these optical modules are swappable with various types, Sometimes you want cheaper short range (multimode) optics, Sometimes you want to want to only use a single glass core (BiDirectional), sometimes you need extended ranges, with pluggable optics you can pick and choose, the switch/router does not need to care (too much).

TOSLINK

On the other end of optics is TOSLINK (“Toshiba Link”), TOSLINK is typically S/PDIF over optical communication. This is present on most HiFi systems and some sound cards. It is typically useful to avoid ground loop problems in sound systems.

So where you have a audio signal, like this that can suffer from interference

TOSLINK/SPDIF turns this into a manchester coded serial signal, at around 1.5Mbps that is much more resiliant to analog interference

The optics involved here are a far cry from the modern telecommunication stuff, TOSLINK use a comparatively huge 1mm plastic core cable (where single mode fiber is 9 micron of glass!)

Because TOSLINK is not intended to go very far, LEDs are typically used rather than lasers. That combined with the multimodal properties from the 1mm core puts the typical maximum TOSLINK cable length at 10 meters.

This made me think…

Idea time

So we already know that SFPs are not doing much thinking

So what if we could use SFPs to send TOSLINK data? Could we extend (for no good reason of course) the range of TOSLINK from 10m to 10 km or more??

To see if we can do this, we are going to have to learn what is on the back of the double sided SFP connector:

I’ve scrubbed out the pins that are not useful for us, as they are mostly related to the microcontroller/EEPROM interface on the optic, since we don’t really need to interact with that interface in order to send stuff.

You might notice that sending (and receiving) a signal involves two pins on the connector, this is because the data interface between a SFP and its host uses differential signaling to reduce cross talk and other undesired issues at high speeds.

For this reason we cannot directly pump a TOSLINK signal directly into an optic, We need to send this through a LVDS Driver and Receiver (on the other end), thankfully the lovely people at Open source mobile communications (osmocom) made the “Sfp-experimenter” board that incorporates all of this stuff for us into a nice handy package, and my friend Sam W made a couple of them for me!

With the SFP interface sorted out, I bought some cheap TOSLINK analog-to-digital converter (ADC) and digital-to-analog converters (DAC) to make testing easy, and hijacked their optical output signals to plumb into the overall contraption, to form something that looks like this on a high level:

In reality the proof of concept looks like this, a very not-TSA-friendly bundle of messy wires and optical cable.

Suspicious looks aside, this proof of concept worked! I was sending audio in from my laptop, and via the ADC, into the optic, and the return signal from the optic (as the cable was placed in a loop) was going to the DAC, and finally back into my headphones with no audible issues!

On the scope, we can see the slight latency introduced by the S/PDIF signal going in and coming out of the SFP at around 34 nanoseconds, this would imply a rough optical distance of 6 meters, given the cable itself was easily 5 meters, I would call that pretty good going!

But can we go further?

Ok, 5 meters is not breaking any TOSLINK records (the plastic core stuff can even do 5 meters). So we now need to find some reasonably long haul piece of fiber that we can then use to send TOSLINK beyond its intended distance.

This came in the form of a friend who has space in two datacenters near each other in the London Docklands area. One being Telehouse North and the other being IP House. The rough walking distance between the two of them is about 650 meters.

Between the two locations is a fiber pair and a CWDM multiplexer, These boxes allow you to place multiple links (each link using different “colors” or “channels”) on a single fiber pair, maximizing the usage of an expensive inter-building (or sometimes inner-building in some places!) fiber pair.

James Rice at Jump Networks generously offered me a channel on their multiplex between IPH and Telehouse. To make testing easy (so that I only need to be in one location at a time) I installed a fiber loop on the telehouse end of the multiplexer, and did the testing in IP House

After much messing around in a particularly loud environment, I managed to get audio being sent up to telehouse, and looped back into IP House through my contraption and for sound to come out of my portable speaker!

On the scope we can see that there is a delay in the signal that we are sending vs receiving of around 11 microseconds

If we plug 11 microseconds and the speed of light in glass into WolframAlpha we get a distance of roughly 2.2 kilometres. This would imply that the real fiber distance end to end is 1.1 km vs the 0.6km walking distance. This is to be expected as there is significant patching work and transport around the building for the fiber cable that would easily add this distance to the final loop.

But this is fantastic news! This means that we have already exceeded the typical maximum TOSLINK distance by around 219 times!

However this still feels like it’s too short of a distance…

But can we go even further?

What if we were to get silly? How far can we reasonably go until we hit practical problems?

Typically (at least in the UK) a lot of long-haul data transmission for small to medium networks is done using rented “wave” capacity, waves are effectively a product in which you offload the need to own fiber (and at least in the UK, mitigate some of the tax burden ) and multiplexing equipment onto somebody else who then takes your optical signal and multiplexes some way to its destination.

The equipment that powers these service offerings are typically built out of a collection of DWDM multiplexers (a more dense version of the previously discussed CWDM), some optical routing systems (RODAMs), and most annoyingly for us a “muxponder”.

Optical systems often incorporate a transponder or muxponder for longer distance transmission, A transponder is a relatively simple device that effectively copy and paste one signal from an optic to another, the idea being is that your customer can communicate to you on a type of optic that is convenient for them (for 10G LR at the common 1310nm) and you can use a transponder to convert that into the required “color”/channel needed to fit into your DWDM mux (and also to match the transmission power, we will get onto that later).

A muxponder however is a lot smarter, and will “glue together” multiple lower speed signals like a set of 8 10G inputs and digitally multiplex them into a single higher speed 100G signal that will only consume a single channel on a DWDM multiplexer, this allows the provider to maximize the usage of a single channel.

The problem is that muxponders need to understand the signal coming into it and are limited to conventional protocols like Ethernet/FiberChannel/Infiniband. Our strange and comparatively low speed TOSLINK signal is not going to be understood by muxponder.

So the goal here is to find a channel on a long distance optical fiber path that someone will let me use and that does not involve a muxponder.

DWDM TOSLINK

Thankfully, VeloxServ (a company I already have colocation with) was willing to lend me a DWDM channel between Telehouse and Interxion LON1, two reasonably far away from each other DC sites in London:

After rigging up a very similar setup as last time (with a loop in Telehouse), I simply just needed to insert a DWDM SFP with the correct channel into my contraption:

And it worked! The round trip latency came out to 84 microseconds or about 16.8km!

All things considered, 16.8km is a nice improvement on 10 meters!

But can we go further?

Inter-City TOSLINK

Luckily LONAP offered a DWDM channel on their mux that went from a building in London Docklands, to Equinix LD6 in Slough, This means we would be sending a signal outside of London (and back)! Inter-city TOSLINK! (Technically not inter-city as Slough is not a city but a town. Cities in the UK are weird and to be a city requires royal involvements and blah blah blah)

Here is roughly our path on the map:

The optical path is around 70 km, and the multiplexer they use (a SmartOptics DCP-M40-C-ZR+) has built in EDFA amplification to achieve this range. The similar setup to last time is follows:

A loop was installed in Slough, however this time the loop also had some optical attenuation.

Because EDFA amplifiers are involved, care must be taken to ensure that every channel going into the amplifier has roughly the same laser light strength, otherwise the EDFA amplifier will not amplify the weaker signals correctly. Because the mux amplifies on the way out of the unit as well as input, without the attenuation my TOSLINK loop would knock out the other in-use channels on this segment (as the mux output signal is far brighter than what their optics are putting out)!

Drumroll

It didn’t work.

After building a test setup using a multimode optic to confirm everything was setup correctly, I then swapped in the DWDM optic (A FS.COM DWDM-SFP10G-80), and found the setup stopped working. After confirming with a USB gigabit SFP NIC that the loop was functional, I concluded that the DWDM optic could not be driven at the lower speeds. But how low could you drive this one?

If I replaced the TOSLINK device with the scope’s signal generator, I discovered some weird looking behaviour if I did a sweep of square wave frequencies:

---

It seems that the optic starts to “work” at 3Mhz, but the signal is clearly broken and not very square. While the signal we are putting in is very much not what this optic is designed for (10G XGMII Ethernet), this behaviour is quite surprising, as the square wave only becomes “sane” at around 6.5Mhz:

6.5Mhz is quite far away from the 2Mhz we generate from TOSLINK, and this was the only optic I had on hand for the channel that had the loop installed and since the other site is quite far away, I called the trip bust, packed up and went home empty handed.

But why, why did this 10G optic do this?

It turns out that finding data about the chips inside these optics is very hard. I opened a few broken/spare 10G DWDM optics and found that the internal chips have had their markings etched off, and on another optic vendor I could not find any signs of the chip existing, let alone a data sheet for it.

Since the shorter range optics that worked previously “only” had a minimum data rate of around 150Khz, I assumed there was some kind of signal processing going on inside the optic for DWDM or modules rated for longer distances, I figured that I could just buy a optic that was specified for lower speeds, but since the cost of these optics is quite high I wanted to make sure that would work before investing more money into this project. So I sent emails to a bunch of friendly looking vendors to see if I could get answers.

Thankfully I got a great response from one of them (FLEXOPTIX, Thank you to Gert & Thomas), that revealed

Another piece of Information that might be helpful for you: Make sure that the transceiver you want to use does not contain a clock and data recovery IC, sometimes also referred to as an eyeopener or retimer. Those only can handle specific data rates and even the multirate ones likely won’t be capable of dealing with your application. The good news is, that most of the low data rate transceivers don’t have those chips, they rely only on limiting amplifiers for the RX path and Laser diode drivers for the TX path. These can handle non standard data rates. The long distance Transceivers for multi gigabit applications are more likely to have those ICs and should thus be avoided.

This makes sense actually, since at long distances the signal gets “ripped apart” by dispersion in the fiber, For example, here is a particularly worse case simulation of the impact of this happeing over 140km:

So for higher speeds it makes sense to include a retimer to present a cleaner signal to the receiving end.

With all of that in mind, I found a cheap-ish 1G optic for sale that after emailing the vendor claimed to not include any CDR/Retiming chip. However when it arrived, things instantly did not look good after opening the box:

Since I would prefer to avoid another failed data center trip, I did some investigation and found that searching for HLSPDW-XE09 brings up a lot of 10G DWDM optics from other vendors. After testing the optic with the scopes signal generator, I found that the optic behaves nearly exactly the same as the previous one, So I can conclude that this optic is almost certainly a 10G DWDM optic with its EEPROM reprogrammed to be “1G”

I returned the optic to the vendor, as we had an email discussion before ordering the optic that “confirmed” that this optic didn’t have anything like a CDR/Retimer/etc, and to their credit they processed the refund with no problem.

I assume that in reality the production of 1G (and slower) DWDM optics ended a long time ago, and since the retimer in this actually-10G is unlikely to cause any issues with a 1G signal, this is deemed an acceptable solution for almost all use cases (just not my insane one!)

140 km TOSLINK, round 2

I asked around in industry channels if people had 1G or slower SFP DWDM optics they were willing to depart with and I ended up borrowing a handful, though sadly their optics were on different channels meaning I would have to go back to Slough to change some physical cabling.

Armed with an old 1G DWDM optic on a different channel (Thank you Brandon Butterworth!), I assembled my test setup to confirm everything worked. Then switched to the DWDM optic and heard the wonderful sounds of music going loop in and out of a city:

Looking towards the scope, it was basically going to be impossible to use previous methods of manually finding the time it takes for a bit sequence to appear on the other end, so instead I configured a trigger on the scope and “tapped” TX side and observed the time it took for the signal to show up on the receive side (since it’s more obvious this way)

This method suggests a 720µs round trip time around the fiber, since that roughly calculates to ~143.2km, this aligns with the expected distance of the circuit! We did it!

Given that the specified maximum distance for a TOSLINK cable is approximately 10 meters, this contraption has given us a 14600x range improvement over the standard setup!

Future possible “innovation”

If the SFP optic didn’t have some of the driving functionality in it, it would be possible to drive the laser directly, send “raw” audio in its full glory (and probably damage the laser).

This is useful if digital audio is not your thing and is sort of how RF Over Glass already works. A friend suggested it is also probably possible to send a significantly high frequency FM signal into a unmodified optic and have something that would resemble music come out of the other side (I have not tested this myself)!

It is tempting to attach a “dialup” modem to both sides, this would probably create the greatest modern day waste of a 100 GHz optical channel, given that it gives a final output bandwidth of ~40 kbit/s, and I assume this would probably confuse an intelligence agency if they were tapping the line.

To conclude

Yes, you can send “low” speed signals over SFP optics! Most optics start working at 150 KHz, but the retimers/CDR chips don’t like working below 6 MHz for most things. But if you can avoid them, you can send pretty arbitrary signals very, very far!

There is not really a good “production” reason to do any of this, as most use cases have better IP based solutions in 2025. But knowing how things work is important! Knowing this stuff means that it is possible to build bigger, better, more horrifying solutions/workarounds to problems, and at the very least I know far more about optical transmission, and general inner optics workings now. Maybe you also do as well!

---

If you want to stay up to date with the blog you can use the RSS feed or you can follow me on Fediverse @benjojo@benjojo.co.uk

Until next time!