IP-over-Toslink

Backstory

At the 38th Chaos Communication Congress (38C3) benjojo gave a very interesting talk titled:
“Going Long! Sending weird signals over long haul optical networks”

Coincidentally, ember posted this joke to the Fediverse:

(Translation: “Overheard: We’re doing IP-over-Toslink now”)

After thinking about it for a while, this idea didn’t seem very far-fetched.
Toslink is a fully digital, bit-perfect way to transport digital PCM audio – aka data. Why not transport IP packets instead?
We could even go full-circle and transport IP-over-Toslink over single-mode fiber!

Implementation

Toslink is really just S/PDIF (aka IEC 958) transmitted over light instead of coaxial cable.
As such, it typically carries 48kHz of uncompressed PCM audio at 16bits in 2 channels:
48000 Hz * 16 bits * 2 channels = 1 536 000 bits/second

1536 kbit/s sounds like a usable connection speed after all…
Fun fact: That’s about the same throughput as a T1 line (1544 kbit/s).

In audio setups, Toslink/SPDIF is sometimes used for digital data as well, to transport digital (often multi-channel) compressed audio formats like AC-3 or DTS.

We need a way to create a continuous bitstream of 1.536 MBit/s and some way to signal framing over a serial connection.

Luckily for us, this problem was solved in the past:
Back in the days of ISDN, HDLC framing was used to transport packets over a synchronous serial connection.
It was also common to use the Point-to-Point Protocol (PPP) over such a link to exchange link parameters, handle compression and do authentication-related tasks.

About 2 years ago, I had already written some code that starts & configures a PPP daemon on Linux, takes the asynchronous framing of the packets and encodes them into synchronous HDLC.
This was released as the yate-ras project, an open source ISDN/synchronous network access server.
yate-ras can be used on a virtual telephony exchange (PBX) to handle & terminate incoming dial-up/ISDN phone calls and provide internet access via them.

This software could be quickly modified to accept data on STDIN and STDOUT.
It was then just a matter of building a pipe between arecord, the hdlc_ppp encap tool and aplay:

arecord -f dat -t raw -D hw:CARD=ICUSBAUDIO7D,DEV=0 --buffer-size 1024 | \
src/yate_hdlc_ppp options.server | \
aplay -f dat -t raw -D iec958:CARD=ICUSBAUDIO7D,DEV=0 --buffer-size 1024

This setup not only has about the same bandwidth as a T1 line, it’s also using the exact same bitstream/protocol on the wire!

arecord & aplay are both standard utilities from ALSA (the Linux audio stack).

-f dat configures 48kHz, 16bit little endian audio.
-t raw disables generation of any .wav headers and just outputs raw interleaved PCM frames.
--buffer-size 1024 sets the internal buffer size to 1024µs, way quicker than the default maximum, reducing latency.

Both client & server side look identical, just with different pppd configuration files.

Just like in yate-ras, sqm_cake is used as a congestion control algorithm to avoid buffer bloat:

#!/bin/bash
# /etc/ppp/ip-up.d/cake
tc qdisc add dev $1 root cake bandwidth 1536kbit

NAT & MSS clamping is configured using:

sysctl -w net.ipv4.ip_forward=1
iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 500
iptables -t nat -A POSTROUTING -o wlxdc15c89bb228 -j MASQUERADE

Clamping the MSS to about 500 bytes reduces worst-case latency on low-throughput links (at the cost of max. throughput).

Hardware setup

I bought 2 of the cheapest (sub-10$) USB sound cards with a Toslink in- & output.

Those typically use the C-Media CM6206 chipset, which is a fully integrated, single-chip USB sound card.
Just add an external crystal oscillator, a 3.3V regulator and some capacitors and you’re done:

The resulting setup between 2 laptops looked like this:

The ThinkPad T14 (on the right) shares its internet connection (via ip_forwarding/MASQUERADE NAT) via the PPP link.
The ThinkPad T430 (on the left) is only connected via the Toslink networking connection.

Benchmarks

manawyrm@nodeA:~$ ping 172.21.118.5
PING 172.21.118.5 (172.21.118.5) 56(84) bytes of data.
64 bytes from 172.21.118.5: icmp_seq=1 ttl=64 time=59.0 ms
64 bytes from 172.21.118.5: icmp_seq=2 ttl=64 time=54.7 ms
64 bytes from 172.21.118.5: icmp_seq=3 ttl=64 time=56.4 ms
64 bytes from 172.21.118.5: icmp_seq=4 ttl=64 time=57.9 ms
64 bytes from 172.21.118.5: icmp_seq=5 ttl=64 time=53.9 ms
64 bytes from 172.21.118.5: icmp_seq=6 ttl=64 time=55.4 ms
^C
--- 172.21.118.5 ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5006ms
rtt min/avg/max/mdev = 53.899/56.217/59.007/1.787 ms

manawyrm@nodeB:~$ iperf3 -s
Accepted connection from 172.21.118.5, port 38226
[  5] local 172.21.118.1 port 5201 connected to 172.21.118.5 port 38240
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   168 KBytes  1.38 Mbits/sec
[  5]   1.00-2.00   sec   181 KBytes  1.48 Mbits/sec
[  5]   2.00-3.00   sec   180 KBytes  1.47 Mbits/sec
[  5]   3.00-4.00   sec   181 KBytes  1.48 Mbits/sec
[  5]   4.00-5.00   sec   180 KBytes  1.47 Mbits/sec
[  5]   5.00-6.00   sec   181 KBytes  1.48 Mbits/sec

As you can see, we’re reaching a real-world TCP throughput of around 1.47 Mbit/s, which is about 95% of the theoretical limit :) The latency (while not great) is also totally usable. If you really wanted to use this (don’t!), it would be possible to use libusb to interface directly with the CM6202 chip from userspace, which will probably decrease latency even further.

If you’re wondering what HDLC/PPP/IP packets sound like when played back as PCM audio: short bursts of static/noise.

Full PPP configuration dump

Server (“nodeA”)

mtu 1500
ms-dns 8.8.8.8
ms-dns 8.8.4.4

debug
nodetach
noauth
nodeflate

172.21.118.1:172.21.118.5

Client (“nodeB”)

mtu 1500

debug
nodetach
noauth
nodeflate

usepeerdns
defaultroute

• ←Previous
• Next→