25GBit/s on macOS & iOS

Extreme network speeds on macOS/iOS devices using ConnectX network cards via USB4/Thunderbolt

By Manawyrm, tSYS and Ronja | Friday, May 17, 2024

macOS speed test

iOS speed test

Background info

Modern versions of macOS and iOS (starting with macOS Ventura/13) ship with NVIDIA/Mellanox ConnectX network drivers. AppleEthernetMLX5 is included by default and can be used with any NVIDIA/Mellanox ConnectX-4, 5 and 6 network card:

ConnectX-4 cards provide dual port 25 GBit/s connectivity (SFP28) and newer cards can go up to 200 GBit/s.
The maximum throughput is limited by the Thunderbolt/USB4 interface (to around 40 Gbit/s).

Photo of a Mac screen, showing Speedtest.net app with 20467 MBit/s download, 5670 MBit/s upload

Warning:

ConnectX-2 and ConnectX-3 cards are NOT supported by the driver. macOS/iOS do not provide drivers for those cards.

We were even successful in connecting a ConnectX-4Lx NIC to an iPad Pro (M1):

iPad, connected via Thunderbolt to a ConnectX-4Lx network card, with SFP28 DAC, showing 10322/9359 MBit/s on the Speedtest.net app

Note:

Not many iPads support Thunderbolt/USB4.
At the time of writing, only the iPad Pro models with Apple Silicon/M-series processors support it.

Hardware setup

Thunderbolt/USB4 NVMe storage enclosures can be (ab)used as an external PCIe enclosure using a M.2 to PCIE x4 adapter. These enclosures are typically used to connect fast NVMe SSDs to a computer, but the used Thunderbolt 3 controller ICs (like Intels JHL7440) will just accept Thunderbolt and turn it into 4 PCIe lanes. These lanes can then be connected to any PCIe device.

Our tests were done using a TBU401 enclosure (~100$):
ConnectX-6 network card in a PCIe->M.2 adapter in a TBU401 Thunderbolt enclosure TBU401 internal PCB, JHL7440 IC, RTL9210B IC, ASM1480 IC, many other random storage and power ICs

Warning:

Cheap NVMe USB storage enclosures (~30$) will not work.
These convert USB mass storage to NVMe, but do not provide PCIe tunneling / actual NVMe connectivity.

ASMedia might provide a cheaper option with their ASM4242 chipset, but we weren’t able to test such an enclosure yet.

Several vendors (on AliExpress, etc.) also offer little PCIe docks, like this “TH3P4 Lite GPU Dock”:
External GPU adapter with ConnectX-6 NIC

Full eGPU-enclosures (typically used for graphics cards) will also work well (but are large and very expensive).

Power supply

PCIe slots provide 2 voltage rails: 12V and 3.3V.
USB-C (without PD) provides 5V.

NVMe enclosures will convert the 5V supply down to 3.3V, but the 12V supply is still missing (and required for ConnectX cards).

Maximum available power on a USB-C port: 15W (5V/3A).

ConnectX-4 card: ~11W
Thunderbolt chipset: ~3W
Conversion losses (5V->12V step-up): ~2W

The total power of this setup will exceed the maximum available power on a USB-C port.

Powering such a setup without using an external power brick won’t be easily possible.
We used an external 12V/2.5A power supply for our tests.

GPUs on a Mac?

This same setup could also be used to connect an external GPU to a computer.
This will work well on Intel/x86 Windows and Linux machines.

There is no hope of GPU support on (current-gen) Apple Silicon, though. Apart from the fact that no drivers are available on macOS, even projects like Asahi Linux run into a hardware limitation with the way memory mapping works on M-series chips.

This is the same limitation also present on other Arm platforms like the Raspberry Pi. See PCIe problems on embedded systems.