Storage bus speeds with example storage endpoints.
|Bus||Lanes||End-Point||Theoretical Bandwidth (MB/s)||Note|
|SAS-3||1||HBA <-> Single SATA Drive||600||SAS3<->SATA 6Gbit|
|SAS-3||1||HBA <-> Single SAS Drive||1200||SAS3<->SAS3 12Gbit|
|SAS-3||4||HBA <-> SAS/SATA Fanout||4800||4 Lane HBA to Breakout (6 SSD)|
|SAS-3||8||HBA <-> SAS/SATA Fanout||8400||8 Lane HBA to Breakout (12 SSD)|
|PCIe-3||1||N/A||1000||Single Lane PCIe3|
|PCIe-3||4||PCIe <-> SAS HBA or NVMe||4000||Enough for Single NVMe|
|PCIe-3||8||PICe <-> SAS HBA or NVMe||8000||Enough for SAS-3 4 Lanes|
|PCIe-3||40||PCIe Bus <-> Processor Socket||40000||Xeon Direct conect to PCIe Bus|
All figures here are the theoretical maximums for the busses using rough/easy calculations for bits/s<->bytes/s. Enough to figure out where the throughput bottlenecks are likely to be in a storage system.
- SATA devices contain a single SAS/SATA port (connection), and even when they are connected to a SAS3 HBA, the SATA protocol limits each SSD device to ~600MB/s (single port, 6Gbit)
- SAS devices may be dual ported (two connections to the device from the HBA(s)) – each with a 12Gbit connection giving a potential bandwidth of 2x12Gbit == 2.4Gbyte/s (roughly) per SSD device.
- An NVMe device directly attached to the PCIe bus has access to a bandwidth of 4GB/s by using 4 PCIe lanes – or 8GB/s using 8 PCIe lanes. On current Xeon processors, a single socket attaches to 40 PCIe lanes directly (see diagram below) for a total bandwidth of 40GB/s per socket.
- I first started down the road of finally coming to grips with all the different busses and lane types after reading this excellent LSI paper. I omitted the SAS-2 figures from this article since modern systems use SAS-3 exclusively.
[pdf-embedder url=”https://www.n0derunner.com/wp-content/uploads/2018/04/LSI-SAS-PCI-Bottlenecks.pdf” title=”LSI SAS PCI Bottlenecks”]
Intel Processor & PCI connections