Storage bus speeds with example storage endpoints.
| Bus | Lanes | End-Point | Theoretical Bandwidth (MB/s) | Note |
| SAS-3 | 1 | HBA <-> Single SATA Drive | 600 | SAS3<->SATA 6Gbit |
| SAS-3 | 1 | HBA <-> Single SAS Drive | 1200 | SAS3<->SAS3 12Gbit |
| SAS-3 | 4 | HBA <-> SAS/SATA Fanout | 4800 | 4 Lane HBA to Breakout (6 SSD)[2] |
| SAS-3 | 8 | HBA <-> SAS/SATA Fanout | 8400 | 8 Lane HBA to Breakout (12 SSD)[1] |
| PCIe-3 | 1 | N/A | 1000 | Single Lane PCIe3 |
| PCIe-3 | 4 | PCIe <-> SAS HBA or NVMe | 4000 | Enough for Single NVMe |
| PCIe-3 | 8 | PICe <-> SAS HBA or NVMe | 8000 | Enough for SAS-3 4 Lanes |
| PCIe-3 | 40 | PCIe Bus <-> Processor Socket | 40000 | Xeon Direct conect to PCIe Bus |
Notes
All figures here are the theoretical maximums for the busses using rough/easy calculations for bits/s<->bytes/s. Enough to figure out where the throughput bottlenecks are likely to be in a storage system.
- SATA devices contain a single SAS/SATA port (connection), and even when they are connected to a SAS3 HBA, the SATA protocol limits each SSD device to ~600MB/s (single port, 6Gbit)
- SAS devices may be dual ported (two connections to the device from the HBA(s)) – each with a 12Gbit connection giving a potential bandwidth of 2x12Gbit == 2.4Gbyte/s (roughly) per SSD device.
- An NVMe device directly attached to the PCIe bus has access to a bandwidth of 4GB/s by using 4 PCIe lanes – or 8GB/s using 8 PCIe lanes. On current Xeon processors, a single socket attaches to 40 PCIe lanes directly (see diagram below) for a total bandwidth of 40GB/s per socket.
- https://en.wikipedia.org/wiki/Serial_ATA#SATA_3_Gbit/s_and_SATA_6_Gbit/s
- https://www.nextplatform.com/2017/07/14/system-bottleneck-shifts-pci-express/
- https://www.hardwarezone.com.sg/feature-understanding-intel-x99-high-performance-computing-platform
- I first started down the road of finally coming to grips with all the different busses and lane types after reading this excellent LSI paper. I omitted the SAS-2 figures from this article since modern systems use SAS-3 exclusively.
[pdf-embedder url=”https://www.n0derunner.com/wp-content/uploads/2018/04/LSI-SAS-PCI-Bottlenecks.pdf” title=”LSI SAS PCI Bottlenecks”]
Intel Processor & PCI connections
