Storage Bus Speeds 2018

Storage bus speeds with example storage endpoints.

Bus Lanes End-Point Theoretical Bandwidth (MB/s) Note
SAS-3 1 HBA <-> Single SATA Drive 600 SAS3<->SATA 6Gbit
SAS-3 1 HBA <-> Single SAS Drive 1200 SAS3<->SAS3 12Gbit
SAS-3 4 HBA <-> SAS/SATA Fanout 4800 4 Lane HBA to Breakout (6 SSD)[2]
SAS-3 8 HBA <-> SAS/SATA Fanout 8400 8 Lane HBA to Breakout (12 SSD)[1]
PCIe-3 1 N/A 1000 Single Lane PCIe3
PCIe-3 4 PCIe <-> SAS HBA or NVMe 4000 Enough for Single NVMe
PCIe-3 8 PICe <-> SAS HBA or NVMe 8000 Enough for SAS-3 4 Lanes
PCIe-3 40 PCIe Bus <-> Processor Socket 40000 Xeon Direct conect to PCIe Bus

 

Notes


All figures here are the theoretical maximums for the busses using rough/easy calculations for bits/s<->bytes/s. Enough to figure out where the throughput bottlenecks are likely to be in a storage system.

  1. SATA devices contain a single SAS/SATA port (connection), and even when they are connected to a SAS3 HBA, the SATA protocol limits each SSD device to ~600MB/s (single port, 6Gbit)
  2. SAS devices may be dual ported (two connections to the device from the HBA(s)) – each with a 12Gbit connection giving a potential bandwidth of 2x12Gbit == 2.4Gbyte/s (roughly) per SSD device.
  3. An NVMe device directly attached to the PCIe bus has access to a bandwidth of 4GB/s by using 4 PCIe lanes – or 8GB/s using 8 PCIe lanes.  On current Xeon processors, a single socket attaches to 40 PCIe lanes directly (see diagram below) for a total bandwidth of 40GB/s per socket.

  • I first started down the road of finally coming to grips with all the different busses and lane types after reading this excellent LSI paper.  I omitted the SAS-2 figures from this article since modern systems use SAS-3 exclusively.
LSI SAS PCI Bottlenecks

 

Intel Processor & PCI connections



Meltdown, speculative execution and side-channels explainer.

There are a lot of explanations for the current Meltdown/Spectre crisis but many did not do a good job of explaining the core issue if how information is leaked from the secret side,  to the attackers side.  This is my attempt to explain it (mostly to myself to make sure I got it right).

What is going on here generally?

  • Generally speaking an adversary would like to read pieces of memory he is not allowed to.  This can be either reading from kernel memory, or reading memory in the same address space that should not be allowed.  e.g. javascript from a random web page should not be able to read the passwords stored in your browser.
  • Users and the kernel are normally protected from bad-actors via privileged modes, address page tables and the MMU.
  • It turns out that code executed speculatively can read any mapped memory.  Even addresses/address that would not be readable in the normal program flow.
    • Thankfully illegal reads from speculatively executed code are not accessible to the attacker.
    • So, although the speculatively executed code reads an illegal value in micro-code, it is not visible to user-written code (e.g. the javascript in the browser)
  • However, it turns out that we can execute a LOT of code in speculative mode if the pre-conditions are right.
    • In fact modern instruction pipelines (and slow memory) allow >100 instructions to be executed while memory reads are resolved.

How does it work?

  • The attacker reads the illegal memory using speculative execution, then uses the values read – to set data in cache lines that ARE LEGITIMATELY VISIBLE to the attacker.   Thus creating a side channel between the speculatively executed code and the normal user written code.
  • The values in the cache lines are not readable (by user code) – but the fact that the cache lines were loaded (or not) *IS* detectable (via timing) since the L3 cache is shared across address-space.
    • First I ensure the cache lines I want to use in this process are empty.
    • Then I setup some code that reads an illegal value (using speculative execution technique), and depending on whether that value is 0 or !=0 I would read some other (specific address in the attackers address space) that I know will be cached in cache-line 1.  Pretend I execute the second read only if the illegal value is !=0
    • Finally back in normal user code I attempt to read that same address in my “real” user space. And if I get a quick response – I know that the illegal value was !=0, because the only way I get a quick response is if the cache line was loaded during the speculative execution phase.
    • It turns out we can encode an entire byte using this method.  See below.
  • The attacker reads a byte – then by using bit shifting etc. – the attacker encodes all 8 bits in 8 separate cache lines that can then be subsequently read.
  • At this point an attacker has read a memory address he was not allowed to, encoded that value in shared cache-lines and then tested the existence or not of values in the cache lines via timing, and thus re-constructs the value encoded in them during the speculative phase.
    • This is known as “leakage“.
  • Broadly there are two phases in this technique
    • The reading of illegal memory in speculative execution phase then encoding the byte in shared cache lines.
    • Using timing of reads to those same cache lines to determine if they were “set” (loaded e.g.”1″) or unset (empty “0”)  by the attacker to decode the byte from the (set/unset 1/0) cache lines.
  • Side channels have been a known phenomena for years (at least since the 1990s) what’s different now if how easy, and with such little error rate – attackers are able to read arbitrary memory addresses.

I found these papers to be informative and readable.

Simple statistics for performance analysts.

As performance analysts we often have to summarize large amounts of data in order to make engineering decisions or understand existing behavior.  This paper will help you do exactly that!  Many analysts know that using statistics can help, but statistical analysis is a huge field in itself and has its own complexity.  The article below distills the essential techniques that can help you with typical performance analysis tasks.

PDF Download.

Statistics for the performance analyst

Work around for bios.hddOrder when creating an OVF/OVA template.

When changing SCSI devices in an ESX based VM, it’s easy to screw up the ability to boot.  The simple fix is to add

bios.hddOrder = “scsi0:0”

to the end of the .vmx file.  This has always worked for me.  The problem with this solution is that any OVF/OVA that is created from the VM will not include the .vmx file hack, and of course VM’s created from the template will not boot until their .vmx file is hand edited.

The solution that worked for me was to simply make the “boot drive” the first .vmdk file that is listed in the .vmx file.  In my case, the Linux OS is stored on the VMDK named “disk.vmdk”

In the before case, this disk is listed last (even though it has SCSI ID 0:0:0) and the VM does not boot.

I simply change the filename from disk_6.vmdk to disk.vmdk (and change the last item from disk.vmdk to disk_6.vmdk).

The beauty of this method is that the ordering is maintained when creating an OVF/OVA.

When the VM boots, the /dev/sd devices may change since the vmdk’s are now attached to different SCSI devices – so mounting using UUID’s in Linux helps keep things sane.

Before

:floppy0.fileName = "Floppy 0"
ide1:0.startConnected = "FALSE"
ide1:0.deviceType = "atapi-cdrom"
ide1:0.clientDevice = "TRUE"
ide1:0.fileName = "CD/DVD drive 0"
ide1:0.present = "TRUE"
scsi3:0.deviceType = "scsi-hardDisk"
scsi3:0.fileName = "disk_6.vmdk"
scsi3:0.present = "TRUE"
scsi3:1.deviceType = "scsi-hardDisk"
scsi3:1.fileName = "disk_1.vmdk"
scsi3:1.present = "TRUE"
scsi2:0.deviceType = "scsi-hardDisk"
scsi2:0.fileName = "disk_2.vmdk"
scsi2:0.present = "TRUE"
scsi2:1.deviceType = "scsi-hardDisk"
scsi2:1.fileName = "disk_3.vmdk"
scsi2:1.present = "TRUE"
scsi1:0.deviceType = "scsi-hardDisk"
scsi1:0.fileName = "disk_4.vmdk"
scsi1:0.present = "TRUE"
scsi1:1.deviceType = "scsi-hardDisk"
scsi1:1.fileName = "disk_5.vmdk"
scsi1:1.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "disk.vmdk"
scsi0:0.present = "TRUE"
vmci0.pciSlotNumber = "32"
1Gethernet0.virtualDev = "vmxnet3

After

:floppy0.fileName = "Floppy 0"
ide1:0.startConnected = "FALSE"
ide1:0.deviceType = "atapi-cdrom"
ide1:0.clientDevice = "TRUE"
ide1:0.fileName = "CD/DVD drive 0"
ide1:0.present = "TRUE"
scsi3:0.deviceType = "scsi-hardDisk"
scsi3:0.fileName = "disk.vmdk"
scsi3:0.present = "TRUE"
scsi3:1.deviceType = "scsi-hardDisk"
scsi3:1.fileName = "disk_1.vmdk"
scsi3:1.present = "TRUE"
scsi2:0.deviceType = "scsi-hardDisk"
scsi2:0.fileName = "disk_2.vmdk"
scsi2:0.present = "TRUE"
scsi2:1.deviceType = "scsi-hardDisk"
scsi2:1.fileName = "disk_3.vmdk"
scsi2:1.present = "TRUE"
scsi1:0.deviceType = "scsi-hardDisk"
scsi1:0.fileName = "disk_4.vmdk"
scsi1:0.present = "TRUE"
scsi1:1.deviceType = "scsi-hardDisk"
scsi1:1.fileName = "disk_5.vmdk"
scsi1:1.present = "TRUE"
scsi0:0.deviceType = "scsi-hardDisk"
scsi0:0.fileName = "disk_6.vmdk"
scsi0:0.present = "TRUE"
vmci0.pciSlotNumber = "32"
1Gethernet0.virtualDev = "vmxnet3

Note: I tried editing the .ovf file and adding a key:value pair to the file, and regenerating the SHA1 and stashing the SHA1 in the .mf file.  The process worked, but the VM still did not boot, and the bios.hddOrder param was not in the .vmx file of the VM that was created from the template.