Microsoft diskspd. Part 2, Testing storage devices.

How to ensure performance testing is stressing the underlying storage devices, not the OS filesystem.

Summary tl;dr

There are two ways to ensure that IO goes directly to the back-end storage (direct attach disk, SAN or HCI datastore)


  1. Use a “raw” or “physical” device (Use the pattern #<diskID> to specify a disk device)
  2. Use files on the with specific flags to bypass the filesystem cache. (-Su or -Sh)

Be very careful about issuing WRITE workloads to a raw disk (using #<diskID>). If there is a filesystem mounted on the disk – you will corrupt the filesystem. My advice is to only use raw disks that have no formatted filesystem.

What’s the difference between “-Su” and “-Sh”?

For Enterprise storage (SAN or HCI) -Su and -Sh should give the same behavior. Using -Sh additionally uses a hint to disable any caching on the hardware device. Enterprise storage usually does not use on-disk-device caching due to possibility of data loss in the event of power failure. When in doubt, use the -Sh. switch.

Below we will see just how different the results can be depending whether caching is allowed and how reading/writing directly to a device can look quite different to using a filesystem.

Continue reading

Microsoft diskpd. Part 1 Preparing to test.

Installing Disk-Speed (diskspd).

Overview

diskspd operates on windows filesystems, and will read / write to one or more files concurrently.

The NULL byte problem

By default, when diskspd creates a file it is a file full of NULL bytes. Many storage systems (at least NetApp and Nutanix that I know of) will optimize the layout NULL byte files. This means that test results from NULL byte files will not reflect the performance of real applications that write actual data.

Suggested work-around

To avoid overly optimistic results, first create the file, then write a randomized data pattern to the file before doing any testing.

Create a file using diskspd -c. e.g. for a 32G file on drive D: then overwrite with random data.

diskspd.exe -c32G D:\testfile1.dat

This will create a 32G file full of NULL bytes

Default write command sends NULL bytes to disk

Then overwrite with a random pattern

diskspd.exe -w100 -Zr D:\testfile1.dat
same file after being overwritten with diskspd -w100 -Zr
Continue reading

How to identify NVME drive types and test throughput

Dmitry Nosachev / CC BY-SA (https://creativecommons.org/licenses/by-sa/4.0)

Start by identifying the NVME drive type (“nvme list“) since the NVME drives use the nvme protocol, they will not sshow up in the output of lsscsi

Identify NVME

Identify the nvme drives using “nvme list”

nutanix@NTNX-18SM3E420118-A-CVM:10.56.6.36:~/tmp$ sudo nvme list
Node SN Model Version Namespace Usage Format FW Rev
/dev/nvme0n1 S3HDNX0KC00137 SAMSUNG MZWLL1T6HEHP-00003 1.2 1 1.60 TB / 1.60 TB 512 B + 0 B GPNA6B3Q
/dev/nvme1n1 S3HDNX0KC00140 SAMSUNG MZWLL1T6HEHP-00003 1.2 1 1.60 TB / 1.60 TB 512 B + 0 B GPNA6B3

The drive types identified as “MZWLL1T6HEHP-00003” the marketing name is PM1725A.

The spec sheet for the PM1725A lists the following performance characteristics

Identify device specs

Random IO size=4K, Sequential=1MB

WorkloadData Sheet ResultMeasured Result
Sequential Read3,300 MB/s3,250 MB/s (QD=8)
Sequential Write2,000 MB/s2,200 MB/s (QD=8)
Random Read800,000 IOPS700,000 (QD=192)
Random Write140,000 IOPS480,000* (QD=192)
Continue reading