Microsoft diskspd Part 3. Oddities and FAQ

Default OIO is “2” if no other parameters are specified

This is documented on the diskpsd page but most workload generators that i use will default to a single OIO, so it’s worth pointing out.

Default case:

8KB Random IO, no thread or OIO specified
Resource monitor shows a Disk Q (OIO) of ~2 (1.9)

The -o parameter is per-disk and per thread.

Run diskspd with -o 32 (single thread) generates a total of 32 OIO

8KB Random IO, 32 OIO (no thread count specified)
Resource monitor shors a Disk Q (OIO) of ~32

Run diskspd with -o 32 and -t2 (two threads) generates a total of 64 OIO

Run diskspd with -o32, -t8 (eight threads) generates a total of 256 OIO

diskpd -o32 -t8

Run diskspd with -o32, -t8 (eight threads) and 4 Disks – generates 1024 total OIO

Threads and OIO are per disk device.

Using the above with 4 disks will yield 1024 OIO more than most physical HBA’s can deliver by default

To keep total OIO constant across disk count use -O and -F

4 Devices

2 Devices

1 Device

Creating a file with -Zr does not fill the file.

To create a file with non-null bytes, you will have to do a separate write phase after initial creation.

The default fill pattern is highly compressible

  • Writing out a file without a switch writes a Repeating pattern every 256 bytes. A 1FFF file will compress by KKK% to JJJbytes
  • Writing out a file with “-Z” writes NULL bytes
  • Writing out a file with “-Zr” writes totally random bytes.

“Raw” devices are supported

diskspd supports writing directly to the drive (e.g. without a filesystem) using the format #<disknum>

diskspd writing direct to disk device

Writing to a file/device without a time period will default to 10 seconds

Anyone coming from e.g. fio might expect the write to fill the entire file.

diskspd with no time parameter defaults to 10s
diskspd writes only 3.5GB – not the entire 33GB

Microsoft diskspd. Part 2 How to bypass NTFS Cache.

How to ensure performance testing is stressing the underlying storage devices, not the OS filesystem.

Summary tl;dr

There are two ways to ensure that IO goes directly to the back-end storage (direct attach disk, SAN or HCI datastore) Before going further see Part 1 – Preparing to test with diskspd

  1. Use a “raw” or “physical” device (Use the pattern #<diskID> to specify a disk device)
  2. Use files on the with specific flags to bypass the filesystem cache. (-Su or -Sh)

Be very careful about issuing WRITE workloads to a raw disk (using #<diskID>). If there is a filesystem mounted on the disk – you will corrupt the filesystem. My advice is to only use raw disks that have no formatted filesystem.

What’s the difference between “-Su” and “-Sh”?

For Enterprise storage (SAN or HCI) -Su and -Sh should give the same behavior. Using -Sh additionally uses a hint to disable any caching on the hardware device. Enterprise storage usually does not use on-disk-device caching due to possibility of data loss in the event of power failure. When in doubt, use the -Sh. switch.

Below we will see just how different the results can be depending whether caching is allowed and how reading/writing directly to a device can look quite different to using a filesystem.

Continue reading

Microsoft diskspd. Part 1 Preparing to test.

Installing Disk-Speed (diskspd).


diskspd operates on windows filesystems, and will read / write to one or more files concurrently.

The NULL byte problem

By default, when diskspd creates a file it is a file full of NULL bytes. Many storage systems (at least NetApp and Nutanix that I know of) will optimize the layout NULL byte files. This means that test results from NULL byte files will not reflect the performance of real applications that write actual data.

Suggested work-around

To avoid overly optimistic results, first create the file, then write a randomized data pattern to the file before doing any testing.

Create a file using diskspd -c. e.g. for a 32G file on drive D: then overwrite with random data.

diskspd.exe -c32G D:\testfile1.dat

This will create a 32G file full of NULL bytes

Default write command sends NULL bytes to disk

Then overwrite with a random pattern

diskspd.exe -w100 -Zr D:\testfile1.dat
same file after being overwritten with diskspd -w100 -Zr
Continue reading