2M IOPS on a single VM with Nutanix HCI
Published: (Updated: ) by . Leave a Comment on 2M IOPS on a single VM with Nutanix HCI.
How to generate a lot of IOPS to a single VM.
Using fio to read from Linux buffer-cache
Published: (Updated: ) by . Leave a Comment on Using fio to read from Linux buffer-cache.
Sometimes we want to read from the Linux cache rather than the underlying device using fio. There are couple of gotchas that might trip you up. Thankfully fio provides the required work-arounds. TL;DR To get this to work as expected (reads are serviced from buffer cache) – the best way is to use the option […]
fio versions < 3.3 may show inflated random write performance
Published: (Updated: ) by . Leave a Comment on fio versions < 3.3 may show inflated random write performance.
TL;DR If your storage system implements inline compression, performance results with small IO size random writes with time_based and runtime may be inflated with fio versions < 3.3 due to fio generating unexpectedly compressible data when using fio’s default data pattern. Although unintuitive, performance can often be increased by enabling compression especially if the bottleneck […]
Specifying Drive letters with fio for Windows.
Published: (Updated: ) by . 1 Comment on Specifying Drive letters with fio for Windows..
fio on Windows Download pre-compiled fio binary for Windows Example fio windows file, single drive This will create a 1GB file called fiofile on the F:\ Drive in Windows then read the file. Notice that the specification is “Driveletter” “Backslash” “Colon” “Filename” In fio terms we are “escaping” the : which fio traditionally uses as […]
Hunting for bandwidth on a consumer NVMe drive
Published: (Updated: ) by . Leave a Comment on Hunting for bandwidth on a consumer NVMe drive.
The Samsung SSD 970 EVO 500GB claims a sequential read bandwidth of 3400 MB/s this is a story of trying to achieve that number.
Beware of tiny working-set-sizes when testing storage performance.
Published: (Updated: ) by . Leave a Comment on Beware of tiny working-set-sizes when testing storage performance..
I was recently asked to investigate why Nutanix storage was not as fast as a competing solution in a PoC environment. When I looked at the output from diskspd, the data didn’t quite make sense.
Single threaded DB performance on Nutanix HCI
Published: (Updated: ) by . 1 Comment on Single threaded DB performance on Nutanix HCI.
tl;dr A Nutanix cluster can persist a replicated write across two nodes in around 250 uSec which is critical for single-threaded DB write workloads. The performance compares very well with hosted cloud database instances using the same class of processor (db.r5.4xlarge in the figure below). The metrics below are for SQL insert transactions not the […]
Using rwmixread and rate_iops in fio
Published: (Updated: ) by . 2 Comments on Using rwmixread and rate_iops in fio.
Creating a mixed read/write workload with fio can be a bit confusing. Assume we want to create a fixed rate workload of 100 IOPS split 70:30 between reads and writes. TL;DR Specify the rate directly with rate_iops=<read-rate>,<write-rate> do not try to use rwmixread with rate_iops. For the example above use. rate_iops=70,30 Additionally older versions of […]
Understanding fio norandommap and randrepeat parameters
Published: (Updated: ) by . 2 Comments on Understanding fio norandommap and randrepeat parameters.
The parameters norandommap and randrepeat significantly change the way that repeated random IO workloads will be executed, and also can meaningfully change the results of an experiment due to the way that caching works on most storage system.
How to identify NVME drive types and test throughput
Published: (Updated: ) by .
How to identify SSD types and measure performance.
Published: (Updated: ) by .
The real-world achievable SSD performance will vary depending on factors like IO size, queue depth and even CPU clock speed. It’s useful to know what the SSD is capable of delivering in the actual environment in which it’s used. I always start by looking at the performance claimed by the manufacturer. I use these figures […]
How to run vertica vioperf tool
Published: (Updated: ) by .
The vertica vioperf tool is used to determine whether the storage you are planning on using is fast enough to feed the vertica database. When I initially ran the tool, the IO performance reported by the tool and confirmed by iostat was much lower than I expected for the storage device (a 6Gbit SATA device […]
Working with fio “distribution /pereto” parameter
Published: (Updated: ) by .
The fio Pareto parameter allows us to create a workload, which references a very large dataset, but specify a hotspot for the access pattern. Here’s an example using the same setup as the ILM experiment, but using a Pareto value of 0:8. My fio file looks like this.. [global] ioengine=libaio direct=1 time_based norandommap random_distribution=pareto:0.8 The […]
The return of misaligned IO
Published: (Updated: ) by .
We have started seeing misaligned partitions on Linux guests runnning certain HDFS distributions. How these partitions became mis-aligned is a bit of a mystery, because the only way I know how to do this on Linux is to create a partition using old DOS format like this (using -c=dos and -u=cylinders)
High Response time and low throughput in vCenter performance charts.
Published: (Updated: ) by .
Often we are presented with a vCenter screenshot, and an observation that there are “high latency spikes”. In the example, the response time is indeed quite high – around 80ms.
Creating compressible data with fio.
Published: (Updated: ) by .
Today I used fio to create some compressible data to test on my Nutanix nodes. I ended up using the following fio params to get what I wanted. buffer_compress_percentage=50 refill_buffers buffer_pattern=0xdeadbeef buffer_compress_percentage does what you’d expect and specifies how compressible the data is refill_buffers Is required to make the above compress percentage do what you’d […]
Things to know when using vdbench.
Published: (Updated: ) by .
Recently I found that vdbench was not giving me the amount of outstanding IO that I had intended to configure by using the “threads=N” parameter. It turned out that with Linux, most of the filesystems (ext2, ext3 and ext4) do not support concurrent directIO, although they do support directIO. This was a bit of a […]
Multiple devices/jobs in fio
Published: (Updated: ) by .
If your underlying filesystem/devices have different response times (e.g. some devices are cached – or are on SSD) and others are on spinning disk, then the behavior of fio can be quite different depending on how the fio config file is specified. Typically there are two approaches 1) Have a single “job” that has multiple […]
Here be Zeroes
Published: (Updated: ) by .
Many storage devices/filesystems treat blocks containing nothing but zeros in a special way, often short-circuiting reads from the back-end. This is normally a good thing but this behavior can cause odd results when benchmarking. This typically comes up when testing against storage using raw devices that have been thin provisioned. In this example, I have several […]