HCI Performance testing made easy (Part 2)

Published: (Updated: ) in X-Ray, , , , by .

A simple benchmark for Random Reads, Random Writes, Sequential Reads, Sequential Writes.

HCI Performance testing made easy (Part 1)

Published: (Updated: ) in X-Ray, , , , by .

How to create a customized performance test using X-ray.

Storage Bus Speeds 2018

Published: (Updated: ) in Storage, , by .

Storage bus speeds with example storage endpoints. Bus Lanes End-Point Theoretical Bandwidth (MB/s) Note SAS-3 1 HBA <-> Single SATA Drive 600 SAS3<->SATA 6Gbit SAS-3 1 HBA <-> Single SAS Drive 1200 SAS3<->SAS3 12Gbit SAS-3 4 HBA <-> SAS/SATA Fanout 4800 4 Lane HBA to Breakout (6 SSD)[2] SAS-3 8 HBA <-> SAS/SATA Fanout 8400 […]

Detecting and correcting hardware errors using Nutanix Filesystem.

Published: (Updated: ) in Nutanix, , by .

It’s good to detect corrupted data.  It’s even better to transparently repair that data and return the correct data to the user.  Here we will demonstrate how Nutanix filesystem detects and corrects corruption.  Not all systems are made equally in this regard.  The topic of corruption detection and remedy was the focus of this excellent […]

Meltdown, speculative execution and side-channels explainer.

Published: (Updated: ) in Miscellaneous by .

There are a lot of explanations for the current Meltdown/Spectre crisis but many did not do a good job of explaining the core issue if how information is leaked from the secret side,  to the attackers side.  This is my attempt to explain it (mostly to myself to make sure I got it right). What […]

Simple statistics for performance analysts.

Published: (Updated: ) in Performance Engineering by .

As performance analysts we often have to summarize large amounts of data in order to make engineering decisions or understand existing behavior.  This paper will help you do exactly that!  Many analysts know that using statistics can help, but statistical analysis is a huge field in itself and has its own complexity.  The article below […]

Your IO performance benchmarks are BS and so are mine.

Published: (Updated: ) in Performance Engineering, , , , by .

When it comes to assessing fitness of purpose, even audited benchmarks are quite useless unless they incorporate failure testing alongside the load test. I helped develop the TPCx-HCI benchmark which mandates the simulation of a node failure. Who cares about infrastructure benchmarking? 

The return of misaligned IO

Published: (Updated: ) in Debugging, , , by .

We have started seeing misaligned partitions on Linux guests runnning certain HDFS distributions.  How these partitions became mis-aligned is a bit of a mystery, because the only way I know how to do this on Linux is to create a partition using old DOS format like this (using -c=dos  and -u=cylinders) 

High Response time and low throughput in vCenter performance charts.

Published: (Updated: ) in Debugging, by .

Often we are presented with a vCenter screenshot, and an observation that there are “high latency spikes”.  In the example, the response time is indeed quite high – around 80ms.

Creating compressible data with fio.

Published: (Updated: ) in fio, , , , by .

Today I used fio to create some compressible data to test on my Nutanix nodes.  I ended up using the following fio params to get what I wanted.   buffer_compress_percentage=50 refill_buffers buffer_pattern=0xdeadbeef buffer_compress_percentage does what you’d expect and specifies how compressible the data is refill_buffers Is required to make the above compress percentage do what you’d […]

Code Red – The healthcare.gov story.

Published: (Updated: ) in War Stories by .

A downtime classic, for several months in 2013 the troubles of a very particular website were front page news across the US.  Full Story from Time Magazine (PDF) An In-Depth Look at the Team that Saved HealthCare.gov    

Specifying Drive letters with fio for Windows.

Published: (Updated: ) in fio, , , , by .

Simple fio file for using Drive letters on Windows. This will create a file called “fiofile” on the F:\ Drive in Windows.  Notice that the specification is “Driveletter” “Backslash” “Colon” “Filename” In fio terms we are “escaping” the “:” which fio traditionally uses as a file separator. [global] bs=1024k size=1G time_based runtime=30 rw=read direct=1 iodepth=8 […]

Cache behavior – How long will it take to fill my cache?

Published: (Updated: ) in Modeling by .

When benchmarking filesystems or storage, we need to understand the caching effects. Most often this involves filling the cache and reaching steady state. But how long will it take to fill a cache of a given size? The answer depends of course on the size of the cache, the IO size and the IO rate. […]

SQL*Server on Nutanix. Force backups to HDD.

Published: (Updated: ) in Nutanix, by .

As an experiment, I wanted to (a) Create a HDD only container, and (b) measure the bandwidth I could achieve when backing up the SQL DB.  This was performed on a standard hybrid platform with only 4 HDD’s in the node. First create a container, but add the special options “sequential-io-priority-order=DAS-SATA random-io-priority-order=DAS-SATA” which means that […]

Things to know when using vdbench.

Published: (Updated: ) in vdbench, , by .

Recently I found that vdbench was not giving me the amount of outstanding IO that I had intended to configure by using the “threads=N” parameter. It turned out that with Linux, most of the filesystems (ext2, ext3 and ext4) do not support concurrent directIO, although they do support directIO. This was a bit of a […]

Work around for bios.hddOrder when creating an OVF/OVA template.

Published: (Updated: ) in Automation & Administration, , by . Leave a Comment on Work around for bios.hddOrder when creating an OVF/OVA template..

When changing SCSI devices in an ESX based VM, it’s easy to screw up the ability to boot.  The simple fix is to add bios.hddOrder = “scsi0:0” to the end of the .vmx file.  This has always worked for me.  The problem with this solution is that any OVF/OVA that is created from the VM […]

SATA on Nutanix. Some experimental data.

Published: (Updated: ) in Storage Hardware and Devices, , , , by . Leave a Comment on SATA on Nutanix. Some experimental data..

The question of  why  Nutanix uses SATA drive comes up sometimes, especially from customers who have experienced very poor performance using SATA on traditional arrays. I can understand this anxiety.  In my time at NetApp we exclusively used SAS or FC-AL drives in performance test work.  At the time there was a huge difference in performance between […]

Impact of Paravirtual SCSI driver VS LSI Emulation with Data.

Published: (Updated: ) in Debugging, , , by . Leave a Comment on Impact of Paravirtual SCSI driver VS LSI Emulation with Data..

TL;DR  Comparison of Paravirtual SCSI Vs Emulated SCSI in with measurements.  PVSCSI gives measurably better response times at high load.  During a performance debugging session, I noticed that the response time on two of the SCSI devices was much higher than the others (Linux host under vmware ESX).  The difference was unexpected since all the devices […]

SuperScalin’: How I learned to stop worrying and love SQL Server on Nutanix.

Published: (Updated: ) in SQL Server, , , , by . 3 Comments on SuperScalin’: How I learned to stop worrying and love SQL Server on Nutanix..

TL;DR  It’s pretty easy to get 1M SQL TPM running a TPC-C like workload on a single Nutanix node.  Use 1 vDisk for Log files, and 6 vDisks for data files.  SQL Server  needs enough CPU and RAM to drive it.  I used 16 vCPU’s  and 64G of RAM. Running database servers on Nutanix is an […]

Lord Kelvin Vs the IO blender

Published: (Updated: ) in Nutanix by .

One of the characteristics of a  successful storage system for virtualized environments is that it must handle the IO blender.  Put simply, when lots of regular looking workloads are virtualized and presented to the storage, their regularity is lost, and the resulting IO stream starts to look more and more random.  This is very similar to […]