Detecting and correcting hardware errors using Nutanix Filesystem.
By: Date: March 15, 2018 Categories: Nutanix Tags: ,

It’s good to detect corrupted data.  It’s even better to transparently repair that data and return the correct data to the user.  Here we will demonstrate how Nutanix filesystem detects and corrects corruption.  Not all systems are made equally in this regard.  The topic of corruption detection and remedy was the focus of this excellent…

Read More →
Simple statistics for performance analysts.
By: Date: January 18, 2018 Categories: Performance Engineering

As performance analysts we often have to summarize large amounts of data in order to make engineering decisions or understand existing behavior.  This paper will help you do exactly that!  Many analysts know that using statistics can help, but statistical analysis is a huge field in itself and has its own complexity.  The article below…

Read More →
The return of misaligned IO
By: Date: May 25, 2017 Categories: Debugging Tags: , ,

We have started seeing misaligned partitions on Linux guests runnning certain HDFS distributions.  How these partitions became mis-aligned is a bit of a mystery, because the only way I know how to do this on Linux is to create a partition using old DOS format like this (using -c=dos  and -u=cylinders) 

Read More →
Creating compressible data with fio.
By: Date: October 18, 2016 Categories: fio Tags: , , ,

Today I used fio to create some compressible data to test on my Nutanix nodes.  I ended up using the following fio params to get what I wanted.   buffer_compress_percentage=50 refill_buffers buffer_pattern=0xdeadbeef buffer_compress_percentage does what you’d expect and specifies how compressible the data is refill_buffers Is required to make the above compress percentage do what you’d…

Read More →
Specifying Drive letters with fio for Windows.
By: Date: October 5, 2016 Categories: fio Tags: , , ,

Simple fio file for using Drive letters on Windows. This will create a file called “fiofile” on the F:\ Drive in Windows.  Notice that the specification is “Driveletter” “Backslash” “Colon” “Filename” In fio terms we are “escaping” the “:” which fio traditionally uses as a file separator. [global] bs=1024k size=1G time_based runtime=30 rw=read direct=1 iodepth=8…

Read More →
SQL*Server on Nutanix. Force backups to HDD.
By: Date: March 10, 2016 Categories: Nutanix Tags:

As an experiment, I wanted to (a) Create a HDD only container, and (b) measure the bandwidth I could achieve when backing up the SQL DB.  This was performed on a standard hybrid platform with only 4 HDD’s in the node. First create a container, but add the special options “sequential-io-priority-order=DAS-SATA random-io-priority-order=DAS-SATA” which means that…

Read More →
Things to know when using vdbench.
By: Date: September 21, 2015 Categories: vdbench Tags: ,

Recently I found that vdbench was not giving me the amount of outstanding IO that I had intended to configure by using the “threads=N” parameter. It turned out that with Linux, most of the filesystems (ext2, ext3 and ext4) do not support concurrent directIO, although they do support directIO. This was a bit of a…

Read More →
SATA on Nutanix. Some experimental data.
By: Date: June 16, 2015 Categories: Storage Hardware and Devices Tags: , , ,

The question of  why  Nutanix uses SATA drive comes up sometimes, especially from customers who have experienced very poor performance using SATA on traditional arrays. I can understand this anxiety.  In my time at NetApp we exclusively used SAS or FC-AL drives in performance test work.  At the time there was a huge difference in performance between…

Read More →
Impact of Paravirtual SCSI driver VS LSI Emulation with Data.
By: Date: March 30, 2015 Categories: Debugging Tags: , ,

TL;DR  Comparison of Paravirtual SCSI Vs Emulated SCSI in with measurements.  PVSCSI gives measurably better response times at high load.  During a performance debugging session, I noticed that the response time on two of the SCSI devices was much higher than the others (Linux host under vmware ESX).  The difference was unexpected since all the devices…

Read More →
Lord Kelvin Vs the IO blender
By: Date: October 2, 2014 Categories: Nutanix

One of the characteristics of a  successful storage system for virtualized environments is that it must handle the IO blender.  Put simply, when lots of regular looking workloads are virtualized and presented to the storage, their regularity is lost, and the resulting IO stream starts to look more and more random.  This is very similar to…

Read More →
Multiple devices/jobs in fio
By: Date: June 17, 2014 Categories: fio Tags: ,

If your underlying filesystem/devices have different response times (e.g. some devices are cached – or are on SSD) and others are on spinning disk, then the behavior of fio can be quite different depending on how the fio config file is specified.  Typically there are two approaches 1) Have a single “job” that has multiple…

Read More →
Here be Zeroes
By: Date: June 12, 2014 Categories: Debugging Tags: ,

Many storage devices/filesystems treat blocks containing nothing but zeros in a special way, often short-circuiting reads from the back-end.  This is normally a good thing but this behavior can cause odd results when benchmarking.  This typically comes up when testing against storage using raw devices that have been thin provisioned. In this example, I have several…

Read More →
Designing a scaleout storage platform.
By: Date: June 12, 2014 Categories: Nutanix Tags: ,

I was speaking to one of our developers the other day, and he pointed me to the following paper:  SEDA: An Architecture for Well-Conditioned, Scalable Internet Services as an example of the general philosophy behind the design of the Nutanix Distributed File System (NDFS). Although the paper uses examples of both a webserver and a gnutella client,…

Read More →