Detecting and correcting hardware errors using Nutanix Filesystem.

Posted on March 15, 2018March 16, 2018 by gary

It’s good to detect corrupted data. It’s even better to transparently repair that data and return the correct data to the user. Here we will demonstrate how Nutanix filesystem detects and corrects corruption. Not all systems are made equally in this regard. The topic of corruption detection and remedy was the focus of this excellent Usenix paper Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to Single Errors and Corruptions. The authors find that many systems that should in theory be able to recover corrupted data do not in fact do so.

Within the guest Virtual Machine

Start with a Linux VM and write a specific pattern (0xdeadbeef) to /dev/sdg using fio.
Check that the expected data is written to the virtual disk and generate a SHA1 checksum of the entire disk.

[root@gary]# od -x /dev/sdg

0000000 adde efbe adde efbe adde efbe adde efbe
*
10000000000

[root@gary]# sha1sum /dev/sdg

1c763488abb6e1573aa011fd36e5a3e2a09d24b0  /dev/sdg

The “od” command shows us that the entire 1GB disk contains the pattern 0xdeadbeef
The “sha1sum” command creates a checksum (digest) based on the content of the entire disk.

Within the Nutanix CVM

Connect to the Nutanix CVM
- Locate one of the 4MB egroups that back this virtual disk on the node.
- The virtual disk which belongs to the guest vm (/dev/sdg) is represented in the Nutanix cluster as a series of “Egroups” within the Nutanix filesystem.
- Using some knowledge of the internals I can locate the Egroups which make up the vDisk seen by the guest.
- Double check that this is indeed an Egroup belonging to my vDisk by checking that it contains the expected pattern (0xdeadbeef)

nutanix@NTNX $ od -x 10808705.egroup

0000000 adde efbe adde efbe adde efbe adde efbe
*
20000000

Now simulate a hardware failure and overwrite the egroup with null data
- I do this by reaching underneath the cluster filesystem and deliberately creating corruption, simulating a mis-directed write somewhere in the system.
- If the system does not correct this situation, the user VM will not read 0xdeadbeef as it expects – remember the corruption happened outside of the user VM itself.

nutanix@ $ dd if=/dev/zero of=10846352.egroup bs=1024k count=4

Use the “dd” command to overwrite the entire 4MB Egroup with /dev/zero (the NULL character).

Back to the client VM

We can tell if the correct results are returned by checking the checksums match the pre-corrupted values.

[root@gary-tpc tmp]# sha1sum /dev/sdg

1c763488abb6e1573aa011fd36e5a3e2a09d24b0  /dev/sdg. <— Same SHA1 digest as the "pre corruption" state.

The checksum matches the original value – showing that the data in entire the vdisk is unchanged
However we did change the vdisk by overwriting one of the. Egroups.
The system has somehow detected and repaired the corruption which I induced.
How?

Magic revealed

Nutanix keeps the checksums at an 8KB granularity as part of our distributed metadata. The system performs the following actions
- Detects that the checksums stored in metadata no longer match the data on disk.
  - The stored checksums match were generated against “0xdeadbeef”
  - The checksums generated during read be generated against <NULL>
  - The checksums will not match and corrective action is taken.
- Nutanix OS
  - Finds the corresponding un-corrupted Egroup on another node
  - Copies the uncorrupted Egroup to a new egroup on the local node
  - Fixes the metadata to point to the new fixed copy
  - Removed corrupted egroup
  - Returns the uncorrupted data to the user

Logs from the Nutanix VM

Here are the logs from Nutanix: notice group 10846352 is the one that we deliberately corrupted earlier

E0315 13:22:37.826596 12085 disk_WAL.cc:1684] Marking extent group 10846352 as corrupt reason: kSliceChecksumMismatch

I0315 13:22:37.826755 12083 vdisk_micro_egroup_fixer_op.cc:156] vdisk_id=10808407 operation_id=387450 Starting fixer op on extent group 10846352 reason -1 reconstruction mode 0 (gflag 0)  corrupt replica autofix mode  (gflag auto)  consistency checks 0 start erasure overwrite fixer 0

I0315 13:22:37.829532 12086 vdisk_micro_egroup_fixer_op.cc:801] vdisk_id=10808407 operation_id=387449 Not considering corrupt replica 38 of egroup 10846352

Data corruption can and does happen (see the above Usenix paper for some of the causes). When designing enterprise storage we have to deal with it
Nutanix not only detects the corruption, it corrects it.
In fact Nutanix OS continually scans the data stored on the cluster and makes sure that the stored data matches the expected checksums.
- Doing this avoids latent errors in the system – a major cause of multiple disk failures.

SQL*Server on Nutanix. Force backups to HDD.

Posted on March 10, 2016March 10, 2016 by gary

As an experiment, I wanted to (a) Create a HDD only container, and (b) measure the bandwidth I could achieve when backing up the SQL DB. This was performed on a standard hybrid platform with only 4 HDD’s in the node.

First create a container, but add the special options “sequential-io-priority-order=DAS-SATA random-io-priority-order=DAS-SATA” which means that all IO will be directed to the HDD only. This also means that data on this container will never be migrated up. This is just fine for a backup that will hopefully never be read, and if it is – only once, sequentially.

ncli> ctr create name=cold-only sequential-io-priority-order=DAS-SATA random-io-priority-order=DAS-SATA sp-name=all
ncli> datastore create name=cold ctr-name=cold-only

Next create a vDisk in that container – this disk will contain the SQL Server backup data

Format and initialize the drive.

Add backup targets to the drive. Adding multiple targets increases throughput because SQL Server will generate 1-2 outstanding IO’s per target. I created 16 total targets (these are just files).

The first backup is a little slow (~64MB/s), because we’re creating the files. A second (and subsequent) backups go faster, around 120 MB/s writing directly to the HDD spindles on a single node with 4 HDDs.

This backup stream drives around 25MB/s per HDD spindle on the Nutanix node. On a larger platform with more spindles – we could easily drive 500MB/s, and still skip SSD by writing directly to HDD.

120 MB/s Each way — Backup just started. About 115MB/s read, 115MB/s write on same node.

Completed backup:

SuperScalin’: How I learned to stop worrying and love SQL Server on Nutanix.

Posted on November 23, 2014December 27, 2017 by gary

TL;DR It’s pretty easy to get 1M SQL TPM running a TPC-C like workload on a single Nutanix node. Use 1 vDisk for Log files, and 6 vDisks for data files. SQL Server needs enough CPU and RAM to drive it. I used 16 vCPU’s and 64G of RAM.

Running database servers on Nutanix is an increasing trend and DBA’s are naturally skeptical about moving their DB’s to new platforms. I recently had the chance to run some DB benchmarks on a couple of nodes in our lab. My goal was to achieve 1M SQL transactions per node, and have that be linearly scalable across multiple nodes.

It turned out to be ridiculously easy to generate decent numbers using SQL Server. As a Unix and Oracle old-timer it was a shock to me, just how simple it is to throw up a SQL server instance. In this experiment, I am using Windows Server 2012 and SQL-Server 2012.

For the test DB I provision 1 Disk for the SQL log files, and 6 disks for the data files. Temp and the other system DB files are left unchanged. Nothing is tuned or tweaked on the Nutanix side, everything is setup as per standard best practices – no “benchmark specials”.

Load is being generated by HammerDB configured to run the OLTP database workload. I get a little over 1Million SQL transactions per minute (TPM) on a single Nutanix node. The scaling is more-or-less linear, yielding 4.2 Million TPM with 4 Nutanix nodes, which fit in a single 2U chassis . Each node is running both the DB itself, and the shared storage using NDFS. I stopped at 6 nodes, because that’s all I had access to at the time.

The thing that blew me away in this was just how simple it had been. Prior to using SQL server, I had been trying to set up Oracle to do the same workload. It was a huge effort that took me back to the 1990’s, configuring kernel parameters by hand – just to stand up the DB. I’ll come back to Oracle at a later date.

My SQL Server is configured with 16 vCPU’s and 64GB of RAM, so that the SQL server VM itself has as many resources as possible, so as not to be the bottleneck.

I use the following flags on SQL server. In SQL terminology these are known as traceflags which are set in the SQL console (I used “DBCC trace status” to display the following. These are fairly standard and are mentioned in our best practice guide.

One thing I did change from the norm was to set the target recovery time to 240 seconds, rather than let SQL server determine the recovery time dynamically. I found that in the benchmarking scenario, SQL server would not do any background flushing at all, and then suddenly would checkpoint a huge amount of data which caused the TPM to fluctuate wildly. With the recovery time hard coded to 240 seconds, the background page flusher keeps up with the incoming workload, and does not need to issue huge checkpoints. My guess is that in real (non benchmark conditions) SQL server waits for the incoming work to drop-off and issues the checkpoint at that time. Since my benchmark never backs off, SQL server eventually has to issue the checkpoint.

Lord Kelvin Vs the IO blender

Posted on October 2, 2014October 2, 2014 by gary

One of the characteristics of a successful storage system for virtualized environments is that it must handle the IO blender. Put simply, when lots of regular looking workloads are virtualized and presented to the storage, their regularity is lost, and the resulting IO stream starts to look more and more random.

This is very similar to the way that synthesisers work – they take multiple regular sine waves of varying frequencies and add them together to get a much more complex sound.

http://msp.ucsd.edu/techniques/v0.11/book-html/node14.html#fig01.08

That’s all pretty awesome for making cool space noises, but not so much when presented to the storage OS. Without the ability to detect regularity, things like caching, pre-fetching and any kind of predictive algorithm break down.

That pre-fetch is never going to happen.

In Nutanix NOS we treat each of these sine waves (workloads) individually, never letting them get mixed together. NDFS knows about vmdk’s or vhdx disks – and so by keeping the regular workloads separate we can still apply all the usual techniques to keep the bits flowing, even at high loads and disparate workload mixes that cause normal storage systems to fall over in a steaming heap of cache misses and metadata chaos.

Designing a scaleout storage platform.

Posted on June 12, 2014March 5, 2017 by gary

I was speaking to one of our developers the other day, and he pointed me to the following paper: SEDA: An Architecture for Well-Conditioned, Scalable Internet Services as an example of the general philosophy behind the design of the Nutanix Distributed File System (NDFS).

Although the paper uses examples of both a webserver and a gnutella client, the philosophies are relevant to a large scale distributed filesystem. In the case of NDFS we are serving disk blocks to clients who happen to be virtual machines. One trade-off that is true in both cases is that scalability is traded for low latency in the single-stream case. However at load, the response time is generally better than a system that is designed to low-latency, and then attempted to scale-up to achive high throughput.

At Nutanix we often talk about web-scale architectures, and this paper gives a pretty solid idea of what that might mean in concrete terms.

FWIW., according to google scholar, the paper has been cited 937 times, including Cassandra which is how we store filesystem meta-data in a distributed fashion.