A Nutanix cluster can persist a replicated write across two nodes in around 250 uSec which is critical for single-threaded DB write workloads. The performance compares very well with hosted cloud database instances using the same class of processor (db.r5.4xlarge in the figure below). The metrics below are for SQL insert transactions not the underlying IO.
I have VMs running on bare-metal instances. Each bare-metal instance is in a separate rack by design (for fault tolerance). The bandwidth is 25GbE however, the response time between the hosts is so high that I need multiple streams to consume that bandwidth.
Compared to my local on-prem lab I need many more streams to get the observed throughput close to the theoretical bandwidth of 25GbE
| # iperf Streams | AWS Throughput | On-Prem Throughput |
| 1 | 4.8 Gbit | 21.4 Gbit |
| 2 | 9 Gbit | 22 Gbit |
| 4 | 18 Gbit | 22.5 |
| 8 | 23 Gbit | 23 Gbit |
