Cross rack network latency in AWS
By: Date: August 20, 2020 Categories: Workloads & Benchmarks Tags: ,

I have VMs running on bare-metal instances. Each bare-metal instance is in a separate rack by design (for fault tolerance). The bandwidth is 25GbE however, the response time between the hosts is so high that I need multiple streams to consume that bandwidth.

Compared to my local on-prem lab I need many more streams to get the observed throughput close to the theoretical bandwidth of 25GbE

# iperf StreamsAWS ThroughputOn-Prem Throughput
14.8 Gbit21.4 Gbit
29 Gbit22 Gbit
418 Gbit22.5
823 Gbit23 Gbit
Difference in throughput for a 25GbE network on-premises Vs AWS cloud (inter-rack)

One thought on “Cross rack network latency in AWS

  1. As documented here:

    Multiple flows are required, effectively one per 5gbps for a link to reach higher speeds. There may also be latency at play, but even cluster placement requires a flow per 10gbps. It’s one of those annoying gotchas that aren’t well-documented in something other than blog posts and powerpoints. In-region latency should still allow this type of scaling, not just in-AZ or in-placement-group.

    But yes, it’s limited to 5gbps per flow, which is why you see the scaling you do.

Leave a Comment