I have VMs running on bare-metal instances. Each bare-metal instance is in a separate rack by design (for fault tolerance). The bandwidth is 25GbE however, the response time between the hosts is so high that I need multiple streams to consume that bandwidth.

Compared to my local on-prem lab I need many more streams to get the observed throughput close to the theoretical bandwidth of 25GbE

# iperf Streams	AWS Throughput	On-Prem Throughput
1	4.8 Gbit	21.4 Gbit
2	9 Gbit	22 Gbit
4	18 Gbit	22.5
8	23 Gbit	23 Gbit