Containers in VM Vs Containers on Bare metal for AI workloads.
Published: (Updated: ) by .
TL;DR – Running a VLLM container inside a VM was either the same performance or slightly better than running the exact same container directly on Bare Metal.
We recently conducted some experiments in the performance lab to compare the impact of running containerized AI workloads running in Virtual Machines (VMs) to running the exact same containerized workloads directly on Bare Metal (BM).
The workload we chose is AIPerf. AIPerf is a widely used benchmarking tool that runs against an inferencing endpoint, meaning that the entire inferencing stack is being exercised – rather than just the GPU itself.
We specifically chose a larger ISL/OSL (Input/Output Sequence Length) to maximize the communication between the CPU and GPU during the preload phase. Any overhead of virtualization should be shown in a test like this.
Results
Running the AIPerf benchmark against the RTX6000 Pro GPU exposing a model via vLLM showed that running the vLLM container in a VM was about the same performance or slightly better than the same containerized workload running directly on Bare Metal.
Experiment Details
- GPU node:
- INTEL(R) XEON(R) GOLD 6548Y+ (2×32 cores @2.5GHz)
- 512GB DDR5
- Other nodes:
- INTEL(R) XEON(R) GOLD 5218R (2×20 cores @2.1GHz)
- 384GB
- Network: 25Gbps
- GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition (96GB, 450W)
Specific versions of hardware and software used in the experiments outlined below:
- Ubuntu 24.04.4 LTS (GNU/Linux 6.8.0-110-generic x86_64)
- Docker: containerd://1.7.29-d2iq.1
- Kubernetes: v1.34.1
- NKP 2.17.0
- AHV 11.0-1900
- AOS: el8.5-release-ganges-7.5-stable-cd8cd937b6288cf2c58a44a0bc1c58d85bf5c0bb
- NAI 2.7.0
- aiperf 0.5.0
- Model for aiperf: llama-3.1-8b-instruct
Comments