Tag: benchmarking

HammerDB: Avoiding bottlenecks in client.

HammerDB is a great tool for running Database benchmarks. However it is very easy to create an artificial bottleneck which will give a very poor benchmark result.

When setting up HammerDB to run against even a moderate modern server, it is important to avoid displaying the client transaction outputs in the HammerDB UI.

In my case just making this simple changed increased my HammerDB results by over 6X. The reason is that HammerDB spends more time updating its own UI, than it does sending transactions to the DB. When I run HammerDB, I select “Log Output to Temp” and “Use Unique Log Name”.

  • Either:
    • Un-check the “Show results” mark.
    • Or Ensure that the results are logged to file, not displayed on screen
  • Otherwise the workload generator will become the bottleneck.
Show Output “Checked”
Show Output “Un-Checked”
HammerDB with “Show results” unchecked. SQL Server uses all the CPU (99%)
HammerDB with “Show results” checked. SQL Servr using ~20% HammerDB driver (Identified as wish86t) is using 99% of one CPU.

Paper: A Nine year study of filesystem and storage benchmarking

A 2007 paper, that still has lots to say on the subject of benchmarking storage and filesystems. Primarily aimed at researchers and developers, but is relevant to anyone about to embark on a benchmarking effort.

A Nine year study of filesystem and storage benchmarkingDownload
  • Use a mix of macro and micro benchmarks
  • Understand what you are testing, cached results are fine – as long as that is what you had intended.

The authors are clear on why benchmarks remain important:

“Ideally, users could test performance in their own settings using real work- loads. This transfers the responsibility of benchmarking from author to user. However, this is usually impractical because testing multiple systems is time consuming, especially in that exposing the system to real workloads implies learning how to configure the system properly, possibly migrating data and other settings to the new systems, as well as dealing with their respective bugs.”

We cannot expect end-users  to be experts in benchmarking. It is out duty as experts  to provide the tools (benchmarks) that enable users to make purchasing decisions without requiring years of benchmarking expertise.