2010-02.2: Benchmarketing

ORIGINALLY POSTED 14th January 2010

9,231 views on developerworks

Two posts in one week. Doing well. I set one of my resolutions this year to post at least once a week – hence my new numbering scheme. Maybe I’ll use the fact that 2010-2.2 and further double posts count for the odd week I may miss 🙂

Benchmarketing. It’s a dodgy term, and even dodgier maths. We all know and love Hitachi’s maths, and I couldn’t help but comment back at SNW about several hundred PB behind a controller than can only do 200K IO/s – glad a few astute listeners got my humour (ianhf)

OMG – someone actually called their business Benchmarketing

Chris M’s recent post on benchmark results for CNA’s did get a few of us storage twitters commenting, and as with all performance numbers, the devil is in the detail.

I don’t see the point in publishing misleading, unrealistic or simply wrong performance numbers. The kind of people that are going to take notice of said numbers are the same sort of people that know what a real benchmark means.

Most of my benchmarking is done using AIX. I’m still running 5.3 but then I know it, understand it and know all the things I need to do to make my test suite run correctly. Recently, for iSCSI (still feels like a swear word) testing I’ve had to use Linux for various reasons. Now the buffer cache maybe a good thing, but I sure wish you could turn it off completely to really get a true number for what the disks are achieving. O_DIRECT doesn’t do it all, for sequential I/O at 512byte I’m still seeing 4KB I/O at the SVC level. Now maybe anyone doing a 512byte sequential read workload in the real world needs their head examined anyway… but it makes you wonder just how much of your own results you can trust… never mind someone else… who specifies no details on what or how they tested.

SUN’s recent 1U SSD box is a prime example, the devil is in the detail. 1.6M iops from 1U… no you need a lot more U for the SAS direct attach, like 12x. When you start to look, most benchmarks are like this, how do you compare like for like.

That’s why I do like the SPC. It’s the same test for everyone, and as some soon to be published results will show, despite EMC’s attempts to claim it is just a scaling of number of HDD, we can now prove them wrong – and with not an SSD in sight… more next week on this.

My point for today is, any vendor can issue or sponsor a test. Any vendor can chose to ignore performance and not admit that they need to put 3 boxes in for 1 of another to achieve the same – and fair enough – if you sell SLAs then so be it, but what about the power, cooling and footprint costs. For any benchmark, without details like host type, tool used, block size, cache hit ratio, and side by side comparisons, a number is just a number.

Many years ago a maths (yes, its mathematics, not mathematic) teacher wisely started a lesson with the statement :Statistics can tell you anything

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: