ORIGINALLY POSTED 9th October 2007

11,690 views on developerworks

The SPC contacted SUN regarding violations of the SPC ‘fair-use’ terms. To that end, John and Valdis have retracted both the newsletter comments and blog posts that claimed ‘faster than SVC’ therefore the links in this post no longer show their original content. I have left this post in place for archive purposes and have disabled further comments. 2nd November 2007

---

We all know that SUN and HP re-badge the Hitachi storage products and call them their own, but it would seem they are also sharing ‘Hitachi math’ too. In particular some very dodgy statistical analysis of the recent SPC-1 USP-V benchmark. I guess they have to put their spin on it too…

Taken from John McLaughlin’s blog – content originally posed on Valdis Weblog and removed at the request of the SPC due to ‘fair use’ infringement.

Sun advantages; 80% less floor space (Sun uses 5 cabinets/racks vs IBM’s 9 cabinets) 512 less disks (which must mean less power usage too) 34% faster at reading data than IBM SVC 59% faster at writing data than IBM SVC I

BM advantages; 26% cheaper 26% more throughput than Sun ST9990V

Now lets look at these statistics. (Throughout I refer to the SUN box as a USP-V, since thats what it is really)

• 80% less floor space (Sun uses 5 cabinets/racks vs IBM’s 9 cabinets) – true, but also with 50% more drives – take away those and you are back to only 6 racks.
• 512 less disks – shouldn’t this be a negative – that this was the limit of what could be tested internally to the ‘monolithic block’ – we needed that many drives to actually get to the 272K number and saturate SVC.
• Valdis’ blog originally mis-represented the USP-V as being ‘faster than SVC’ – this was what caused the issue with ‘fair use’ as the SVC can clearly handle more SPC-1 IOPS – The comparison was for the following figures is still mis-leading, as they compare both systems at 95% utilisation – apples with oranges… One system is doing 190K iops and the other 258K iops at 95% system use. The closest we can actually get from the SPC-1 executive summary reports is 200K vs 219K where I grant you the Hitachi box is about just over 1ms faster (but doing 10% less IOPS) The 4x cache capacity used in the test is likely to contribute a great deal of that. While our EMC friends frequently comment that SPC tries to negate the cache, its not a case of negating as much as determining how quickly a device can detect and make use of a new area of ‘cache hits’ – the larger the cache, the better chance you have of quickly switching to a new area of cache hits.
  • 34% faster at reading data than IBM SVC – again looking at the 200K vs 219K point we are talking about 1.2ms
  • 59% faster at writing data than IBM SVC – yet again about 1.1ms
  Given that SVC was using mid-range DS4000 controllers, I’d expect there to be some difference in the average, as some I/O will have to come from disk, especially when SVC had less cache.
  
  One other major point to note is that the these Hitachi tests were done with the controller directly attached to the host so no switch fabric to traverse which will add to the latency of I/Os. I wonder how many USP’s are attached diectly?

• 26% cheaper – The USP-V is actully 45% more expensive when comparing the $/IOP figures – the 26% cheaper neglects to mention to 50% more disks and SAN infrastructure, and the 36% more throughput.
• 26% more throughput than Sun ST9990V – If you carry on the Hitachi maths used to generate the statistics above 272/200 = 1.36 so actually 36% more throughput!

However, I’ve been studying this new form of ‘Hitachi maths’ that the SUN bloggers have so deftly used, so here’s my novice attempt :

• The USP-V needs 4x the cache capacity to get about 1ms of improved average response time.
• The USP-V needed 2x the fibre channel ports to get 36% less throughput.
• The USP-V was 45% more expensive for 36% less throughput – and doesn’t include the price of the SAN fabric where SVC does.
• SVC can handle over 36% more throughput than USP-V

Anyway, all that said, if you really are comparing SVC using underlying mid-range DS4000 based storage, to a high end controller that happens to virtualize external attach, then I’d be interested to see the equivalent external attach benchmark, if nothing else the resulting spin should be amusing.

The SVC SPC-1 results are available on the SPC website : http://www.storageperformance.org/results/benchmark_results_spc1#a00052
The Hitachi SPC-1 are also available : http://www.storageperformance.org/results/benchmark_results_spc1#a00054
SUN version : http://www.storageperformance.org/results/benchmark_results_spc1#a00055

15th October : Due to the SPC’s contact with SUN and their requests not to use the benchmark figures in a way as to mis-represent the results, Valdis’ retraction of his original post and John McLaughlin’s modified post, I have amended my response since its original posting on the 8th October