I found out something interesting on Friday, HP is offering a 16GB DIMM in their blades! My first thought was wow, that sucker is gonna be expensive (and it is!). But, after that I started to dig deeper as I always do, I found out something that is slightly disturbing. The 16GB DIMM is actually a quad rank DIMM and not a dual rank DIMM and it is only 1066Mhz speed. Many of you are saying... So what?
Well, this actually can make a difference from a design perspective. Take a look at the Memory tables in the BL460 and BL490 QuickSpecs and you will see what I mean.
HP BL460 G6 QuickSpecs
HP BL490 G6 QuickSpecs
Most of the DIMMs sold by the major vendors are dual rank and 1333Mhz Speed. Let me explain the concept of ranks first. According to the Intel Nehalem architecture, you can only have 8 ranks per memory channel. Each memory channel consisted of either 2 or 3 DIMMs per channel. I have more information on memory layout in this article I wrote on Scott Lowe's site awhile back. I never really discussed ranks because it was a limit you didn't hit. You were only using either 4 ranks (2xdual rank dimms on the BL460) or 6 ranks (3xdual ranks on the BL490). Quad rank DIMMs blows the math out of the water. You can only put 2 on a memory bus to generate 8 ranks. This means the BL490 no longer brings extra memory capacity to the table. Both the BL460 and BL490 top out at 192GB with the 16GB DIMMs.
Memory speed is the next issue. If you are running a memory bandwidth intensive application, you will expect about a 7% boost in performance by keeping the memory speed at 1333MHZ instead of dropping down to 1066Mhz. Because the maximum speed of the 16 GB DIMM is 1066Mhz you will never reach a 1333Mhz speed. Furthermore, populating both slots in the memory channel (the max of 12 DIMMs) drops the speed from 1066 Mhz to 800 Mhz. The performance drop from 1333 Mhz to 800 Mhz is over 30%!! This leads to an interesting trade off of memory capacity vs bandwidth speed.
While I applaud HP for thinking outside the box and bringing a 16GB DIMM to market, don't assume it is the same DIMM as the others. Remember, "One of these is not like the others...."
7 comments:
Aaron,
Another bummer for the HP 16GB DIMM is that it only runs at 1066Mhz when have just (1) DIMM per channel, for 96GB.
If you populate (2) 16GB DIMMs per channel the memory system will run at just 800Mhz. So yes, you can have 192GB of RAM in an HP blade with the 16GB DIMMs, but it comes with a significant performance penalty.
For a high performance memory footprint of 192GB on a single server, you would be much better off with the Cisco UCS B250 blade which can run up to 384GB at 1066Mhz!
Cheers,
Brad
Doh! I missed that! Thank you. I meant to say something. Updating now.
Aaron, great post. As the Cookie Monster instilled in all of us as a ripe young age, the devil is in the details!
UCS through the expander has a latency penalty so you can't compare them just on bus speed. Not to mention that workloads that are memory I/O bound are basically only found in Database and HPC scenarios.
afidel - Have you seen a value attached to the latency score? I haven't seen one so far and I keep hearing this but I'm wondering what it really is.
According to http://knudt.net/vblog/post/2009/10/05/UCS-Boot-Camp-Day-1.aspx it's 6ns which is pretty significant considering it's nearly doubles the latency on a CAS7 chip.
afidel - Interesting... Thank you very much for that find! I read Brian's articles but I missed that.
I guess the big question is anyone really buying the 490 or the b250 that are memory I/O capped? In my experience that answer is no. The main reason my customers are interested in the BL490, b250, or HS22V is because of the number of DIMM slots. They want more memory at a cheaper cost than an 8GB or 16GB DIMM. They want a bunch of 4GB DIMMs to keep the cost down.
That reason was why I wrote the article in the first place. While it is great HP came out with a 16GB DIMM, who is gonna buy it? It is expensive, limits the memory capacity of the 490 to the same as the 460, and drops the memory speed. Where is the win for the customer here? I can go from 144GB to 192GB at who knows how much more cost??
Post a Comment