Keeping the vMotion Tiger in the 10GB Cage - Update

There has been a lot of activity regarding my post from last Sunday.  Again, all credit goes out to Don Mann and the ePlus Engineering Team (Rob Quast in particular) for all their hard work surrounding Don's presentation at VMworld.  The title of the session was 10GB and FCoE Design Considerations and it was great!  If you get a chance to catch it on the VMworld replay, I highly suggest it.

I wrote the first post because even though vSphere 4.1 has been out for a few months, it is Don's session that set off the light bulb in my head that made me start asking questions.  I wanted to know from VMware if it was true that vMotion (and really any traffic that isn't controlled) can saturate a 10GB link with vSphere 4.1.  Is this a new design criteria that I now MUST consider?  I asked on the forums and it was confirmed by none other than Dilpreet himself.  Thank you again Dilpreet for taking the time to post!

In the meantime, Sean McGee and Brad Hedlund also wrote articles to further explain the concepts as well as lay out some architecture solutions.  Sean's post is here and Brad's post is here.  Please take the time to read them both, great stuff!

Due to some other things going on, I won't be posting the follow ups that I promised in Part One.  I'm very sorry about that and I hope to get to them someday but the circumstances right now just won't allow that to happen.  Besides, Sean and Brad did a great job (probably better!) than I could have.

That's A Lot of Hardware!

Just a quick post today.  As some saw on Twitter yesterday I will be getting my hands on some pretty impressive hardware.  My company has decided to move our customer demo lab to our office and all the gear arrived yesterday.  Here's a few pictures for now but we will be setting all of this up over the next few weeks.  I will be posting some impressions and tips as I go.  With my HP and IBM Blade background I am hoping to write a good bit on the UCS experience.  In addition to the EMC NS-120, I am hoping to integrate our existing EMC NS-960 for some experience with that hardware as well.  Should be interesting!!

Picture #1 - 2x Cisco UCS Chassis each with 4 blades, 2x Cisco 6120 Nexus, 1x Cisco Nexus 5020

Picture #2 - A LOT of NetApp disk shelves (NetApp controller not pictured)

Picture #3 - EMC NS-120 still in the box (but not for long!)

Cisco 4001i Nexus Switch for IBM BladeCenter in Depth

I have been talking to a few customers recently about the Cisco Nexus 4001i Switch for the IBM BladeCenter.  The product looks very nice and I have recently discovered some additional information that I wanted to share.

If you are unfamiliar with the product, head over to Kevin's site and take a look at this link and this link.  He has some very good links from Cisco about the switch.  In addition, I found a link to the IBM RedPaper on the switch here.  For those of you that don't like links, here's the summary: It is a 20 port (14 down to blades, 6 uplinks), non-blocking 10GB FCoE capable switch utilizing the Nexus OS.  The FCoE functionality is optional.  To enable FCoE you need to purchase the FC Enablement Kit (IBM Part Number 49Y9983).

Here are some additional notes:

• The 4001i is NOT an FC Forwarder, it is a FIP Snooping switch
• You may have already noticed but this switch DOES NOT have FC ports.  To talk FC you will need to go out of the 4001i into a Nexus 5k and break out the FC there
• If using the Emulex Virtual Fabric Blade Adapter there is no vNIC functionality
• The switch only supports Cisco SFP's/SFP+ cables and it doesn't ship with any. You will need to purchase them separately to go with the switch.  They are NOT resold through IBM.  Since there are 6 uplinks, you will need a maximum of 6 SFP's (or copper SFP+ cables) per switch.
• The switch will support a 1GB connection down to the IBM Blade 2port/4port 1GB Adapter. I questioned why you would need this but on second thought I really like this. This way you can provide additional 1GB connections to a server that may not need 10GB without the purchase of additional 1GB switches.  The fact that you can "share" a 1GB and 10GB CFFh slot is VERY nice!
  

I'll post more on the switch as I dig deeper!

My FCoE Light Bulb Moment - FCoE Isn't So Scary After All

Have you ever had one those moments where the light bulb just goes off for something and you just see everything differently? I had that kind of light bulb moment with FCoE Sunday evening. Now that I've had a bit of time to digest it, I wanted to share some thoughts. This all starts from Twitter conversations with Brad Hedlund and Steve Chambers.

I could post a whole bunch of quotes but here are the statements that came out of it.

• Some people view(ed) FCoE as a replacement for FC - If so, the big objection with FCoE is probably that it isn't multi-hop capable and FC is. If FCoE is "FC 2.0" and it isn't multi-hop, then it isn't as good, Right?? But, do I NEED multi-hop today? My knee jerk answer is yes, otherwise it isn't as good! But, how many people are going to fork life upgrade their mature and costly FC infrastructure? Not many I'd bet. So, If FCoE is a transition technology then we can roll it in slowly at the edge and transition the multi-hop FC infrastructure to FCoE over time as the features are added and the product matures. What's so bad about that?
• Some people see FCoE as a convergence of the physical layers of network and FC - It may be sad but I never considered this. I was caught up on the statement above. If it isn't as good as FC, I didn't want to hear about it. Once I got over that point, you do have a nice savings here in the convergence. One thing I LOVE about UCS is how clean everything is. The simplicity is top notch. More about that in another article.
• Some people equate(d) Cisco UCS with FCoE - Cisco UCS isn't FCoE, it is 10GB. It could be FCoE, iSCSI, NFS, etc. As Steve stated, it's all about the AND instead of the OR with Cisco UCS. I think the reason for this is because Cisco is pushing Nexus and FCoE so hard along with UCS. You can't speak to Cisco about one and not get the other. But, at the end of the day, you don't NEED Nexus and FCoE... (gasp! what did I just say!?!?)

To continue the point from my previous article on the subject, what about Cisco UCS over 10GB NFS to the storage for VMware? As I've stated before, this solution to is actually cleaner on the NetApp side and the transport layer is easier to set up. I can't wait to test this configuration!

FCoE vs. NFS on Cisco UCS/VMware/NetApp with Dedupe

This article is quick and dirty. Just the title makes my brain hurt a bit. This came about because of Twitter conversations this afternoon with Brad Hedlund at Cisco regarding UCS design considerations. I am thinking out loud here and wondering what the community thinks about the following.

NOTE: I have not set this up yet so my assumptions may be a little off. My company is supposed to be getting UCS in the near future to demo for customers and at that time I will be able to prototype all of this.

Assume for a second that have Cisco UCS with VMware and you are attaching to a storage system that will do FC or NFS. In this case I'm going to use NetApp because that is what I know.

You have two big options, 1. The customer has FC in place already to the storage. In this case I agree with Brad, use FCoE!

But, what if they don't have FC in place today? Maybe it is a new installation or maybe they are doing IP based storage. I have a large number of customers in this situation.

If there is no legacy FC in place, why would you go FCoE? In my opinion, NFS is much easier to implement in this situation. Let me elaborate.

1. The Transport Layer - Instead of FCoE, I propose you could use straight 10GB IP and run NFS on top of that. I admit, I'm not sure how that will work on the UCS side but I'm assuming that NFS on an ip kernel from VMware is possible to implement inside vSphere. Your end to end configuration is all IP based and MUCH easier to set up than FCoE based on the experience I have had with it to date with the Nexus switches.

2. The Storage Side - NFS on NetApp is handled at the volume level. So is dedupe. Everyone on NetApp with VMware wants dedupe these days. It might as well be a requirement because the savings are just too good to pass up. Because both NFS and dedupe are at the volume level, everything is nice and clean.

Lun based Storage will be required for FC/FCoE. For those that don't know, a lun is basically a file inside a volume for NetApp. It is a level down. Because of this, dedupe on LUNs is tricky. Don't let anyone at NetApp tell you otherwise. In order to fully take advantage of it you need to use thin provisioning and play some other tricks. If you thin provision, you now have introduced a level of risk that thick provisioning LUNS does not have. To be safe in thin provisioning, you need to monitor the environment closely or use alerting. I can go into this much more but it is probably a set of blogs in itself. (For instance, do you do multiple luns inside the volume to take advantage of space, do you thin provision the volume and push the free space to the aggregate, etc)

In summary for NFS: Set up IP storage, create the volume on the NetApp, dedupe the volume

In summary for FCoE: Set up FCoE (more difficult than IP), create a volume, create a lun inside the volume, thin provision the lun, monitor the lun closely, dedupe up at the volume level

It isn't the FCoE that makes it difficult, it is a combination of the more difficult transport configuration combined with dedupe on luns on NetApp!

DISCLAIMER: This entire post was written in about 15 minutes. I reserve right to to make a bunch of writing mistakes and be wrong on the UCS side but I have configured both NFS and FCoE and I have done dedupe on NetApp until my head hurts.

FCoE Planning Article on Scott Lowe's Site

Just a quick note that I wrote an article for Scott Lowe's site on the future of FCoE and potential ways to integrate it into your data center.

Link to Planning for FCoE Article