Keeping the vMotion Tiger in the 10GB Cage - Update

There has been a lot of activity regarding my post from last Sunday.  Again, all credit goes out to Don Mann and the ePlus Engineering Team (Rob Quast in particular) for all their hard work surrounding Don's presentation at VMworld.  The title of the session was 10GB and FCoE Design Considerations and it was great!  If you get a chance to catch it on the VMworld replay, I highly suggest it.

I wrote the first post because even though vSphere 4.1 has been out for a few months, it is Don's session that set off the light bulb in my head that made me start asking questions.  I wanted to know from VMware if it was true that vMotion (and really any traffic that isn't controlled) can saturate a 10GB link with vSphere 4.1.  Is this a new design criteria that I now MUST consider?  I asked on the forums and it was confirmed by none other than Dilpreet himself.  Thank you again Dilpreet for taking the time to post!

In the meantime, Sean McGee and Brad Hedlund also wrote articles to further explain the concepts as well as lay out some architecture solutions.  Sean's post is here and Brad's post is here.  Please take the time to read them both, great stuff!

Due to some other things going on, I won't be posting the follow ups that I promised in Part One.  I'm very sorry about that and I hope to get to them someday but the circumstances right now just won't allow that to happen.  Besides, Sean and Brad did a great job (probably better!) than I could have.

Keeping the vMotion Tiger in the 10GB Cage - Part One

I had a light bulb moment as I was sitting in fellow ePlus employee Don Mann's session at VMworld.

A little background is needed first.  In vSphere version 4.0 we didn't really have a need to control the traffic in 10GB connections.  Even if all the traffic types were combined into a single connection with no traffic management, you rarely ran into contention on the link.  vMotion was the most likely to act up due to the "bursty" nature of the traffic pattern (hit the connection really hard for a few seconds until the vMotion is complete and then settle down) but this was limited because vMotion in vSphere 4.0 was capped at two concurrent vMotions at about 2.6 Gbps each for a total of 5(ish) Gbps maximum for vMotion.  If you assume a little over 9 Gbps usable capacity (the rest lost to protocol overhead) on a 10GB link you still have room for other traffic and you never burst high enough to saturate the network.

Then, along came vSphere 4.1....

vSphere 4.1 introduced significant performance enhancements to vMotion over 4.0.  vSphere 4.1 increases the number of concurrent vMotions to eight in a 10GB environment and the speed has been increased to 8 Gbps.


When I heard this, a light bulb went off in my head and I've been poking at this idea with a stick for awhile now.  I've asked around in the community over the last few days and there seems to be confusion over the numbers.  Does that mean eight vMotions, each one at 8Gbps for a total of 64 Gbps maximum or does that mean eight concurrent vMotions consuming a total of 8Gbps maximum.  I don't have a definitive answer to this question but tests I have seen conducted point to EACH vMotion consuming up to 8Gbps each.  If this is true, anything above ONE vMotion at a time without some form of traffic control may not be a good thing!

Does it matter if I'm utilizing 64 Gbps for vMotion or 8Gbps for vMotion??


The more I think about it, it really doesn't.  Let's assume best case for a second and say that eight vMotions will consume a total of 8Gbps (I don't think it works this way but I'm being an optimist).  If vMotion can consume a maximum of 8Gbps of a 10GB pipe, you will need to design around this fact.  Some form of Traffic Shaping and/or Quality of Service to manage the traffic will be necessary in 4.1 when it was often considered optional previously.

I did a little digging and the issue is confirmed in VMware's NetIOC Best Practices document.  To summarize, your results may vary (and not in a good way) if you aren't putting some form of control on your vMotion traffic in conjunction with 10GB links.

Oh, before I get a bunch of comments telling me this: I'm picking on vMotion here but you could just as easily perform a global replace in this article with (your favorite chatty and/or spikey traffic type) for vMotion in this article.  The concepts to solve network congestion are the same.

How do we solve this issue?

There are two main ways to solve bandwidth contention.  One is to place a cap on the amount of traffic vMotion can use.  This is often referred to as rate limiting the links.  The second is to give priority based on a weighted system that kicks in when contention takes place.  This is called Quality of Service or QoS.  With QoS, everyone gets some bandwidth, but no one is allowed to take over completely and priority is given to critical traffic.  I wrote an article on the concepts in the past here and Brad Hedlund wrote a great article on the concepts with cool Flash animations here.  Don't get hung up that we both wrote about HP and Cisco, the concept of rate limits vs QoS still stands.

In my opinion a QoS or shares based priority model is much more effective to control this traffic.  This allows for better utilization of the bandwidth and provides a more flexible alternative to rate limiting.

How do Rate Limits and QoS fit into vSphere?

Here is a simple graphic to illustrate the virtual switch options in vSphere today:

This concludes the first article in this series.  I will explore the rate limiting options (vSS and vDS with 4.0) in the next article and conclude with the QoS based options (vDS with 4.1 and Cisco 1000v).

Lastly, a big Thank You!! to the following people for their help on the article and for allowing me to bounce questions off them: Don Mann, Ron Fuller, Joe Onisick, Sean McGee, Brad Hedlund & Stevie Chambers

Do you any information to add?  What are your thoughts?  Please leave a comment!

Cisco UCS QoS vs. HP Flex-10 vNICs in VMware

This post will be more conceptual than technical.  I recently was asked how Cisco's UCS &  HP's Flex-10 network design approaches affect vSphere designs.  Even though the industry is moving towards a unified 10GB fabric, there are different ways to move data through this big "pipe" and still ensure/prioritize delivery.  As you would guess, Cisco and HP approach this problem very differently.  Cisco takes a network centric approach to the problem and HP takes a server centric approach to the problem.

HP's Flex-10

HP Flex-10 takes a 10GB connection and carves it up into multiple virtual NICs.  The size of the "pipes" can be turned up and down to match the amount of bandwidth needed for the NIC.  Think of it as placing smaller pipes in the big 10GB pipe.  This approach is great for vSphere admins because the virtual switches in vSphere can be configured to look just like they did with a bunch of 1GB links into the server.  The transition to this technology is seamless for the vSphere administrator.  I'll borrow a diagram from Barry's awesome article on Flex-10.  If you haven't read it, please do!

What is the down side to this method?

The down side to this approach is by placing multiple pipes within the larger pipes, you have now placed a CEILING on how much data can pass through that particular pipe.  Let's say you present a 1GB vNIC to vMotion and during a vMotion it would be to your advantage to have access to more bandwidth.  Too bad, 1GB is all you will ever get.

 Cisco UCS's QoS

Cisco UCS uses a method known as Quality of Service (QoS).  Most of us "server guys (and gals)" have no idea what this is.  Here is how I have come to understand it.  If this is wrong, please correct me.  Network traffic is given a priority and this priority kicks in WHEN THERE IS CONTENTION on the network.  So, instead of smaller pipes inside a large pipe, you have more of a priority system in place to guarantee certain levels of service.  Think of this as a FLOOR model.  You can have as much as you want as long as everyone else gets their minimums (they get their quality/guarantee of service).  If something needs to spike and there is room, it can spike and then return to normal.  Here is a diagram of our Cisco UCS with traditional switches.  This isn't 1000v but you get the idea.  As you can see, two big 10GB pipes into the virtual switches instead of smaller pipes into multiple virtual switches.

As the vSphere administrator, this looks very different from my old multiple 1GB links into my multiple virtual switches!

What is the down side to this method?

At this time, QoS for Cisco UCS appears complex to configure and represents a shift in thinking for the vSphere administrator. 

How is the QoS implemented for Cisco UCS and VMware?

That is a very good question.  I can't seem to find any documentation on how to actually do this yet.  I'm sure there is a Cisco internal doc somewhere but I haven't found anything public that lays out the hardware that is needed (do I need 1000v or Palo for this, can I use a CNA and the standard switches?) nor have I found a "cook book" that documents how to properly make QoS happen in a vSphere environment.  I'm sure this will happen in time and if you have a link, please leave a comment!

Which is better?

It depends on your point of view and the comfort level of your team.  I can easily see advantages to both approaches.  One is easier to implement, the other appears to be a more elegant (but complex) solution.  Cisco has once again brought a disruptive technology to the table that can't be ignored.  What are your thoughts?