Keeping the vMotion Tiger in the 10GB Cage - Update

There has been a lot of activity regarding my post from last Sunday.  Again, all credit goes out to Don Mann and the ePlus Engineering Team (Rob Quast in particular) for all their hard work surrounding Don's presentation at VMworld.  The title of the session was 10GB and FCoE Design Considerations and it was great!  If you get a chance to catch it on the VMworld replay, I highly suggest it.

I wrote the first post because even though vSphere 4.1 has been out for a few months, it is Don's session that set off the light bulb in my head that made me start asking questions.  I wanted to know from VMware if it was true that vMotion (and really any traffic that isn't controlled) can saturate a 10GB link with vSphere 4.1.  Is this a new design criteria that I now MUST consider?  I asked on the forums and it was confirmed by none other than Dilpreet himself.  Thank you again Dilpreet for taking the time to post!

In the meantime, Sean McGee and Brad Hedlund also wrote articles to further explain the concepts as well as lay out some architecture solutions.  Sean's post is here and Brad's post is here.  Please take the time to read them both, great stuff!

Due to some other things going on, I won't be posting the follow ups that I promised in Part One.  I'm very sorry about that and I hope to get to them someday but the circumstances right now just won't allow that to happen.  Besides, Sean and Brad did a great job (probably better!) than I could have.

Keeping the vMotion Tiger in the 10GB Cage - Part One

I had a light bulb moment as I was sitting in fellow ePlus employee Don Mann's session at VMworld.

A little background is needed first.  In vSphere version 4.0 we didn't really have a need to control the traffic in 10GB connections.  Even if all the traffic types were combined into a single connection with no traffic management, you rarely ran into contention on the link.  vMotion was the most likely to act up due to the "bursty" nature of the traffic pattern (hit the connection really hard for a few seconds until the vMotion is complete and then settle down) but this was limited because vMotion in vSphere 4.0 was capped at two concurrent vMotions at about 2.6 Gbps each for a total of 5(ish) Gbps maximum for vMotion.  If you assume a little over 9 Gbps usable capacity (the rest lost to protocol overhead) on a 10GB link you still have room for other traffic and you never burst high enough to saturate the network.

Then, along came vSphere 4.1....

vSphere 4.1 introduced significant performance enhancements to vMotion over 4.0.  vSphere 4.1 increases the number of concurrent vMotions to eight in a 10GB environment and the speed has been increased to 8 Gbps.


When I heard this, a light bulb went off in my head and I've been poking at this idea with a stick for awhile now.  I've asked around in the community over the last few days and there seems to be confusion over the numbers.  Does that mean eight vMotions, each one at 8Gbps for a total of 64 Gbps maximum or does that mean eight concurrent vMotions consuming a total of 8Gbps maximum.  I don't have a definitive answer to this question but tests I have seen conducted point to EACH vMotion consuming up to 8Gbps each.  If this is true, anything above ONE vMotion at a time without some form of traffic control may not be a good thing!

Does it matter if I'm utilizing 64 Gbps for vMotion or 8Gbps for vMotion??


The more I think about it, it really doesn't.  Let's assume best case for a second and say that eight vMotions will consume a total of 8Gbps (I don't think it works this way but I'm being an optimist).  If vMotion can consume a maximum of 8Gbps of a 10GB pipe, you will need to design around this fact.  Some form of Traffic Shaping and/or Quality of Service to manage the traffic will be necessary in 4.1 when it was often considered optional previously.

I did a little digging and the issue is confirmed in VMware's NetIOC Best Practices document.  To summarize, your results may vary (and not in a good way) if you aren't putting some form of control on your vMotion traffic in conjunction with 10GB links.

Oh, before I get a bunch of comments telling me this: I'm picking on vMotion here but you could just as easily perform a global replace in this article with (your favorite chatty and/or spikey traffic type) for vMotion in this article.  The concepts to solve network congestion are the same.

How do we solve this issue?

There are two main ways to solve bandwidth contention.  One is to place a cap on the amount of traffic vMotion can use.  This is often referred to as rate limiting the links.  The second is to give priority based on a weighted system that kicks in when contention takes place.  This is called Quality of Service or QoS.  With QoS, everyone gets some bandwidth, but no one is allowed to take over completely and priority is given to critical traffic.  I wrote an article on the concepts in the past here and Brad Hedlund wrote a great article on the concepts with cool Flash animations here.  Don't get hung up that we both wrote about HP and Cisco, the concept of rate limits vs QoS still stands.

In my opinion a QoS or shares based priority model is much more effective to control this traffic.  This allows for better utilization of the bandwidth and provides a more flexible alternative to rate limiting.

How do Rate Limits and QoS fit into vSphere?

Here is a simple graphic to illustrate the virtual switch options in vSphere today:

This concludes the first article in this series.  I will explore the rate limiting options (vSS and vDS with 4.0) in the next article and conclude with the QoS based options (vDS with 4.1 and Cisco 1000v).

Lastly, a big Thank You!! to the following people for their help on the article and for allowing me to bounce questions off them: Don Mann, Ron Fuller, Joe Onisick, Sean McGee, Brad Hedlund & Stevie Chambers

Do you any information to add?  What are your thoughts?  Please leave a comment!

VMworld - Vblock Overview Session

I was able to attend the VCE Coalition's super-session at VMworld and I was very impressed by the content!  The title of the session was (SS1011) How Cisco, EMC, and VMware are Changing the IT Landscape with Vblock Converged Infrastructure (whew! that's a mouthful!).  The presentation was given by Phil Harris, VP of Engineering at Acadia and Chad Sakac from EMC ran the demonstrations.

It's no secret that I'm a fan of stack infrastructures.  I firmly believe that by helping customers remove some of the installation and configuration (nerd knobs) and creating a solution that is certified and proven to run your customers' applications, you create a new way to serve your customers in today's evolving industry.

Phil went over some bullet points that address why Vblock was brought to market:

• It is estimated that 28% of all IT projects are non-recurring projects
• Of this 28%, a large percentage (he said, I didn't catch the number) are actually never completed
• This means that money is simply lost to IT infrastructure costs
• We need a way to accelerate the way products and solutions are brought to market to minimize the costs and complexity associated with delivering applications to our customers
• The VCE Coalition's goal of Vblock is to provide pre-integrated and validated solutions with a guaranteed service level
• If an application is already certified on a Vblock, the configuration burden is removed from the customer and they can now more rapidly deploy their applications
• Vblock allows for standardization - It is predictable, this allows for many environmental factors (power, space, cooling, fault isolation, etc) to be predicted ahead of time to assist in growth planning

Next Phil addressed the security and management model.  To best describe Vblock's model, Phil used the term "built-in vs. bolt-on".  Many "stacks" today are bolt-on.  Other vendors have taken existing off the shelf products and "bolted" them onto their hardware stacks.  Phil contends that the Vblock management and security is "built-in".  He used a couple of examples including Cisco's UCS RBAC was designed from the start with secure multi-tenancy in mind and the UIM software that is used to provision and carve up a Vblock.

Vblock is designed to integrate seamlessly into an existing environment.  In the industry we use the terms "green field" and "brown field" a lot.  A green field is an installation of new hardware into a clean, new environment.  A brown field is installing and integrating hardware into an existing environment and integrating the new components into the existing.  Vblock is a hybrid model; it is "green field in a brown field".  I've never thought about that before but it is very fitting.  It is a self contained solution (green field) that is designed to plug right into a customer's existing data center (brown field).  Good Stuff.

From this section we moved into Chad's demonstrations and customer references.  Chad did a great job as always and Phil went over a Service Provider (CSC Australia) as well as an Enterprise (Alcon) customer and how they have integrated Vblocks into their environments.

Next up were the future roadmaps.  I'm not sure what I can and can't say here so I'm just going to say it appears that all three companies will be working in lock step to revise the Vblock architecture going forward.  There was certainly a lot more happening on the roadmap than I thought and this appears to be a long term commitment from all three organizations.

The next section was the section I have been waiting for.  Phil introduced how VMware's vCloud Director will be integrated into the Vblock.  By optionally adding vCloud Director, a Vblock now provides two major abstraction points.  The first abstraction point is the hardware with the UIM software; the second abstraction point is now the virtual machines with vCloud Director.  I have been waiting for this to happen and I'm very happy to see them embrace this.  By abstracting both the hardware and virtual machine layers we achieve a simplification goal by removing some configuration complexity from the end user.

The last section was about the scaling of applications within, and by adding, multiple Vblocks.  A UC on UCS architecture was presented that displayed 15,000 IP phones and an Exchange infrastructure on a single Vblock One.  Lastly, application performance testing was demonstrated utilizing first 8, then 16, then 32 servers to process a workload to demonstrate elasticity in the data center.

All in all, a very impressive session!

VMworld - vCloud Director Technical Overview Session

I'm playing a little catch up this week.  Sorry for the delay in getting my session write up's posted.  As usual, I took notes as fast as I could but I may be missing a few things and this post may contain some slight inaccuracies here and there.

The vCloud Director Technical Overview session was probably one of my favorite sessions at VMworld.  They did a very nice job staying low level to explain everything while also keeping it interesting at the same time.  The session started off with an overview of the product.  There have been many other good articles already published on this subject so I'll it.

Network Overview

 The first topic was a networking overview.  There are two basic types of networks that can be created in vCloud Director(vCD), External (shared) networks and Internal (pools) networks.

An External network communicates with the outside world and can be shared with other resources.  This type of network is attached to a port group on a virtual switch and isolation occurs through the use of Layer 2 VLAN's similar to regular virtual machine port groups today.  External networks are creating by the cloud administrator, they can't be self provisioned.

An Internal network is where things get interesting.  An internal network isolates a vApp (group of machines) within an organization from other vApps as well as other organizations.  Internal networks are created by the users based on pre-defined values that are entered into pools by the cloud administrator.  Internal networks will be self provisioned.  There are three types of internal networks: 1. portgroup based 2. VLAN based & 3. vCD Network

1. Portgroup Based Pool - The port groups are created ahead of time and "handed out" to vApps as needed.   The cloud administrator must create all necessary port groups ahead of time and they will attached as needed to the virtual switches
2. VLAN Based Pool - This pool type allows for creation and isolation based on VLANs.  The cloud admin would create a number of VLANs ahead of time and each vApp would grab a VLAN from the pool upon check out.
3. vCloud Director Based Pool - This type of network is very similar to Lab Manager's fencing.  A virtual machine (in this instance it is the new vShield Edge product) will sit in between the vApp and the outside world and perform Mac in Mac based isolation, NAT, DHCP, etc to communicate with the outside world.  This feature is limited to one VLAN per vApp at this time.  vCD based networks allow for Internal (fenced & isolated from the outside world) and External Routed (fenced but NOT isolated) networks.

Organization Overview 

Up next we had Organizations.  An organization in vCD is basically a "tenant" or a group that you would like to manage and isolate from others.  Each organization will have a unique web address created for them to use the vCD default GUI.  There are three ways to authenticate against an organization: 1. vCD Local Database (not recommended) 2. Global LDAP & 3. Per Organization LDAP.  I really like the ability of vCD to tie into an LDAP per organization; this will provide multiple tenants from different organizations an easy integration point.

Leases and Quotas

There are two types of leases and quotas in the vCD product, time based and storage based.  The concept of a lease allows users to check out vApps for a specified amount of time and then renew the lease as needed.  The user may also allow the lease to expire and the resources will return to the pool.  Resources can be configured to be suspended and held for an amount of time to make sure the lease expiration was intended.  A good example of this is a user going on vacation during the lease expiration.  The user could not renew the resources and would be very upset if they were destroyed.  This mechanism would prevent that situation.

Quotas are exactly as they sound.  A user is prevented from creating too much sprawl within an organization by checking out and consuming resources they may not need.  Only a certain amount may be checked out at a given time or only a certain amount of storage may be provisioned.

The idea of quotas brought up a number of questions around approvals and provisioning of vApps in the vCD product.  Currently, there is no approval mechanism built into the vCloud Director product but it MAY be incorporated into a future release.  What this means is that if a user has access to provision, there is nothing stopping them today except leases and quotas.

Allocation Models 

There are three types of allocation models built into vCD: 1. pay as you go (no over commit) 2. Allocation Pool (cloud admin over commits resources) & 3. Reservation Pool (organization admin over commits resources)

The use of the Allocation Pool and Reservation Pool models allow you to "thin provision your data center".  You can over commit your resources at either the Cloud or Organization levels based on your desire.  As with anything thin provisioned, use caution, understand the design, carefully monitor the environment, and generate alerts when trouble starts.

Organization Networks

As with everything else in this post, there are three types of Organization Networks.  An Organization Network is a template that is defined at the Organization level and will be associated with a vApp configuration:

1. Internal - An internal network utilizes the vShield Edge device to both isolate and fence the vApp group of machines from the outside world.  There is no external communication in this configuration.
2. External Routed - This network type also utilizes the vShield Edge device but allows communication to the outside world through NAT on the Edge device.
3. External Direct Connect - The vShield Edge device isn't utilized and the vApp group is connected directly to a Port Group on a virtual switch.

Conclusion

That about does it for this post.  I could write more but this is getting long and I'm sure you're sick of reading by now.  I will have more vCloud Director posts in the very near future.  As always, thanks for coming by!

VMworld Keynote - Announcing VMware vCloud Director

As I write this the VMworld 2010 Keynote is about to kick off.  Paul Maritz (VMware CEO) & Dr. Stephen Herrod (CTO) will be speaking about VMware's new Cloud Computing product, vCloud Director.  This is a live blog so it will be quick and dirty, sorry for the lack of formatting ahead of time.

• Theme - Virtual Roads, Actual Clouds
• Most customers are on a 3 phase journey
1. IT Production - getting your production systems into a virtualized environment\
2. Business Production - Quality Production, many users find their apps run better in a virtualized environment using VMware tools to provide greater access, performance, and uptime to their applications
3. IT as a Service (ITaaS) - Business value is provided by automating the business process with industry standards
• ITaaS = Optimizing IT production for business consumption
• First 2 phases were optimizing production of IT Services, 3rd phase is optimizing business consumption of IT Services
• Paul Maritz is introduced
• We are at a tipping point - the number of hosts deployed on virtual has now topped the number of physical hosts for the first time
• In 2010 over 10 million virtual machines will be deployed
• Paul mentioned Tasty Kake as a customer (I'm from Pennsylvania and they are the best snack food ever!), just wanted to add that
• Innovation comes through reduction of OpEx using Automation and Management while maintaining security as resources are consolidated
• VMware believes the future is around providing access to applications in a fashion that will be portable and independent (Secure Hybrid Cloud)
• Infrastructure is a means to an end (providing applications)
• New Enterprise Applications will be written on new application platforms using new open frameworks and tools that will allow migration to the cloud
• Operating System will have a changing role - used to talk to hardware and abstract it out for applications - that is changing because OS is on common virtual hardware already
• Talking about "non-Windows" based devices like smart phones and iPad like devies, the base OS is now changing which further increases the need for an independent way to deliver applications
• "The New Stack" - New, more efficient way to deliver applications
• Steve Herrod is introduced
• It's now all about the applications and no longer about the virtual machines
• The New Stack needs to provide efficient pooling, elastic resource scheduling, automation through policy, while still being open and inter operable
• Steve is discussing the increased capabilities of vSphere 4.1
• Managing The New Stack (Steve referfed to this as the Virtual Giant)
• VMware acquired Integrien for proactive analytics and monitoring
• A few screenshots were put up, looks a lot like a vFoglight dashoard
• What matter most to customers is the applications they use, not the resources they run on
• vCloud Director announced
• I just hit publish on my vCloud Director article so I missed a bit about the vShield product line, more on that later this week (vShield Edge, App, and Endpoint)
• demo of vCloud Director on stage
• Secure vCloud Services - They are showing the network isolation features using vShield Edge
• Showing off levels of service (Gold, Silver, Bronze)
• Eventual Goal of cloud computing VMware vFabric to enable cloud resources to become portable across private and public cloud platforms
• How are we going to get there?
• Modernize the Desktop Experience
• VMware View 4.5 Announced
• Local Desktop experience
• Windows 7 Support
• Mac Support
• vSphere 4.1 scalability
• Reduce both CapEx and OpEx costs
• Unify Application Management
• Project Horizon Announced
• Central Application Provisioning (based on ThinApp)
• A user is entitled to applications on their virtual desktop from a central location
• Showing off moving from a desktop to an iPad View Client

 

vCloud Director Security Model

A few moments ago VMware announced their new Cloud Computing product, vCloud Director.  Yesterday I was able to attend a session dedicated to the security model of this product.  Vishal Kumar from VMware presented the concepts and did an excellent job.  This post is a bit of a brain dump based on how fast I could write down my notes so I may have missed a few things, my apologies for that.

If you are familiar with the design concepts of VMware's Lab Manager, think of vCloud Director as Lab Manager on steroids for production environments.  If you know Lab Manager, you will feel right at home in this product.  vCloud Director is designed from the ground up with secure multi-tenancy in mind.  The environment is achieved by providing a layer of abstraction above the existing VMware Virtual Center and vSphere resources.

Before I go any further, some definitions are needed:

Organization - This is essentially a tenant or a consumer of resources.  This could be different customer, departments, or any other group of people and resources that you would like to present computing capacity


Provider Virtual Data Center (vDC) - An aggregate pool of vSphere resources that represents the entire amount of capacity.  Think of this as an entire pie

Organization vDC - A subset of resources from the Provider vDC that are assigned to a specific organization.  Think of this as a slice (or slices from the pie)

vApps - A vApp is a group of virtual machines that represent a pre-configured grouping that can be delivered to a vCloud Organization.  This is what the Organization users will see.

Cell or Pod - An underlying group of Virtual Center servers and vSphere servers. vCloud Director is designed to scale horizontally by adding more Pods over time as needed.  There is currently a limit of 25 vCenters.

Based on all of this, you will have a nested structure that looks something like this:

As you can see, vCloud Director removes the VMware resources and provides objects that can be assigned to Organizations and users in a way that allows them to "help themselves" while at the same time isolates them from each other.

There are two ways to access vCloud Director as a consumer.  One is to use the built in GUI, the other is through the vCloud API.  I haven't seen the API in depth yet but I'm told it can serve as a complete replacement to the GUI for task automation or it can also provide an alternate GUI specific to your needs.

The security model of vCloud Director is based on a Role Based Access Control (RBAC) model.  RBAC roles and permissions can be assigned at both the top level (System or Cloud level) as well as the Organization level.  vCloud Director is also able to integrate with multiple LDAP directories at once in case your organization or customers use different LDAP resources.

Network security and isolation is very complex in this product and I'm actually attending sessions later in the week to get more information in depth.  I will present this as another post later in the week but here is a quick summary.  Network resources can be isolated through the concept of "fencing".  There is also a firewall product built-in called the vShield Edge device.  This device is a virtual machine and allows vCloud Director to provide both NAT and DHCP capabilities to vApps.  Syslog server support is also included in the product.

In summary, vCloud Director is a fully multi-tenant middleware with charge back, billing models, and service tiers integrated into the product.  I'm very excited to see what the future of vCloud Director holds!

Thank You VMware for the VMworld 2010 Fun Run!

Yesterday I had the privilege of participating in the VMworld 2010 Fun Run.  This year's run was near the Golden Gate Bridge.  The course was a 5k out and back course and the turnout was fantastic.  It wasn't a race, just a fun run (with beer at the finish!!).

A few of us decided to take the 5k course and turn it up a little bit; we decided to conquer the bridge (out and back).  I will say this in hindsight; the bridge is a LOT longer than it looks!  I ended up running about 7.2 miles all total and took lots of pictures along the way.

A big THANK YOU to VMware for the event!!  For those of you that think picture or it didn't happen, here are some pictures from the far side of the bridge as well as other pictures taken throughout the run.

My VMworld Survival Kit

I'm very sorry for the lack of posts recently.  Summer, time with the family, secret projects I can't talk about yet, etc.

I have assembled a nice little survival kit for VMworld.  I won't be taking my laptop on the floor because it weighs too much and the battery life on it is crap.  It would be dead by lunchtime.  So, here is what I'll be taking in my bag.

1. Palm Pre Plus - Main reason I bought this phone, FREE 3G Mobile Hotspot, love it.  I'll never own another phone without this feature
2. iPod Touch - It is an older 2G iPod but it still works great
3. New Trent IMP-500 Charging Pack - Plenty of juice to keep the Palm and iPod going!  This arrives in the mail today so I'm hoping it works out
4. Panasonic Lumix DMC-ZS3 - Best camera I've ever owned, period.  12x Zoom, 10 MegaPixel, and it takes INCREDIBLE HD Video with stereo sound, all in one device. I have an extra battery and a 32GB SD Card for it to hold the large files that it generates
5. A plain old paper notebook (*gasp*) - The iPad Fairy hasn't made it to my house yet so old fashion pen and paper for me.

I'll be heading to VMworld on Sunday in time for the 5k run and the events Sunday evening.  I haven't even had a chance to look at my schedule; I might post that once I figure it out.