My #1 Issue with VMware ESXi Today

It is no secret that VMware has anointed ESXi the future hypervisor of choice.  I am often asked what I think of ESXi and if I think it is functionally equivalent to ESX.  My short answer is NO, not today.  My opinion is due to one key reason, realignment of vmdk's on VMFS datastores today.  If you don't know what I mean, you need to understand the concepts below.

What is the difference between vmdk alignment and realignment and why should you care?

The alignment of vmdk's is the big white (or pink if you prefer) elephant in the room from a performance standpoint.  It is often ignored by many customers and misalignment can lead to as much as a 30% degradation in performance!!  To their credit, both NetApp and EMC have recognized this and and released documentation to address the issue.  Here are links to VMware & NetApp papers on the subject.  EMC documents H2197, H2529 & H5536 on PowerLink also provide information on the subject.  EVERY storage vendor has this problem; EMC and NetApp are the only two I know of that have spoken about it.  If aren't aligning/realigning your virtual machines as part of standard best practices, you should be!

What is the difference between alignment and realignment?

As pointed out to me by Duncan Eppping in a Twitter conversation, there is a big difference between alignment and realignment.  Alignment happens when the partitions are created but before the OS is installed.  Let's be honest, it is a pain in the butt and many don't learn about alignment issues until they start to have problems.  Because it is a pain, many choose realignment to remedy the problem.  Realignment is the concept of using tools (vOptimizer and NetApp's mbrscan/mbalign come to mind) to align an existing vmdk that is misaligned.  This process requires taking the machine offline and then rewriting the entire vmdk in an aligned format.  I'm speaking about realignment in this article.

Are all virtual machine operating systems affected?

NO - Windows 2003 and earlier are misaligned by default but Microsoft changed the alignment of Windows 2008 so it is now aligned by default.  Be careful though, Windows Dynamic Disks, Cirtix Servers, and many other special cases often won't work. See my article on the NetApp tools for more information.  I'm not an expert on the Linux vm's but if you have experience, please leave a comment!  Lastly, if you align your template, all deployments from that template will also be aligned.  This is really nice IF you remember to align your template before installation of the OS and before you deploy all your machines!

What does this have to do with ESXi?

Both vOptimizer and NetApp mbrscan/mbralign utilize the service console so ESX works just fine.  Due to the lack of a service console, ESXi is a different story.  With NFS datastores, you can mount a Linux host to the datastore and perform the alignments using the NetApp mbrscan/mbralign.  This process is documented by Nick Triantos here.  I'm not sure if you can use vOptmizer in this way.  VMFS is another story.  Currently, there doesn't appear to be a clear method to realign vmdk's.  No service console to run the utility and no way to access the VMFS outside of ESXi.  What do you do if your storage vendor doesn't support NFS?  What would you do if your storage vendor is VMFS (LUN based) only?  You have problems my friend.

I did a poll on Twitter yesterday and the results confirmed the same findings as my customer base.  Many still use win2k03 based servers and most P2V's are Windows 2003 server and earlier.  In addition, ESXi is trending to the market faster than windows 2008 virtual machines yet many have no idea about the concept of vmdk alignment/realignment.

What are the work-arounds?

• Create a linux host and attach it to an NFS datastore and run NetApp mbrscan/align from there (not sure if that will work on vOptimizer)
• Set up a single ESX host to perform the alignments on VMFS datastores
• Align your Windows 2003 template and deploy aligned machines on VMFS and P2V all machines to NFS

When/How could this be fixed (I'm guessing here, I have no inside knowledge of any vendor products)?

• ESXi includes the ability to detect and realign vmdk's
• Storage Vendors include the ability to detect and realign vmdk's
• Incoming P2V's of older OS's are aligned in transit
• Windows Server 2008 Server becomes the standard 

Bottom Line - Can you get around the issue?  Yes you can.  Is it a pain?  Yes it is.  Until this is fixed I will continue to recommend ESXi with NFS environments but I will continue to resist ESXi with VMFS unless the customer fully understands the ramifications of this issue.  I look forward to using ESXi as my standard and we get closer and closer everyday but we also need to be careful.

NetApp mbrscan and mbralign for Virtual Machine Alignment In-Depth

Alignment of VMware virtual machines has been an issue for quite some time. This issue exists no matter who is the storage vendor is but I will use NetApp because it is what I know. Here are some links to get you up to speed in case you don't fully understand the situation:

Link to VMWare document on alignment
Link to NetApp document on alignment

What has NetApp done about the situation? I hear that NetApp will be releasing a tool that plugs into vCenter specifically for vSphere 4.0 and vSpherei (ESXi) 4.0 shortly. In the meantime, if you are still on ESX 3.5 (or vSphere with a Service Console) there is another answer that you can use today. This will not work on ESXi since you need a service console for the tools.

Eric Forgette at NetApp created a set of tools about a year ago that has matured and found its way into the NetApp VMware Host Utilities Kit version 5.1. If you do not have this loaded on your ESX/vSphere servers and you are connecting to NetApp, go load it NOW! (A reboot is required for the settings to take affect) Most of the following is research I have conducted in conjunction with conversations directly with Eric over at this thread on NetApp Communities.

Included in this tool are two executables, mbrscan (scans a vmdk for alignment) and mbralign (performs the alignment). The default installation location is /opt/netapp/santools.

While the readme does a good job of going over the basics, there are a number of caveats to run the tools correctly. I will go into each executable in depth but before I go down into the weeds, you need to know when NOT to run it!

You can not or do not want to use the alignment tool for the following:

• Windows 2008 Server is aligned if the machine was created as a Windows 2008 server. If the machine is upgraded from Windows Server 20003, it will not be aligned.
• Citrix Servers are not supported because they remap the c:\
• Windows Dynamic Disks are not supported and will be corrupted if an alignment is performed (but mbrscan will detect them - see below)
  
• Linux LVM volumes are not supported (mbrscan may NOT detect all LVM partitions)
• Windows Server 2003 non-boot disks that have been added (d:, e:, etc) will need to be remapped in Computer Management. The drive letter will be lost on alignment.
• GRUB booted Linux and Solaris will need to have GRUB reinstalled after alignment

With all that out of the way, it is a basic two step process: 1. run mbrscan, 2. run mbralign on machines as needed.

Step #1 - mbrscan

• In order for the mbrscan to give reliable results, the machine must either be powered off or have a VMware snapshot!
• I have a simple script that I put together that just takes a VMware snapshot on all machines on an ESX/vSphere host.
• I then execute mbrscan using the scan all virtual machines parameter: mbrscan --all
• After I have the scan results I need I execute another script to remove the VMware snapshots I just created for all machines on the host
• NOTE: Windows Dynamic Disks will report a partition type of: unknown - 0x42. Do Not Align These Partitions!

Step #2 - mbralign

• In order to perform an alignment, ALL VMware snapshots MUST be removed!
• In order to perform the alignment, you NEED an amount of free space equal to the size of the vmdk. mbralign will make a backup of the file as the first step. This file have an extension: -mbralign-backup.
• In order to perform the alignment, the virtual machine MUST be powered off!
• If the virtual machine has multiple vmdk's, only one can be aligned at a time! 
  
• Execute mbralign against the vmdk - I usually get about 1-2GB per minute speed
  
• Boot up the virtual machine. If it works, delete the -mbralign-backup file
• On Windows systems it will ask you to reboot one more time because it detects the hard disks as new hard disks
  
• If it doesn't work, run mbralign again and it will detect the -mbralign-backup and ask you if you would like to restore the file. Very Nice!