Creating a VMware Virtual (Home) Lab for VMware VCP & VCAP - Step Three: HA Funny Stuff

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In this third part I discuss some of the quirks of VMware HA I encountered.

I'm finally getting around to picking up my virtual lab again.  Due to work commitments I really haven't had any time to dedicate to the lab recently.  Right after I finished the previous article, I couldn't get HA to configure.  I created a cluster but if I added the hosts and tried to enable HA, it would fail.

The first error was around memory allocation...  uh oh.  Looks like the memory tweaks in my previous article don't hold up if you want to enable HA.  I bumped up the memory to 2GB and the memory allocation error went away but I still had a super generic error message: HA agent on in cluster in has an error: Cannot complete the HA configuration

Great... All the usual tricks didn't work (remove the host, add it back, enable HA again, double check DNS, etc). Finally, I bumped the memory to 2.5 GB, still nothing. As a last resort I reinstalled and made the local disk 10GB instead of 5GB (and kept the memory at 2.5 GB just because I can).

It worked! I reinstalled the second host and it also worked the first time. Due to the overall quirky nature of VMware HA (HA and I have a long history together and it's not all good) I can't say for sure if either the memory or disk change did the trick but I thought I'd pass it along.

Creating a VMware Virtual (Home) Lab for VMware VCP & VCAP - Step Two: Install AD/DNS & ESXi

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In the third part I discuss some of the quirks of VMware HA I encountered.

Step One: Install AD & DNS
Nothing sexy here.  I created a w2k3 R2 standard edition virtual machine and then did a dcpromo to make it both my Active Directory and DNS server.  My domain is called VIRTUALLAB with a DNS name of virtuallab.local

Step Two: Install ESXi
I covered this in my post on how to install ESXi in Fusion.

Step Three: Configure ESXi
I will only cover this for the first server but I completed the steps for ESXi2 as well.

We first want to configure the network on the ESXi host.  Hit F2 from the ESXi splash screen and then choose Configure Network Management:

Select IP Configuration and you will see the following screen.  Modify the IP address to static and enter the IP of the host as well as the default gateway (it will always be XXX.XXX.XXX.2 in Fusion):

Return to the previous screen and select DNS Configuration.  Enter the DNS Server and the hostname:

Return to the previous screen.  I'm not 100% sure if it is needed, but I always add the DNS suffix as well.  Better safe than sorry. Choose Custom DNS Suffixes and enter the DNS entry:

Step Four: Prepare the ESXi host & join the Active Directory
Using the VI Client, connect to the ESXi host and point NTP at the AD server.  This step is needed to keep Kerberos in sync:

Create a Group an ESX Admins group in Active Directory (it must be EXACTLY like that).  Thanks to Maish and Brian for information on this.  Create the proper users and populate the group.  Once this is complete, insert the ESXi host into the domain:

Because you already created the ESX Admins group, the permissions are set for you automagically!

Step Five: Configure DNS
Create an A record in DNS (I also add a reverse look up entry):

Step Six: Test
You should now be able to log into the server using your Active Directory credentials:

Next up, Creating the Virtual Center Server

Creating a Virtual (Home) Lab for VMware VCP & VCAP - Step One: Design

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In the third part I discuss some of the quirks of VMware HA I encountered.

A few months ago I attempted to create a Virtual Lab based on my laptop but due to work and life craziness, the project fizzled.  It's now time to pick it back up and provide some clarity and direction to the project.  Consider this a reboot.

First off, what am I trying to do?  I wanted a lab to help study for VCAP but this could easily be used for VCP as well.  I decided not to purchase dedicated servers and storage because I really don't want all that gear at my house.  I manage enough hardware on a daily basis, no need to add to that.

What hardware do you need?  I recommend the maximum amount of memory your PC will support as well as an SSD based disk to handle the IOP's the virtual machines will create.  I've tried this on a regular HD in the past, trust me, it's very painful.  My MacBook Pro now has 8GB and a 240GB SSD.  Details on that are here.

What can you do in the lab?  The goal of this lab isn't to provide an environment to run a bunch of virtual machines.  Because I'll be running ESXi inside Fusion (Player and Workstation work great as well), that really isn't the intent here.  The focus is on Virtual Center and ESXi, not the virtual machines.

What other benefits are there to the virtual lab?  Two other major benefits are that the lab is always with me since it is on my laptop and it is much more cost effective (8GB RAM and SSD = $500-700 vs. Dedicated Home Lab = $2k+)

What will the project look like?  Here is a quick graphic of what I hope to be the end result.  I will have two ESXi hosts, a Virtual Center server, an AD, DNS & SMTP relay, and lastly a storage VSA for iSCSI and NFS mounts.  For details on how I "trick" the Fusion/Player/Workstation networking by using static IP's instead of DHCP, see this article.  That's it for today, next time we'll install ESXi!

Installing and Tweaking ESXi for VMware Fusion

This article is part of my Virtual Lab Series.

NOTE: If you want to enable VMware HA on your cluster, you might want to read this article first.  I *THINK* I had to bump up the memory to 2.5GB and the disk to 10GB.  Your results may vary!

Now that the SSD is installed in my laptop, it's time to throw some virtual machines at it!  I'm going to start by installing and tweaking ESXi in preparation for creating linked clones of this template to run my virtual vSphere Cluster.  If you've installed ESXi in a virtual machine before, skip ahead to the memory tweaks section; it has changed a bit with ESXi 4.1 as you'll see below.

Start by creating the virtual machine in Fusion and choosing VMware ESX as the OS:

Before clicking Finish, click Customize Settings:

Change the size of the Hard Disk from the default of 40GB to 5GB:

The rest of the install is a standard ESXi install.  The only issue I had was some keyboard "funkiness".  Some Function keys required me to hold down the Command key in addition (F11 was Command-F11 for instance).  I haven't assigned a machine name or static IP address yet because I'm going to create Linked Clones in Fusion so I will personalize the installs after the split.  After install, ESXi only consumed about 2.4 GB of disk space on the SSD.

The last step is to tweak the memory consumption of ESXi.  As stated in Duncan's original article and my previous post on setting this up in VMware Player, you can "trick" ESXi into using less than 2GB of memory by removing the memory check.  We have to perform a few additional steps is ESXi 4.1 because Tech Support Mode is disabled by default as shown below (Alt-F1 to see this and Alt-F2 to flip back to the ESXi screen)

Enable Local Tech Support Mode under the Troubleshooting Options menu:

With Local Tech Support enabled, we can now log in to the console on the ESXi virtual machine (remember Alt-F1 to flip to the Tech Support screen).  Log in as root:

As per Duncan's article, edit the /etc/vmware/esx.conf and add the line:

/vmkernel/minMemoryCheck = “false”

Once the entry is in place, shut down the server and set the memory to 1700 MB.  In ESXi 4.0 we could set it as low as 1200 MB but ESXi 4.1 seems to require a bit more.  What happens when you set the memory too low you ask?  The server will PSOD on boot like this:

Once all of this is complete, you should have a bootable ESXi virtual macine in a minimal footprint (less than 3GB of SSD and 1.7 GB of memory) and this will serve as a nice base configuration for our linked clones in the next article.

Care and Feeding (TRIM Support) of SSD's in Apple's Mac OSX

This article is part of my Virtual Lab Series.

As a follow up to yesterday's article on installing an SSD in my Mac, the next question is now what?

Before I start loading virtual machines for my virtual lab I wanted to relay some information on TRIM support and Mac OSX.  First, what is TRIM?  According to the Wikipedia TRIM entry:

In computing, a TRIM command allows an operating system to inform a solid-state drive which blocks of data are no longer considered in use and can be wiped internally.

TRIM was introduced soon after SSDs started to become an affordable alternative for traditional hard disks as permanent storage in PCs. Because low-level operation of SSDs differs significantly from traditional hard disks (see details below), the typical way in which operating systems handle operations like deletes and formats (not explicitly communicating the involved sectors/pages to the underlying storage medium) resulted in unanticipated progressive performance degradation of write operations on SSDs.^[1] TRIM enables the SSD to handle garbage collection overhead, that would otherwise significantly slow down future write operations to the involved blocks, in advance.^[2]

Although tools to "reset" some drives to a fresh state were already available before the introduction of TRIM, they also delete all data on the drive, which makes them impractical to use for ongoing optimization.^[3] More recent SSDs will often contain internal idle/background garbage collection mechanisms that work independently of TRIM; although this successfully maintains their performance even under operating systems that do not (yet) support TRIM, it has the drawback of increased write amplification and associated increased wear of the flash cells.^[4]

Why is this a big deal?  Simply put, without TRIM, your SSD performance will degrade over time.  The problem here is Mac OSX doesn't support it!  Most operating systems these days support TRIM but old Steve Jobs doesn't consider this a high priority right now (even though he's pushing SSD's in just about every model they sell - trying to keep the Genius Bar fees up maybe?).  As they would say in Princess Bride, "Inconceivable!"

I have found two articles very useful to maintain the performance of the SSD overtime.  In a nutshell, right now you have to wipe the drive every so often and perform a manual TRIM.  Although this isn't ideal, it isn't a large issue considering how easy it is to reinstall (or clone) the SSD on the Mac compared to Windows.  Here are the articles:

• The awesome Jason Nash's blog article
• This MacRumors Thread (Long but worth the read)  (Thank you to Jason from yesterday's article for leaving a comment with this link!)

That's what I have found so far.  Am I missing anything?  If so, please leave a comment!

Installing a MacBook Pro SSD for a VMware Virtual Lab

This article is part of my Virtual Lab Series.

I decided recently to upgrade the hard disk in my MacBook Pro to a Solid State Disk (SSD) so I can run a "Data Center in a Box", much like the previous one I created before.  This post will cover the installation of the SSD and subsequent articles will cover the set up and configuration of my new DC in a box.  As you will see in this post, I know just enough about the Mac platform to be really dangerous!

To create this environment I purchased two items, an OCZ Vertex 2 240GB SSD and a RocketFish 2.5" SATA HDD Enclosure.  The reason for the external enclosure is twofold; first to clone the SSD from the HDD and the second to swap them long term and use the SSD for boot and the existing HDD for "Tier Two Storage" as needed.  Here's a photo of the parts and the SSD in the case:

 

Step One: Install the SSD in the RocketFish Enclosure (pictured above right).

Step Two: Plug the SSD in and format (Erase) the SSD drive using the Apple Disk Utility.

Step Three: Clone the Internal HDD to the External SSD.  For this step I picked Carbon Copy Cloner.  It is very easy to use, just pick the source and the destination and click clone.  As you can see it took me 2.5 hours for this to clone 142 GB.

Step Four: Swap the internal and external drives.  This step requires a small phillips screwdriver and a Torx T6 screwdriver.  I followed this video from Other World Computing:

Step Five: I swapped the drives and powered on the laptop and got the following.  My first thought, this isn't good!!

Here is where I learned a little about about Mac OSX. Unlike Windows, the system was still looking for the old internal hard disk.  So, I plugged in the old HDD into the USB and the system booted again!  From there I went into System Preferences and changed the SSD to be the boot drive.  One reboot and I was back in business!

Lastly, I ran Xbench to see the before (top) and after (bottom) results.  So far, I'm loving it and will post more as I get the lab setup.

UPDATE: Make sure you read my follow up article on how to maintain the SSD performance over time.