Creating a VMware Virtual (Home) Lab for VMware VCP & VCAP - Step Three: HA Funny Stuff

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In this third part I discuss some of the quirks of VMware HA I encountered.

I'm finally getting around to picking up my virtual lab again.  Due to work commitments I really haven't had any time to dedicate to the lab recently.  Right after I finished the previous article, I couldn't get HA to configure.  I created a cluster but if I added the hosts and tried to enable HA, it would fail.

The first error was around memory allocation...  uh oh.  Looks like the memory tweaks in my previous article don't hold up if you want to enable HA.  I bumped up the memory to 2GB and the memory allocation error went away but I still had a super generic error message: HA agent on in cluster in has an error: Cannot complete the HA configuration

Great... All the usual tricks didn't work (remove the host, add it back, enable HA again, double check DNS, etc). Finally, I bumped the memory to 2.5 GB, still nothing. As a last resort I reinstalled and made the local disk 10GB instead of 5GB (and kept the memory at 2.5 GB just because I can).

It worked! I reinstalled the second host and it also worked the first time. Due to the overall quirky nature of VMware HA (HA and I have a long history together and it's not all good) I can't say for sure if either the memory or disk change did the trick but I thought I'd pass it along.

Creating a VMware Virtual (Home) Lab for VMware VCP & VCAP - Step Two: Install AD/DNS & ESXi

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In the third part I discuss some of the quirks of VMware HA I encountered.

Step One: Install AD & DNS
Nothing sexy here.  I created a w2k3 R2 standard edition virtual machine and then did a dcpromo to make it both my Active Directory and DNS server.  My domain is called VIRTUALLAB with a DNS name of virtuallab.local

Step Two: Install ESXi
I covered this in my post on how to install ESXi in Fusion.

Step Three: Configure ESXi
I will only cover this for the first server but I completed the steps for ESXi2 as well.

We first want to configure the network on the ESXi host.  Hit F2 from the ESXi splash screen and then choose Configure Network Management:

Select IP Configuration and you will see the following screen.  Modify the IP address to static and enter the IP of the host as well as the default gateway (it will always be XXX.XXX.XXX.2 in Fusion):

Return to the previous screen and select DNS Configuration.  Enter the DNS Server and the hostname:

Return to the previous screen.  I'm not 100% sure if it is needed, but I always add the DNS suffix as well.  Better safe than sorry. Choose Custom DNS Suffixes and enter the DNS entry:

Step Four: Prepare the ESXi host & join the Active Directory
Using the VI Client, connect to the ESXi host and point NTP at the AD server.  This step is needed to keep Kerberos in sync:

Create a Group an ESX Admins group in Active Directory (it must be EXACTLY like that).  Thanks to Maish and Brian for information on this.  Create the proper users and populate the group.  Once this is complete, insert the ESXi host into the domain:

Because you already created the ESX Admins group, the permissions are set for you automagically!

Step Five: Configure DNS
Create an A record in DNS (I also add a reverse look up entry):

Step Six: Test
You should now be able to log into the server using your Active Directory credentials:

Next up, Creating the Virtual Center Server

Creating a Virtual (Home) Lab for VMware VCP & VCAP - Step One: Design

This article is part of my Virtual Lab Series.

In the first part of this series I discussed the over all design of this lab.

In the second part of this series I discussed how I installed ESXi.

In the third part I discuss some of the quirks of VMware HA I encountered.

A few months ago I attempted to create a Virtual Lab based on my laptop but due to work and life craziness, the project fizzled.  It's now time to pick it back up and provide some clarity and direction to the project.  Consider this a reboot.

First off, what am I trying to do?  I wanted a lab to help study for VCAP but this could easily be used for VCP as well.  I decided not to purchase dedicated servers and storage because I really don't want all that gear at my house.  I manage enough hardware on a daily basis, no need to add to that.

What hardware do you need?  I recommend the maximum amount of memory your PC will support as well as an SSD based disk to handle the IOP's the virtual machines will create.  I've tried this on a regular HD in the past, trust me, it's very painful.  My MacBook Pro now has 8GB and a 240GB SSD.  Details on that are here.

What can you do in the lab?  The goal of this lab isn't to provide an environment to run a bunch of virtual machines.  Because I'll be running ESXi inside Fusion (Player and Workstation work great as well), that really isn't the intent here.  The focus is on Virtual Center and ESXi, not the virtual machines.

What other benefits are there to the virtual lab?  Two other major benefits are that the lab is always with me since it is on my laptop and it is much more cost effective (8GB RAM and SSD = $500-700 vs. Dedicated Home Lab = $2k+)

What will the project look like?  Here is a quick graphic of what I hope to be the end result.  I will have two ESXi hosts, a Virtual Center server, an AD, DNS & SMTP relay, and lastly a storage VSA for iSCSI and NFS mounts.  For details on how I "trick" the Fusion/Player/Workstation networking by using static IP's instead of DHCP, see this article.  That's it for today, next time we'll install ESXi!

Installing and Tweaking ESXi for VMware Fusion

This article is part of my Virtual Lab Series.

NOTE: If you want to enable VMware HA on your cluster, you might want to read this article first.  I *THINK* I had to bump up the memory to 2.5GB and the disk to 10GB.  Your results may vary!

Now that the SSD is installed in my laptop, it's time to throw some virtual machines at it!  I'm going to start by installing and tweaking ESXi in preparation for creating linked clones of this template to run my virtual vSphere Cluster.  If you've installed ESXi in a virtual machine before, skip ahead to the memory tweaks section; it has changed a bit with ESXi 4.1 as you'll see below.

Start by creating the virtual machine in Fusion and choosing VMware ESX as the OS:

Before clicking Finish, click Customize Settings:

Change the size of the Hard Disk from the default of 40GB to 5GB:

The rest of the install is a standard ESXi install.  The only issue I had was some keyboard "funkiness".  Some Function keys required me to hold down the Command key in addition (F11 was Command-F11 for instance).  I haven't assigned a machine name or static IP address yet because I'm going to create Linked Clones in Fusion so I will personalize the installs after the split.  After install, ESXi only consumed about 2.4 GB of disk space on the SSD.

The last step is to tweak the memory consumption of ESXi.  As stated in Duncan's original article and my previous post on setting this up in VMware Player, you can "trick" ESXi into using less than 2GB of memory by removing the memory check.  We have to perform a few additional steps is ESXi 4.1 because Tech Support Mode is disabled by default as shown below (Alt-F1 to see this and Alt-F2 to flip back to the ESXi screen)

Enable Local Tech Support Mode under the Troubleshooting Options menu:

With Local Tech Support enabled, we can now log in to the console on the ESXi virtual machine (remember Alt-F1 to flip to the Tech Support screen).  Log in as root:

As per Duncan's article, edit the /etc/vmware/esx.conf and add the line:

/vmkernel/minMemoryCheck = “false”

Once the entry is in place, shut down the server and set the memory to 1700 MB.  In ESXi 4.0 we could set it as low as 1200 MB but ESXi 4.1 seems to require a bit more.  What happens when you set the memory too low you ask?  The server will PSOD on boot like this:

Once all of this is complete, you should have a bootable ESXi virtual macine in a minimal footprint (less than 3GB of SSD and 1.7 GB of memory) and this will serve as a nice base configuration for our linked clones in the next article.

Care and Feeding (TRIM Support) of SSD's in Apple's Mac OSX

This article is part of my Virtual Lab Series.

As a follow up to yesterday's article on installing an SSD in my Mac, the next question is now what?

Before I start loading virtual machines for my virtual lab I wanted to relay some information on TRIM support and Mac OSX.  First, what is TRIM?  According to the Wikipedia TRIM entry:

In computing, a TRIM command allows an operating system to inform a solid-state drive which blocks of data are no longer considered in use and can be wiped internally.

TRIM was introduced soon after SSDs started to become an affordable alternative for traditional hard disks as permanent storage in PCs. Because low-level operation of SSDs differs significantly from traditional hard disks (see details below), the typical way in which operating systems handle operations like deletes and formats (not explicitly communicating the involved sectors/pages to the underlying storage medium) resulted in unanticipated progressive performance degradation of write operations on SSDs.^[1] TRIM enables the SSD to handle garbage collection overhead, that would otherwise significantly slow down future write operations to the involved blocks, in advance.^[2]

Although tools to "reset" some drives to a fresh state were already available before the introduction of TRIM, they also delete all data on the drive, which makes them impractical to use for ongoing optimization.^[3] More recent SSDs will often contain internal idle/background garbage collection mechanisms that work independently of TRIM; although this successfully maintains their performance even under operating systems that do not (yet) support TRIM, it has the drawback of increased write amplification and associated increased wear of the flash cells.^[4]

Why is this a big deal?  Simply put, without TRIM, your SSD performance will degrade over time.  The problem here is Mac OSX doesn't support it!  Most operating systems these days support TRIM but old Steve Jobs doesn't consider this a high priority right now (even though he's pushing SSD's in just about every model they sell - trying to keep the Genius Bar fees up maybe?).  As they would say in Princess Bride, "Inconceivable!"

I have found two articles very useful to maintain the performance of the SSD overtime.  In a nutshell, right now you have to wipe the drive every so often and perform a manual TRIM.  Although this isn't ideal, it isn't a large issue considering how easy it is to reinstall (or clone) the SSD on the Mac compared to Windows.  Here are the articles:

• The awesome Jason Nash's blog article
• This MacRumors Thread (Long but worth the read)  (Thank you to Jason from yesterday's article for leaving a comment with this link!)

That's what I have found so far.  Am I missing anything?  If so, please leave a comment!

Installing a MacBook Pro SSD for a VMware Virtual Lab

This article is part of my Virtual Lab Series.

I decided recently to upgrade the hard disk in my MacBook Pro to a Solid State Disk (SSD) so I can run a "Data Center in a Box", much like the previous one I created before.  This post will cover the installation of the SSD and subsequent articles will cover the set up and configuration of my new DC in a box.  As you will see in this post, I know just enough about the Mac platform to be really dangerous!

To create this environment I purchased two items, an OCZ Vertex 2 240GB SSD and a RocketFish 2.5" SATA HDD Enclosure.  The reason for the external enclosure is twofold; first to clone the SSD from the HDD and the second to swap them long term and use the SSD for boot and the existing HDD for "Tier Two Storage" as needed.  Here's a photo of the parts and the SSD in the case:

 

Step One: Install the SSD in the RocketFish Enclosure (pictured above right).

Step Two: Plug the SSD in and format (Erase) the SSD drive using the Apple Disk Utility.

Step Three: Clone the Internal HDD to the External SSD.  For this step I picked Carbon Copy Cloner.  It is very easy to use, just pick the source and the destination and click clone.  As you can see it took me 2.5 hours for this to clone 142 GB.

Step Four: Swap the internal and external drives.  This step requires a small phillips screwdriver and a Torx T6 screwdriver.  I followed this video from Other World Computing:

Step Five: I swapped the drives and powered on the laptop and got the following.  My first thought, this isn't good!!

Here is where I learned a little about about Mac OSX. Unlike Windows, the system was still looking for the old internal hard disk.  So, I plugged in the old HDD into the USB and the system booted again!  From there I went into System Preferences and changed the SSD to be the boot drive.  One reboot and I was back in business!

Lastly, I ran Xbench to see the before (top) and after (bottom) results.  So far, I'm loving it and will post more as I get the lab setup.

UPDATE: Make sure you read my follow up article on how to maintain the SSD performance over time.

"VMware Data Center in a Box" using VMWare Player

I need to study for my upcoming VCP exam and I have always liked the idea of a "Data Center in a Box".  By this I mean an ESX server, an ESXi server, and a vCenter server, all in virtual machines on my laptop.  It was possible with Workstation 6.5 but it appeared even easier with the release of VMware Player 3.  Combine the ease of setup with VMware Player's free download and this was a winner to me!  My goal was to create an environment to study for my VCP.  It isn't perfect, it isn't even complete.  But, it works!  I'll work on it more AFTER my VCP.  One thing at a time...

Before I begin, I wanted to say a big Thank You to Paul Braren at IBM for his presentation on this topic using VMware Workstation 6.5 at the IBM Technical Conference, Rich's post on ESXi in Player, and Duncan's post on conserving memory in virtual ESX and ESXi.  This article would not have been possible without their information.

Here's the environment: HP Elitebook 8530p w/ 4GB of RAM running XP 32 bit with PAE extensions (yeah, I know... more on that at the end), Virtualization enabled in the BIOS, VMware vSphere 4 and vCenter 4 GA code, and of course VMPlayer 3.0.


Step One - Design the Network Environment

The first thing I needed to do was figure out how to set static IP addresses to work in VMPlayer.  I decided to use the NAT network in Player.  When I set up a machine using DHCP on the NAT network with my older Workstation 6.5 I could see the following settings for the network.  The settings have remained the same for Player:

The NAT network is using the 192.168.116.0 subnet and the first DHCP address is .128.  You can set static IP's below the DHCP range and they will route fine in Player.  I am using the following IP's:

• a DNS/SMTP server at 192.168.116.10 (optional, could be combined into the vCenter server)
  
• vCenter at 192.168.116.30 named esx-vc.testlab.int
  
• vSphere at 192.168.116.31 named esx4.testlab.int
  
• ESX4i at 192.168.116.32 named esx4i.testlab.int

I also decided to use the Private Network for a second Service Console (not really needed, just messing around) and vMotion in anticipation of adding shared storage later on.  Using the method above I determined my private network uses the 192.168.27.0 network.  Based on that, I set IP's on the Private network as follows:

• vSphere SC#2 at 192.168.27.31 and vMotion at 192.168.27.61
• ESXi4 SC#2 at 192.168.27.32 and vMotion at 192.168.27.62
  

Step Two - Install vCenter and DNS

For my environment I already had DNS on another virtual server but it could easily be set up on the vCenter server.  I installed the DNS server and created a namespace called testlab.int.  I then created A records for both vSphere servers as well as reverse PTR lookup records for both vSphere hosts.

Since 2GB is the recommended minimum, I installed vCenter using this amount.  I accepted all the defaults.  After the installation was complete and the vCenter was running, I reduced the amount of memory from 2GB to 1GB.

Step Three - Install vSphere ESXi Server


Using Rich's Post, I installed ESXi using the "Easy" method.  This method set up the virtual machine hardware settings to work fine but it also uses DHCP for the network.  Once the machine booted up, I changed the IP information to my static information and changed the DNS server to my DNS server.

I added a second NIC to attach to my Private network for the second management port and the vMotion port.  Here is a screen shot of the final network configuration:

The last tweak I needed was an adjustment to the memory requirements.  Out of the box, ESXi requires 2GB and that is just too much in this tight environment.  Thanks to Duncan's post, I was able to reduce the amount of memory to 1.2GB.  At 1 GB the server PSOD'd on me.  Bumping it to 1.2 GB was the minimum amount I could set for it to boot.

Step Four - Install ESX vSphere Server

This step was a little rough.  I installed ESX using Rich's method but I couldn't do anything with the server.  For some reason it would reply to pings but I couldn't add it into vCenter and it wouldn't respond in a browser.  I performed a quick re-install but I got the same results.  As a last resort I tweaked the hardware settings slightly (removed the floppy drive, disabled the 3D graphics, hard set the processor types to Intel-VTx or AMD-V) and then I was able to add it into the cluster.  I doubt the changes made a difference but who knows.

I set the network settings the same as the ESXi server by adding the second NIC, etc.

Lastly, back to Duncan's post to tweak the memory consumption.  According to a comment at the bottom you can set the RequiredMemory value to 516096 and use 768 MB of memory.  I set this and it worked!  I was getting a warning that I am consuming over 90% of host memory but it was functional.

Step Five - Create vCenter Cluster and Tweak vCenter

Once the machines were added into vCenter, I created a cluster in vCenter.  The ESX server added fine but the ESXi machine wouldn't configure HA.  I checked DNS, pulled it out, added it back, disabled HA, etc.  Nothing.  Oh well, can't be perfect I guess.  I'll look at it later.

Lastly, the ESX server was in a warning state because over 90% of the memory was in use.  I bumped the alarm up to 95% to warn and 97% to alert and it went away.

For those of you out there who believe screen shot or it didn't happen, here ya go!

 

Next Steps

I still have some work to do.  I would like to scrap the 32 bit XP for a 64 bit OS.  I'm not sure if I'll go to Ubuntu or Windows7.  Ubuntu is attractive because it should only take about 400MB (according to the live CD when I booted it) while Windows7 takes about 1.2GB.  That 800MB difference is another virtual machine!  The problem with Ubuntu is my underlying OS won't run the applications that I need.  I've done that before and I'm not sure I wanna go back.

Lastly, I need some shared storage.  I already have the NetApp Simulator running in a virtual machine but I'm out of memory!  I have a nice 3 node configuration (2nodes in an HA cluster and another stand alone).  All three nodes consume about 1.5 GB.  I am thinking of slimming that down and just running one simulator at 384MB or 512MB.  That is a post for another day.

As always, more to follow...