How Cisco UCS Deals with Split Brains

This will be a short post this morning.  I wanted to pass along how Cisco UCS deals with a split brain scenario.  I'll start by explaining how you would get into a split brain scenario.  In normal operations, one of the 6100's is the active brain and the other is the stand-by.  A split brain in UCS would happen when both of the cluster interconnects betweenthe 6100 Fabric Interconnects are severed (the L1 and L2 ports).  The active brain still thinks he is active and the stand-by no longer sees the active so he tries to take over.  You now have a potential power struggle because both brains think they are in charge.

Luckily the Cisco UCS folks are way ahead of this scenario.  They added logic to the Serial EPROM (SEEPROM) in the UCS chassis to resolve the situation.  The odd number of chassis that are added to a UCS Domain act as judges during split brains.  For example with four chassis, three are acting as judges.  A marker is added to the SEEPROM on these chassis to make them quorum resources.  To clarify this a little further bit more, if there is an odd number of chassis, all of them will be used.  If there is an even number of chassis, it will drop the last one (n-1) so the number of quorum chassis will always be odd.

When the split brain is detected, both 6100's will immediately demote themselves and then claim as many of the quorum resources as possible.  Whoever claims the most quorum chassis wins and promotes himself back to the active manager. The scenario would look something like the following.  Pretty slick!