What is a Fabric Extended (FEX)?
In very simple terms, the FEX serves as the "pipe" between the blades in a UCS chassis and the 6100 Fabric Interconnects (FI's). Each FEX has a maximum of four 10GB connections. Think of it this way, you can "choose" your bandwidth back to the FI's in one, two, or four 10GB increments (the three uplink option isn't supported). If you plug in one, you get 10GB bandwidth spread over 8 blades. Need more bandwidth? Plug in the second to get 20GB. If you really need the maximum then you can go for all four connections for a total of 40GB per FEX. Remember, each UCS chassis contains two FEX's and each FEX is connected to one (and only one - don't cross connect them!) 6100. If you plug in all eight connections, you achieve a maximum of 80GB to the chassis or 10GB per blade. If you are interested in how the traffic flows from the blades to the FEX port (referred to as pinning), here is a great link from Rodney Haywood that details this relationship. Here is a picture of a FEX close up:
Here is how you would cable them (don't cross connect them!!)
Why does this matter?
FEX uplinks directly affect three different areas:
- The bandwidth from the 6100's to the chassis
- The number of chassis supported per pair of 6100's
- The number of vNICs supported by the upcoming UCS Palo blade card.
1. Maximum Bandwidth per Chassis based on FEX Uplinks -> (FEX's * Uplinks *10)
To calculate the amount of bandwidth available to a chassis: (FEX's * Uplinks *10). So, If I have two FEX's, each with 2 uplinks, I have 40GB at my disposal (2*2*10).
2. The number of chassis supported per 6100 is inversely proportional to the number of FEX uplinks -> ((total ports - uplinks) / FEX Uplinks per chassis)
Every time you use more than one FEX uplink, you actually reduce the number of chassis you can plug into the system. Let me use a simple example.
Let's say you have a 20 port Cisco 6120 with the 6x10GB module in it for a total of 26 ports. To make the math simple, you decide to dedicate six 10GB links for northbound (out of the chassis) traffic. You can support up to 20 chassis by using a single FEX uplink per chassis. What if you need a second FEX uplink? The number of chassis goes down to 10 because you need two per chassis but you only have 20 ports to physically plug into. If you need need 4 FEX uplinks, then you can only support 5 chassis per 6120.
To calculate the maximum number of chassis, this is the formula to use: (total ports - uplinks) / FEX Uplinks per chassis. To use the example above, a 6120 with 4 uplinks yields 5 chassis ((26-6) / 4 = 5)
UPDATE: As pointed out by UnixPlayer, what about uplinks out of the chassis?? Great question and I admit it was late when I wrote this. I have now updated this to include uplinks. You of course need some uplinks or this will happen to you!
3. The number of vNICs and vHBAs supported on a Palo card is proportional to the number of FEX uplinks -> ((15*FEX uplinks)-2)
As pointed out by Kevin on his article, the number of uplinks determines the number of vNICs the Palo card can present. The thoeretical maximum of the card is 128 but only 58 can be used currently. This is for both vHBA's and vNICs.
As you can see, there are some interesting design choices to be made based on bandwidth, scalability, and virtual I/O. I really like the customization ability of the UCS system to tailor to the requirements of the customer but it is also very important to understand the relationships presented above when designing the system.
4 comments:
Its work nothing for point 1 that this is the CHASSIS bandwidth not per place. The IOM does pinning so each vNIC in the blade can only get a max of 10Gb or less. The or less depends on how many uplinks you have combined with how many of the other blades are active on your shared uplink.
Nice post.
Rodos
Thank you very much! You make an excellent point!
I wonder what the maximum throughput will look like down to storage.
Hey Simon - Your answer is "it depends". To get to the storage you need to set up a COS (Class of Service) queue. It also depends if you need traditional FC block (FCoE) or something IP based like iSCSI/NFS.
One really nice thing about the UCS is you do have so many choices. You can dedicate as little or as much bandwidth to each blade or chassis as you need.
In normal FCoE scenarios you would set up COS 2 to handle all FC traffic, set it to the highest priority, and then tag it as a no drop since all packets need to get to the storage FAST.
Post a Comment