FCoE vs. NFS on Cisco UCS/VMware/NetApp with Dedupe

This article is quick and dirty. Just the title makes my brain hurt a bit. This came about because of Twitter conversations this afternoon with Brad Hedlund at Cisco regarding UCS design considerations. I am thinking out loud here and wondering what the community thinks about the following.

NOTE: I have not set this up yet so my assumptions may be a little off. My company is supposed to be getting UCS in the near future to demo for customers and at that time I will be able to prototype all of this.

Assume for a second that have Cisco UCS with VMware and you are attaching to a storage system that will do FC or NFS. In this case I'm going to use NetApp because that is what I know.

You have two big options, 1. The customer has FC in place already to the storage. In this case I agree with Brad, use FCoE!

But, what if they don't have FC in place today? Maybe it is a new installation or maybe they are doing IP based storage. I have a large number of customers in this situation.

If there is no legacy FC in place, why would you go FCoE? In my opinion, NFS is much easier to implement in this situation. Let me elaborate.

1. The Transport Layer - Instead of FCoE, I propose you could use straight 10GB IP and run NFS on top of that. I admit, I'm not sure how that will work on the UCS side but I'm assuming that NFS on an ip kernel from VMware is possible to implement inside vSphere. Your end to end configuration is all IP based and MUCH easier to set up than FCoE based on the experience I have had with it to date with the Nexus switches.

2. The Storage Side - NFS on NetApp is handled at the volume level. So is dedupe. Everyone on NetApp with VMware wants dedupe these days. It might as well be a requirement because the savings are just too good to pass up. Because both NFS and dedupe are at the volume level, everything is nice and clean.

Lun based Storage will be required for FC/FCoE. For those that don't know, a lun is basically a file inside a volume for NetApp. It is a level down. Because of this, dedupe on LUNs is tricky. Don't let anyone at NetApp tell you otherwise. In order to fully take advantage of it you need to use thin provisioning and play some other tricks. If you thin provision, you now have introduced a level of risk that thick provisioning LUNS does not have. To be safe in thin provisioning, you need to monitor the environment closely or use alerting. I can go into this much more but it is probably a set of blogs in itself. (For instance, do you do multiple luns inside the volume to take advantage of space, do you thin provision the volume and push the free space to the aggregate, etc)

In summary for NFS: Set up IP storage, create the volume on the NetApp, dedupe the volume

In summary for FCoE: Set up FCoE (more difficult than IP), create a volume, create a lun inside the volume, thin provision the lun, monitor the lun closely, dedupe up at the volume level

It isn't the FCoE that makes it difficult, it is a combination of the more difficult transport configuration combined with dedupe on luns on NetApp!

DISCLAIMER: This entire post was written in about 15 minutes. I reserve right to to make a bunch of writing mistakes and be wrong on the UCS side but I have configured both NFS and FCoE and I have done dedupe on NetApp until my head hurts.