This post has always been one of the top items hit on our site as well as something that is constantly referenced and linked to. In effort to keep things easy, I’ve decided to move it to it’s own dedicated page.
Thanks for all the comments and feedback as always!
This past week, I’ve been working deploying a vBlock1. In case you weren’t aware, the vBlock solutions are basically reference architecture’s consisting of EMC storage, Cisco UCS blade servers, and VMware. They come in three different flavors (vBlock0, vBlock1, and vBlock2) each with different models of SAN’s and quantity of UCS chassis/blades. Specifically, this vBlock1 solution being deployed consisted of an EMC CX4-480 array, two UCS blade chassis with eight full width blades, and obviously VMware, which would be installed on to each blade server.
During design discussions with the client, one thing discussed quite a bit consisted of how to ensure uniqueness of systems and resources residing in this particular vBlock. Specifically here, I’m referring to WWPN/WWNN and MAC Addresses.
The UCS system allows the creation of Pools that will automatically assign WWPN/WWNN and MAC Addresses to servers as service profiles are applied to them. Just as with any server, we need to ensure these values are unique within the environment. Because portions of each of these values are adjustable by UCS Administrators, it’s important to put in a standard practice for determining what these values are to be. Further, with a standard in place these values can be easily used by Network and Storage Administrators to determine what type of system is accessing what within their world.
Digging some for best practices surrounding this and through personal experience, the below is how we determined what these pools would consist of.
First – WWPN/WWNN:
The IEEE standard for WWPN/WWNN’s dictates a value of eight octets. By default if we look at what UCS Manager automatically uses we’ll see – 20:00:00:25:B5:00:00:00. When creating a pool you would change portions of this value (in order to make it unique) and then determine how many WWPN/WWNN’s you require for your environment. This ultimately creates a block of WWPN/WWNN’s. As service profiles are associated to blade servers, WWPN/WWNN’s are derived from this block and assigned to servers. So how can we ensure UCS clusters are unique within a given environment housing multiple clusters? Looking at the below helps break out that eight octet value to something we can work with and better understand.
Breaking this all down we can say:
- A – Denotes this is an IEEE standard format. This means we must leave the value of 2 intact and unchanged.
- B:CD – Are vendor specific so we can put what we like here. For now, my suggestion is leave B unchanged. More on the CD values in a minute
- EF:GH:IJ – These 24 bits are the OUI. This should be the hardware vendor assigned OUI. Because of this, best practice is to not touch these values.
- KL:MN:OP – The last 24 bits are fair game to change as needed.
The default UCS entry put in the above picture would look like this:
So how do we come up with a standard unique pattern? As mentioned already, there are certain bits that must remain untouched so lets look at those we can. The second octet – CD, can be utilized to denote the UCS domain (or cluster). If this is the first UCS cluster in the environment, I suggest making this 01 then increasing the count as new clusters are deployed (i.e. the second cluster would be 02, third 03, and so on). For the last 24 bits, there are no limits as to what we can use here so wouldn’t it be useful to have something that helped Storage Administrators know what it was checking into their environment. Some proposed values may look like:
- KL = Identifies this as a WWPN or WWNN.
- FF = Denotes this is a WWNN
- 0A = Denotes an WWPN on Fabric A
- 0B = Denotes an WWPN on Fabric B
- MN:OP = Sequential numbering handled automatically by UCS Manager
Putting this to practice I created the following for the first UCS cluster:
WWNN Pool = 20:01:00:25:B5:FF:00:01
WWPN Pool Fabric A = 20:01:00:25:B5:0A:00:01
WWPN Pool Fabric B = 20:01:00:25:B5:0B:00:01
Now let’s look at the MAC Address Pools. This pool is not as detailed or cryptic as the WWPN/WWNN’s so we’ll get through this quickly.
MAC Addresses are derived from a total of six octets. Like WWPN/WWNN’s certain values cannot be changed. Let’s break each octet out again to better understand.
Breaking this down we can say:
- AB:CD:EF – Like the WWPN/WWNN these first 24 bits are specific to the hardware vendor OUI. We should leave these set and fortunately, UCS Manager will not allow you to change these.
- GH:IJ:KL – The last 24 bits can be changed as needed.
The default UCS entry put in the above picture would look like the below:
I had some great input from a few people (@stevie_chambers, @jonisick, and @jeremywaldrop) on the original post on how best to handle these pools. Me being an old server guy, I thought being able to tell what OS was running on the blade from it’s MAC would be helpful. In addition, the networking gents made a suggestion that including the fabric ID would also be helpful. Something I very much agree with here as should you start to see MAC’s from an A fabric running over the B fabric you would know something is wrong.
With the above in mind, to make MAC addresses unique, I’d propose the following breakdown for the last 24 bits:
- GH = UCS Domain
- IJ = Denote what host OS is to be installed onto the blade along with the specific fabric ID.
- 0A = VMware Fabric A
- 1A = Linux Fabric A
- 2A = Windows Fabric A
- 3A = Solaris Fabric A
- 0B = VMware Fabric B
- 1B = Linux Fabric B
- 2B = Windows Fabric B
- 3B = Solaris Fabric B
- KL = Sequential number handled automatically by UCS Manager.
Putting this to practice, I created the following pools:
ESX MAC Address Pool Fabric A = 00:25:B5:01:0A:01
ESX MAC Address Pool Fabric B = 00:25:B5:01:0B:01
Windows MAC Address Pool Fabric A = 00:25:B5:01:2A:01
Windows MAC Address Pool Fabric B = 00:25:B5:01:2B:01
Some final thoughts and proposed best practices pertaining to both WWNN/WWPN and MAC Addresses pools would be:
- Create WWN pools in block sizes of 16. This is a best practice as you can easily determine which derived WWPN/WWNN will get assigned to a specific blade.
- Match WWPN with WWNN blocks. This ensures when both are assigned to a server profile, they line up with one another and can be easily distinguished.
- Only create block sizes large enough for your environment. Additional blocks can be created as required.
- Remember if you’re utilizing the Palo card, you’ll need to figure out how many dynamic NIC’s are to be allocated to each server. Each vNIC will have an assigned MAC Address.
Like most things, I don’t think there is one way to do this. The above is just something that made sense to me and will ensure uniqueness across any given environment (big or small).
If you have a scheme you have used in the past, I’d love to hear how you handled things…