N-Port Virtualization (NPV), and N-port ID Virtualization (NPIV) have been around for quite some time now. Enhancements have been made to the traditional NPV and NPIV implementations, making it more convenient for unified fabric topologies (which is what we will be discussing today). This blog, part 1 in a 2-part series, will be discussing the ‘fcoe-npv’ implementation of NPV/NPIV, while the next blog will be focused on the traditional implementation.
NPV and NPIV were created as a method in which we could add additional switches (i.e. port density), to a given fabric, without consuming additional domain-id’s, or adding to the administrative burden of a growing SAN (managing zoning, domain-id’s, principle switch elections, FSPF routing, etc…). A lot of this concern stemmed from the fact that the Fibre Channel standard limits us to 239 usable domain id’s. Essentially 8-bits, or the most significant byte in the Fibre Channel ID (FCID), is reserved for this domain-id. This byte is what is used within FSPF protocol to route traffic throughout a Fibre Channel fabric. While this gives us 256 addresses, only 239 are usable, as some are reserved. Beyond this, many vendors restrict us too a much smaller number of domain-id’s on the fabric. Brocade, for example, will typically impose a 24 edge switch limit per fabric, while Cisco recommends staying under 40; though some of the larger MDS switches state the number 80 as their tested and theoretical maximum.
The feature ‘fcoe-npv’ came to fruition in NX-OS 5.0(3)N2(1). It offers what Cisco describes as an ‘enhanced form of FIP snooping,’ as it implements FIP snooping as well as providing the legacy benefits of NPV with traffic-engineering and VSAN management. The fcoe-npv feature requires the FCOE_NPV_PKG license on the device but does not require a reload like the legacy NPV (and subsequent configuration wipe). Also, it should be noted that when you utilize fcoe-npv, you cannot have the ‘feature fcoe’ separately enabled, nor can your device support native Fibre Channel interfaces.
SW4(config)# feature fcoe
ERROR: Cannot enable feature fcoe because feature fcoe-npv is enabled. Disable feature fcoe-npv, reload the system, and try again.
SW4(config)# slot 1
SW4(config-slot)# port 31-32 type fc<s/span>
ERROR: Cannot change port types to fc as feature fcoe-npv is enabled
Now enough with the perfunctory stuff… Let’s take a look at the topology that we will be using! Below we see a diagram with 2 Nexus 5548’s (SW3 and SW4), as well as a couple of FEX’s and a C200 M2 with a P81 VIC card in it. SW3 in this diagram will represent our NPIV switch, while SW4 will be our fcoe-npv test dummy. We will be simulating FLOGI’s from the C-series server into the network.
So the first thing we need to do is prepare the infrastructure, and verify at a basic level, that everything is operating correctly. So let’s go ahead and enable our features.
Step 1 is now complete! I didn’t show it here, but I did have to manually turn on the service policies to support FCoE, as this switch did not apply them automatically. Next we need to prepare our Ethernet infrastructure. So, as you all probably know, in order to run FCoE, we need to run FCoE over 802.1q trunks. So I am going to go ahead and configure the interface e1/5 between SW3 and SW4 as a trunk, as well as port 1 off of both Fabric Extenders.
Now we can put in our VSANs, as well as FCoE VSANs.
Good, now on to the VFC (virtual fibre channel) interface configuration! We will start with the link between SW3 and SW4.
Now for the checks and balances we as CCIE candidates strive for! What we should expect is that SW4’s VFC5 interface will FLOGI on one of the two available VSANs using it’s fwwn (fabric world-wide name), up to SW3. So let’s check out SW3, and see if that is so.
So we do see a FLOGI on VSAN 20! Lets make sure that it is the vfc5 interface’s fwwn, off of SW4.
We now have verification that SW4’s VFC5 interface sent a FLOGI to log into SW3. This is the nature of NPV. The first FLOGI will always be the NPV switches NP port logging into the NPIV switch.
Now we need to configure the host ports. To properly simulate, we are going to place one port in VSAN 10, and the other in VSAN 20. We want to make certain that the FLOGIs are converted to FDISC’s (fabric discovery messages) and proxied to the upstream NPIV device. What you will notice here is that we do NOT turn on the feature ‘fport-channel-trunk.’ This feature is intended for native FC only. By default, FCoE interfaces have trunk mode set to on, and, in fact, we cannot disable this!
So now we will configure our 2 host facing VFC interfaces. We essentially want something like this to happen within our topology:
Now we can check our VFC interfaces.
So as of right now, we have nothing! What we need to do in our lab, as the C-series is not actively pursing storage, is reboot it. We have told the FC vHBA’s to boot from SAN, and we have put in some fake pWWNs for it to try and boot towards. While this is not perfect, it will, if configured correctly, FLOGI initially while trying to boot-up. Lets go into the CIMC of that server. Here are my 2 vHBA’s, and their pWWN’s, and the link we will click to reload the server.
A couple of notable debugs, if you’re a debugging sort of person:
- Debug flogi init << To see the FLOGI’s on the NPIV switch come in
- Debug npv flogi-fsm events << To see the FLOGI’s from the NPV switches perspective
After waiting a couple of minutes for the boot-up process to commence, we should see some FLOGIs proxied towards our NPIV switch. Normally we will see one side FLOGI, and then 30 seconds to a minute later, the other side should login as well. Once complete, we should see something resembling this on the NPIV switch:
Notice we now see our 2 vHBA’s logged into their respective VSANs (they even have the same FCID in their respective VSANs in this example!), as well as the vfc5, interfaces initial login from SW4’s NPV port.
If we look at SW3, we can now see that both VSAN’s are in the ‘UP’ state (remember just 20 was prior, as nothing was active on VSAN 10), and we can see the proxied pWWN’s.
So everything works as we expected within this topology. A couple of key takeaways here however….the NPV switch no longer takes up a domain-id, nor does it participate in zoning or FSPF routing. It’s basically a fabric extender on the storage network, and the nodes logging in through the NPV switch will be logged in on the upstream NPIV switch, using its domain-ID within their FCID’s. Utilizing ‘fcoe-npv’ we did NOT have to reboot our switch, lose any configuration, nor did we have to enable trunking, or the feature fport-channel-trunking, to allow VSANs to be tagged across our VFC E-port interfaces. Trunking is always on with regards to VFC interfaces.
In the next part, we will explore the native NPV/NPIV functionality, and some of the quirks that go along with it! If you want to read a little bit more about fcoe-npv, I have included some reference links below (as well as links for some information annotated within this post).
iPexpert Study Materials Covering NPV/NPIV
- Cisco CCIE Data Center Technology Workbook (Vol. 1)
- CCIE Data Center Lab Video on Demand
- CCIE Data Center Written Exam Video on Demand
