FCIP – The Beginning

FCIP is notably a part of the CCIE Data Center lab exam blueprint. It is also a sticking point for a lot of candidates who have not done a whole lot on the storage networking side. Luckily FCIP has many correlations to the modern-day Ethernet networking that we all know and love, as it’s really just another tunneling technology! After some thought, I have decided to break this down into 2 blog posts. This one will cover FCIP basics, and another that will cover some more advanced FCIP options that you might have to use during the CCIE lab examination.

FCIP is used for extending a Fibre Channel (FC) network over an IP backbone. It encapsulates FC in IP so that SCSI and non-SCSI FC frames can be sent over an IP network. Normally most organizations are not going to do this simply for the sake of extending their FC network (why extend a lossless network over a lossy medium?), but rather for backup or replication jobs that need to occur between storage systems that are across some geographical distance. A typical deployment scenario is shown below:

Here we have two SANs separated by an IP network. Now, the MDSs currently in scope for the CCIE DC lab exam are MDS 9222is, which have 4 Gigabit interfaces native to the system. They also have the needed SAN_EXTN_OVER_IP license shipped natively with the system. The MDS 9222i can support up to 3 FCIP tunnels per gigabit interface, giving us a maximum of 12 FCIP tunnels available by default. So we can easily take one of these Gigabit interfaces, and create an FCIP tunnel across the IP network using its addressing as our tunnel source.

The configuration of an FCIP tunnel is actually really simple. There are a couple of ‘housekeeping’ items however, that we must take care of prior to beginning the configuration. 1 – We must put an IP address on our Gig interfaces, and 2 – We must make sure we have IP reachability to the other side (where we wish to terminate the tunnel). This may seem like a no-brainer, but it’s often the simple things that trip us up in our exams! It must be noted that the MDSs do not support dynamic routing either. So if we have to extend past a L3 boundary, we must put in static routes. Let’s assign some IP addressing to our sample topology:

MDS1
MDS1(config)# int gig 1/1
MDS1(config-if)# ip address 192.168.10.1 255.255.255.0
MDS1(config-if)# no shut
MDS1(config-if)# show int gig 1/1
GigabitEthernet1/1 is up
Hardware is GigabitEthernet, address is 000d.bd85.4a88
Internet address is 192.168.10.1/24
MTU 1500 bytes
Port mode is IPS
Speed is 1 Gbps
Beacon is turned off
Auto-Negotiation is turned on
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 packets input, 0 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
1 packets output, 42 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors

MDS2
MDS2(config-if)# int gig 1/1
MDS2(config-if)# ip address 192.168.10.2 255.255.255.0
MDS2(config-if)# no shut
MDS2(config-if)# show int gig 1/1
GigabitEthernet1/1 is up
Hardware is GigabitEthernet, address is 0017.5ab5.2f58
Internet address is 192.168.10.2/24
MTU 1500 bytes
Port mode is IPS
Speed is 1 Gbps
Beacon is turned off
Auto-Negotiation is turned on
5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 packets input, 0 bytes
0 multicast frames, 0 compressed
0 input errors, 0 frame, 0 overrun 0 fifo
1 packets output, 42 bytes, 0 underruns
0 output errors, 0 collisions, 0 fifo
0 carrier errors

Now that we have placed our addressing, and verified that the interfaces are up, we want to establish that we have basic IP reachability. The best method to verify this is via a PING!

MDS1(config-if)# ping 192.168.10.2
PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.
64 bytes from 192.168.10.2: icmp_seq=2 ttl=255 time=0.397 ms
64 bytes from 192.168.10.2: icmp_seq=3 ttl=255 time=0.390 ms
64 bytes from 192.168.10.2: icmp_seq=4 ttl=255 time=0.380 ms
64 bytes from 192.168.10.2: icmp_seq=5 ttl=255 time=0.385 ms

--- 192.168.10.2 ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.380/0.388/0.397/0.006 ms

We are good to go! Again, had we needed to pass across a L3 boundary, we would have needed a static route. The ‘basic’ FCIP configuration can be broken down into 2 very easy components:

1 – The FCIP Profile configuration
This consists of declaring our tunnel source.
2 – The FCIP interface configuration
We will reference our FCIP profile (for our tunnel source), and we will declare a tunnel destination.

It really is that simple! An FCIP tunnel is ALWAYS an E port, or rather a virtual E (vE) port. So it will be extending the fabric in exactly the same manner as a traditional E-port, even trunking VSANs if you wish it to do so. So zoning changes, principle switch elections, etc…will all be extended across this IP boundary between the storage area networks.

So, in our current demo topology I have setup the JBOD’s VSAN 10. Each MDS has a locally attached JBOD, and some local FLOGI entries. There is no other connection between the MDSs, so the only entries in the FCNS database, are the local entries also present in the FLOGI DB:

MDS1(config-if)# show flogi database
-----------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
-----------------------------------------------------------------------
fc1/5 10 0x0b00e1 21:00:00:11:c6:a6:3c:72 20:00:00:11:c6:a6:3c:72
fc1/5 10 0x0b00ef 21:00:00:14:c3:a0:68:ed 20:00:00:14:c3:a0:68:ed

Total number of flogi = 2.

MDS1(config-if)# show fcns database

VSAN 10:
--------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------
0x0b00e1 NL 21:00:00:11:c6:a6:3c:72 scsi-fcp:target
0x0b00ef NL 21:00:00:14:c3:a0:68:ed scsi-fcp:target

Total number of entries = 2

MDS2(config-if)# show flogi database
-----------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
-----------------------------------------------------------------------
fc1/5 10 0x1600da 22:00:00:11:c6:a6:25:78 20:00:00:11:c6:a6:25:78
fc1/5 10 0x1600e2 22:00:00:14:c3:a0:68:ee 20:00:00:14:c3:a0:68:ee

Total number of flogi = 2.

MDS2(config-if)# show fcns database

VSAN 10:
--------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------
0x1600da NL 22:00:00:11:c6:a6:25:78 scsi-fcp:target
0x1600e2 NL 22:00:00:14:c3:a0:68:ee scsi-fcp:target

Total number of entries = 2

So when we get the FCIP tunnel up and running, one of our first verifications steps will be to check the FCNS database, ensure that the fabric has merged, and that we see 4 entries (2 per side).

Our first configuration step is to enable the feature, and create our FCIP profiles. Again, this is fairly straight-forward. We simply want to create the profile, with any number (it’s locally significant), and bind it to our LOCAL IP address. Remember, this is our tunnel source.

MDS1
MDS1(config-if)# feature fcip
MDS1(config)# fcip profile 1
MDS1(config-profile)# ip address 192.168.10.1

MDS2
MDS2(config-if)# feature fcip
MDS2(config)# fcip profile 1
MDS2(config-profile)# ip address 192.168.10.2

Now we want to create our FCIP interfaces. These are logical interfaces, and again can be any number as they are only locally significant. We will reference our FCIP profile, which we want to use as our tunnel source, and we will declare a tunnel destination.

MDS1
MDS1(config-profile)# int fcip1
MDS1(config-if)# use-profile 1
MDS1(config-if)# peer-info ipad 192.168.10.2
MDS1(config-if)# no shut

MDS2
MDS2(config-profile)# int fcip1
MDS2(config-if)# use-profile 1
MDS2(config-if)# peer-info ipad 192.168.10.1
MDS2(config-if)# no shut

Believe it or not, that’s all we really need in order to get FCIP up and going! Let’s verify that our FCIP interfaces came up.

MDS2(config-if)# show int fcip1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:10:00:0d:ec:34:67:40
Peer port WWN is 20:10:00:0d:ec:10:52:40
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 1 Gbps
Trunk vsans (admin allowed and active) (1,10,301-303)
Trunk vsans (up) (1,10,301-303)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Interface last changed at Fri Dec 26 15:53:16 2014

This is awesome! Our interface is up and trunking, and our VSAN is even in the UP state! Let’s see if the fabrics have merged, and if we see the FCNS database populated properly.

MDS1(config-if)# show fcns database

VSAN 10:
--------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------
0x0b00e1 NL 21:00:00:11:c6:a6:3c:72 scsi-fcp:target
0x0b00ef NL 21:00:00:14:c3:a0:68:ed scsi-fcp:target
0x1600da NL 22:00:00:11:c6:a6:25:78 scsi-fcp:target
0x1600e2 NL 22:00:00:14:c3:a0:68:ee scsi-fcp:target

Total number of entries = 4

MDS2(config-if)# show fcns database

VSAN 10:
--------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------
0x0b00e1 NL 21:00:00:11:c6:a6:3c:72 scsi-fcp:target
0x0b00ef NL 21:00:00:14:c3:a0:68:ed scsi-fcp:target
0x1600da NL 22:00:00:11:c6:a6:25:78 scsi-fcp:target
0x1600e2 NL 22:00:00:14:c3:a0:68:ee scsi-fcp:target

Total number of entries = 4

That, my friends, is winning! We see now that we have all 4 entries in our FCNS database, meaning that our fabrics have, in fact, merged properly. There are a couple more things that we can verify here however. Let’s take a look at the full output of the command ‘show interface fcip1.’

MDS2(config-if)# show int fcip1
fcip1 is trunking
Hardware is GigabitEthernet
Port WWN is 20:10:00:0d:ec:34:67:40
Peer port WWN is 20:10:00:0d:ec:10:52:40
Admin port mode is auto, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 1
Speed is 1 Gbps
Trunk vsans (admin allowed and active) (1,10,301-303)
Trunk vsans (up) (1,10,301-303)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()
Interface last changed at Fri Dec 26 15:53:16 2014

Using Profile id 1 (interface GigabitEthernet1/1)
Peer Information
Peer Internet address is 192.168.10.1 and port is 3225
Write acceleration mode is configured off
Tape acceleration mode is configured off
Tape Accelerator flow control buffer size is automatic
FICON XRC Accelerator is configured off
Ficon Tape acceleration configured off for all vsans
IP Compression is disabled
Maximum number of TCP connections is 2
QOS control code point is 0
QOS data code point is 0
TCP Connection Information
2 Active TCP connections
Control connection: Local 192.168.10.2:3225, Remote 192.168.10.1:65508
Data connection: Local 192.168.10.2:3225, Remote 192.168.10.1:65510
18 Attempts for active connections, 6 close of connections
TCP Parameters
Path MTU 1500 bytes
Current retransmission timeout is 200 ms
Round trip time: Smoothed 2 ms, Variance: 3 Jitter: 150 us
Advertized window: Current: 33 KB, Maximum: 25 KB, Scale: 5
Peer receive window: Current: 29 KB, Maximum: 29 KB, Scale: 5
Congestion window: Current: 14 KB, Slow start threshold: 112 KB
Current Send Buffer Size: 25 KB, Requested Send Buffer Size: 0 KB
CWM Burst Size: 50 KB
Measured RTT : 0 us Min RTT: 0 us Max RTT: 0 us
5 minutes input rate 2680 bits/sec, 335 bytes/sec, 2 frames/sec
5 minutes output rate 2632 bits/sec, 329 bytes/sec, 2 frames/sec
808 frames input, 100688 bytes
792 Class F frames input, 98872 bytes
16 Class 2/3 frames input, 1816 bytes
0 Reass frames
0 Error frames timestamp error 0
814 frames output, 99900 bytes
798 Class F frames output, 98084 bytes
16 Class 2/3 frames output, 1816 bytes
0 Error frames

This is one of the best troubleshooting and verification commands available for FCIP! We can immediately note a couple of things here. The default TCP port for FCIP is 3225, and we have 2 TCP streams by default. One of these streams is for control traffic, and the other for data traffic. We will see in the next post, that we can assign certain DSCP values to these streams, so that they can have QoS policies give them certain treatment throughout the network. We can also tell here, that MDS1 initiated the connection to MDS2, as MDS2’s local port is 3225, while MDS1’s port is a high-numbered ephemeral port. This is another item that we will learn to control, and manipulate, in the next blog post!

If you have any questions or comments please feel free to leave them here on the blog, or shoot me an email directly at jlunde@ipexpert.com. I look forward to hearing from you, and please check back soon for the second post in this series where we will tweak some nerd knobs with regards to FCIP!