5.1 ISL/E-port configuration – The right way.

To many it has always been a mystery what happens when you connect fibre-channel switches to each other and all of a sudden magic happens and you can have an host “talk” to an array or an other device on an other switch. The same mystery however applies when this doesn’t work and you see “E-port segmented, port disabled”. In later FOS codes you may see some additional cryptic reasons like “ESC mismatch” but to many this is as gibberish as particle physics.

This post explains most of the important settings on an Brocade switch port destined to become an E-port in either a standalone master or in a slave role as part of a trunk. I’ll also highlight the importance of some settings when it comes to virtual channel initialization on both short and long distance settings as well as things seen on the wire when an ISL is segmented due to a fabric configuration problem. This post also touches on C/DWDM connectivity in relation to Brocade ISL’s.

Introduction.

For this post I have connected two directors to a JDSU Xgig analyser so you will see some example how things look on a wire. There is one DCX-4S (8G) and a DCX8510 (16G) which is these days often seen in customer environments. They will gradually migrate to an all 16G environment. Advances in ASIC design have added new features but also resulted in some of these to become obsolete; this also caused some confusion in design and configuration options which often results in fabric operations not working as they were intended. Especially when long distance equipment like CWDM/DWDM is involved this may even result in very unpredictable behavior. (I will come back to long distance links with CWDM/DWDM in a later post.)

The FOS version used is 7.4.0a which, at the time of writing, is the latest available from Brocade for the above mentioned platforms.

E-port initialization

Before a switch port is able to determine if it needs to become F-port, FL-port or E-port it goes through an initialization process. This process follows a certain flow which is determined by the FC-SW standard from the T11 committee. Below is a short flowchart of how this works.

E-port init

Be aware this is only a small snippet of the entire initialization process. As soon as a port becomes one of the end-states (FL,E or F) additional processes take place as for instance registration of FC4 types (in case of F and FL-port), exchange of switch and fabric parameters plus zoning etc when a port becomes an E-port. (ESC and EFP are examples of frames.)

Also note the start status has already been preceded by a speed negotiation process in order to check on the highest mutual speed these ports can achieve. The decision and status points are subject to timers. These timers are also defined by the FC-SW standards and are in general fairly gracious. So for example when a port has sent an ELP it will wait a maximum of 2*E_D_TOV+4 seconds (which in most environments translates to 8 seconds) before it determines if it failed or not. If in that timeframe an SW_ACC is received it will immediately transition into the next state and not wait for that timeout.

As you can see depending on the configuration you’ve set on these ports there may be a fair few steps being taken by a port before you reach the end-result. Lets have a look at these options

Port configuration options

Ports of Slot 2        24  25 
----------------------+---+---
Octet Speed Combo       1   1 
Speed                  AN  AN 
AL_PA Offset 13        ..  .. 
Trunk Port             ON  ON 
Long Distance          LS  .. 
VC Link Init           ON  .. 
Locked L_Port           -   - 
Locked G_Port          ..  .. 
Disabled E_Port        ..  .. 
Locked E_Port          ..  .. 
ISL R_RDY Mode         ..  .. 
RSCN Suppressed        ..  .. 
Persistent Disable     ..  .. 
LOS TOV mode            0   0 
NPIV capability        ON  ON 
NPIV PP Limit          126 126 
NPIV FLOGI Logout      ..  ..  
QOS Port               ON  AE 
EX Port                ..  .. 
Mirror Port            ..  .. 
Rate Limit             ..  .. 
Credit Recovery        ON  ON 
Fport Buffers          ..  .. 
Eport Credits          ..  .. 
Port Auto Disable      ..  .. 
CSCTL mode             ..  .. 
D-Port mode            ..  .. 
D-Port over DWDM       ..  .. 
Compression            ..  .. 
Encryption             ..  .. 
FEC                    ON  ON 
FEC via TTS            ..  .. 
Fault Delay             0   0 
SIM Port               ..  .. 
Non-DFE                ..  .. 
TDZ mode               ..  .. 

I highlighted the settings that are important in relation to the flowchart.

Speed.

When you set the speed to a fixed value you will save on speed-negotiation time as only one iteration of the normal detection algorithm needs to be executed in order to determine if the port can obtain a bit and word synchronization (see here and other posts why it sometimes is troublesome). In contrast with the Ethernet world where auto-negotiation does a whole lot of other stuff (see here ), in the FC world there is nothing against setting a port to a fixed speed.

If you know a port will be set for a specific goal or will connect to a remote link that has different speed capabilities set the port-speed fixed to the highest mutual speed.

Locked L-port.

If a port is set fixed to Locked L-port it will not traverse the PC-PH step after the FC-AL step. If it is able to use FC-AL to the attached device it will remain there. If not it will move to a Offline state state. You will see this in the “switchshow” output.

Locked G-port

In today’s data-centers you need to look hard to to encounter a device that is unable to use FC-PH connectivity (ie become an N-port connected to a switch instead of L). In all these cases there really is no need to start initiating loop negotiation at all and directly move to FC-PH. The Locked G-port setting does just that. It bypasses the L-port negotiation phase and thus will not send out a LIP (Loop Initialization Process) with all its sub-statuses. Normally a loop phase passes pretty quickly but if you have a device (HBA, Tape, Array) that shows issues during this phase you might want to try setting the port to “Locked G-port” mode. Also see “Port stuck in L-port mode

Locked E-port

A locked E-port is just what is says. After is gained sync with its peer it will bypass the loop initialization, and will immediately transmit an ELP frame (Exchange Link Parameters) and wait for a predefined time for the remote side to respond with an ACK on that ELP and transition to E-port status. It will ignore any other frames (Like FLOGI) coming from the remote side. If the remote side is an end-device it will not receive an ACK and SW_ACC back hence initialization times out after which it will move into an Offline state.

If you know the remote side of this port will be a switch enable this setting with “portcfgeport <portnumber>

Disable E-port

The “Disable E-port” setting is often used as a security precaution. Albeit a bit clunky as there are better alternatives it simply prevents someone plugging in a switch and basically obtain full administrative privileges over a fabric. Remember that if a switch in its default settings is attached to an existing fabric it will simply adopt all fabric-wide settings except security options as these are local to a switch. This means that as soon as this person has done this it can adjust (or even totally disable) the zoning configuration with all accompanied consequences. A better and more flexible alternative is to create SCC policies with the “secpolicycreate” command in a strict setup. Even when a person is able to hook up a switch to an existing fabric the ISL will not come online as long as the policy does not include the new switch’s WWN.

The ELP

One of the most important frames going back-end forth between two switch ports is the ELP (Exchange Link Parameters). It is sent to the fabric controller’s well-know-address FFFFFD active on each switch (Do not confuse this with anything fabric related like zoning or principal switch selection, that will come later). This frame contains the parameters supported or requested from the link between the two switch-ports.  The ELP exchanges information like BB-credits, R_A_TOV, E_D_TOV, switchname (WWN), class of service port parameters, and the flow-control mode and its associated parameters.

Of these parameters the fixed values of R_A_TOV, E_D_TOV and class of service parameters need to be compatible and pre-defined in each switch. These values are set to a certain value by default and only in very extreme corner-cases these need to be adjusted. The other values like BB_SC_N, service parameters and flow control will be negotiated during this sequence.

If any of these have a discrepancy or are unable to negotiate a common denominator the port will fail.

Most of the values are often fixed and have been the default for a long time simply because they work very well. The option that is flexible here is the flow control mode and its associated parameters. This needs to be configured per ISL and has to be consistent on all ISL’s that a part of a trunk. (more on trunking later)

Flow control mode

The buffer-to-buffer flow control mode initially designed in the FC-SW standard was the same as for F-port and followed the R_RDY methodology. A port sends frames based on the number of credits it received from the remote side and had to wait for R_RDY (Receiver_Ready) primitive signals to be returned before it was allowed to send subsequent frames. This is still a valid option in some circumstances (more on that later) but restricts quite a few options which improve overall frameflow, flexibility and Quality of Service.

The FC standard specifies the flowcontrol mode as follows

  • 0001 – Vendor Specific
  • 0002 – R_RDY
  • 0003-1FFF – Vendor Specific
  • 2000 – VC_RDY
  • AE02 – Reserved for AE use (Avionics ie Jetfighters etc.)

Brocade uses 4 modes of operation in line with these.

  1. VC-mode – Proprietary version of option 4
  2. R_RDY – for compatibility primarily.
  3. Dual-credit mode – Seems to be backdating to the earlier LOOM asic. not used anymore.
  4. QoS mode – Extended VC mode

As you can see there are a fair few “Vendor Specific” options available and this is most of the time the reason why interoperability between switches from different vendors is so hard. As soon as a switch (physical or logical) is configured in, so called, “Interop mode”, the E-port connectivity options are one of the first to revert back to their most primal mode.

R_RDY

The R_RDY mode is the most compatible with 3rd party equipment especially when looking at DWDM/CWDM extension products. If you have a DWDM link and observe performance problems or just generic connectivity issues this is the mode to set the port to. This is also the mode that provides flexibility of using IDLE fillwords. When C/DWDM equipment is non-passive (and by that I mean using either transponders or TDM devices) the R_RDY mode with IDLE fillwords most often work as the extension product have to crack open the Fibre-Channel frameflow and enable mapping these streams to different wave-lenghts or bit-rates. If your C/DWDM is 100% passive (ie using filters to separate light on wavelenght only) it will not interfere with frameflow and primitives and thus all options available on a port can be set to the optimum value for that specific ISL.

The command to use R_RDY mode is “portcfgislmode <slot>/port 1“. Setting the fillword to IDLE is optional and not recommended on 8G and higher speeds. (To be honest, as soon as a Brocade switch “sees” that it talks to another Brocade switch it will change the fillword to ARB(EF,EF) anyway irrespective of what is set. On a condor 3 based ASIC you can’t adjust it.

In a FC analyser trace this will show like this:

isl-flow-control-2

VC_RDY

The VC usage was hijacked from the, now obsolete, class 4 service. It divides an ISL in multiple virtual channels each with it own flow control. Where the R_RDY is a simple primitive indicating that a buffer on the receiving side is freed, the VC_RDY adds the virtual channel signaling the sending port it can send a frame on that particular VC. As seen Brocade uses either mode 1 or mode 4. Mode 1 is the default which devides the ISL in 8 channels

  • VC 0 – Fabric class VC
  • VC 1 – Not Used
  • VC 2 – Data (class 2/3)
  • VC 3 – Data (class 2/3)
  • VC 4 – Data (class 2/3)
  • VC 5 – Data (class 2/3)
  • VC 6 – Multicast
  • VC 7 – Broadcast

The Fabric class VC 0 always has the highest priority. No matter how many data-frames you have queued up on the VC’s 2-5 if any frame is destined to a fabric service on a remote switch originating from either a receiving ISL or the local fabric-controller is will bypass everything and will be sent out on the wire first.

The dispersion of frames over these VC’s depends on a predefined selection mapping based on the ALPA id of the destination fabric address. So for example if a frame has a destination address of 0x02d300 the ALPA id is d3. The mapping algorithm will then send this frame over VC5. This map is static so in general is susceptible to an imbalance of workload over the VC’s. If a large portion of destination id’s is very busy and is mapped to the same VC you may observe that one or more ISL are actually underutilized. Brocade has implemented some updated algorithms in their ASIC drivers that prevent this such as VC’s may borrow credits from others and some other nifty things.

An FC trace shows the mode like this:

isl-flow-control-1

Extended VC mode

The Extended VC mode is also a “Vendor Specific” mode of flow control. So in a FC analyser trace you will see this as well.

isl-flow-control-4Given the fact the vendor specific parameters are obviously not part of the FC standard the trace only shows the number of bytes associated with the flow control mode.

The Extended VC mode shows up when QoS is enabled on the ports on both sides of the link. If one of the ports does not support QoS ot it has been turned off the entire ISL will revert to flow mode 1. IF even that is not supported or one of the ports has been set to R_RDY mode it will use that state.

When QoS is enabled 6 additional VC’s are assigned to the ISL which can be used for the following purposes:

  1. Differentiating traffic
  2. Prioritizing traffic
  3. SDDQ or Slow Drain Device Quarantine

The post QoS will show more on this subject. (click here).

DWDM/CDWM and long distance.

The majority of problems on ISL formation is when DWDM or CWDM equipment sit in between the switches that need to form the ISL. Especially with the transition to 8G and ARB fillwords a lot of “older” equipment or outdated firmware prevented the E-port initialization.

There are 4 solutions that fall under the DWDM/CWDM umbrella:

  1. WDM Transponder based
  2. Filter based
  3. SFP Optics based
  4. TDM Time Division Multiplexed based.

Of these 4 only numbers 2 and 3 are 100% passive. The SFP based solution simply sends out light on a very specific wavelength. The concentrator where these come together combines those to be sent out on a single fibre. The reverse happens on the remote side.

The filter based (2) solution collects all connected cables and based on the the DWDM solution and configuration it filters a certain color out of that receiving light. The similar prism concept as (3) is then used to transport these specific wavelengths over a single fibre. You may see this as a reversed bit of these specific SFP’s.

The transponder based solution (1) does a conversion methodology where it receives the incoming signal, it converts it to electrical, determines on which wavelength it needs to be send out based on the configured settings and then converts it back to optical again to be sent out over that wavelength. The remote side needs to have a compatible configuration to be able to map the correct wavelength back to the intended destination port.

TDM devices map multiple incoming signals onto a higher bit-rate channel so it can be send on a single wavelength. TDM devices are very prone to compatibility issues as these play a very active part in opening up the FC traffic stream and align the data-frames to these higher frequencies. Furthermore they seem to be more susceptible to jitter (variable latency) which can result in synchronization issues.

In a FC environment it’s best to avoid TDM. If you do require TDM you have to turn off Brocade specific feature like QoS, Credit Recovery and use IDLE primitives.

portcfgqos –disable <slot>/<port>

portcfgcreditrecovery –disable <slot>/<port>

portcfglongdistance <LS|LD> 0 -distance <# km>

C/DWDM High availability

Most of the providers of C/DWDM equipment provide, so called, OPS or Optical Protection Switching. This is a mechanism where the DWDM solution switches a data stream from one optical path to the other for whatever reason. DWDM solutions are often build in a dual-ring topology. This allows the DWDM to send a stream over a predefined path and have the other as a back-up. There are two problems with this configuration if not set-up correctly.

  1. OPS is causing invalid encoding words on the FC side.
  2. It might invalidate a trunk configuration.

A Loss-of-Sync condition between two FC ports is attained when a bit or word synchronization is lost for 100ms. The majority of DWDM equipement is able to switch the optical path within 50ms however there will always be loss of frames and primitives due ot the encoding/decoding discrepancy. DWDM does not switch specifically on a word or frame boundary so this invalidation of frames and primitive will always happen. The upper layer protocols in hosts or array will need to retry their IO operation. If this switchover occurs when a VC_RDY or R_RDY is on the wire you can imagine that one or more of these will not be seen by the remote side hence causing credit depletion. In some configurations where Credit-Recovery is turned off you will run into a performance reduced mode simply because you’re not using the full capacity of the link. (See post on buffer credits here.)

Secondly a DWDM configuration has no clue of trunking configurations. This means that if you have a trunk going over path A which is 20 KM long you’ll see no problem until the DWDM decides to fail-over one or two paths of that trunk (for whatever reason) and now these run over over a link which is 40KM long. Given the way Brocade trunking works you will see a lot of issues on these links. A Brocade port is unable to determine what happened on that link and only if an error-condition occurred which requires a link to bounce the ISL-trunk is able to re-assess the distance on each of its members and adjust the, so called, skew value. If that crosses a threshold it will not be made part of that trunk but simply becomes an isolated ISL. Make sure that such scenarios are prevented by grouping all trunk-members as a single entity in the DWDM configuration. Refer to the documentation of your respective DWDM solution.

Long Distance buffer credits

Some DWDM solutions provide an alternative to buffer-credit reservation in the Brocade ASIC by spoofing these credits. It stores the frames in a large buffer pool and therefor off-loads the ASIC from needing to provide these and thus they can be allocated to other E- or F-ports. The flip-side obviously is that the DWDM side needs to be able to support virtual channels and, if need, extended VC’s. If it can’t you may need to miss out on a lot of Brocade features which  improve ISL stability and optimized traffic flow.

FILLwords

On E-ports the fillword is even more important than on F-ports. As many of the Brocade ISL features depend on ARB fillwords the brick wall is often hit with incompatible DWDM solutions who’s outdated hardware or firmware does not support this primitive. Feature like QoS and Credit recovery require the use of the ARB primitive so it is imperative that you ask (or demand) your DWDM provider for this to be supported.

A long distance connection will use the ARB primitive by using the vc-link init parameter of the portcfglongdistance command.

portcfglongdistance <slot>/<port> LS/LD 1 -distance <# km>

Dark Fiber

If you have a medium to long distance connection between the sites a dark fibre (ie a very long optic cable) the only thing you need to take care of are:

  1. Licenses – You need the Extended Fabric License in order to be able to reserve/allocate increased buffer credits,
  2. Optics – Make sure you have supported optics. (See here)
  3. A high quality cable.

The quality of the cable is very important as the db values need to be supported by the receiving SFP. If the db fall outside the thresholds the ISL will mostly not even move to a synch state.

The rest of the features supported by Brocade can (and I advise to do so) be enabled.

Trunking, QoS, Credit Recovery and FEC

As you’ve seen in the portcfglongdistance command there is the option to turn on FEC (Forward Error Correction)however this should not be done on DWDM links. On dark fibre links this should be enabled. QoS and Credit recovery depend on the capability of the equipment in between the two switches as mentioned above. On local ISL’s, and ISL’s traversing dark-fibre this should be turned on.

The same is true for trunking. If you need more ISL’s between two switches for redundancy or throughput requirements use trunking. The trunking functionality sits below the portlevel in the FOS feature stack. This means that an ISL can go offline without causing a FSPF reconfiguration. In other words the fabric status does not change if a trunk member goes offline or resets for whatever reason. (This used to be different when master-less trunking was not available but that is no longer the case.) Brocade supports a maximum of 8 ports in a trunk. From a fabric perspective this translates into one ISL. It is on the ISL level where FSPF and the routing tables are calculated and created. The load-balancing algorithms still work on the trunk level so you may see a discrepancy in port-usage when you compare the workload of a single trunk over each individual member.

Credit recovery needs to be turned of whenever possible. If R_RDY or VC_RDY primitives are lost the CR algorithm can determine if and how many are lost and return the value to its original status. This is imperative if you see encoding errors out-of-frames. (The enc_out) column.

I’ve provided an overview of QoS over here. There is one additional feature that requires the QoS feature to be enabled and that is SDDQ. I will come back to that in a later post.

FEC is a error correction methodology which has been in use for decades in the telecommunications sector. Every simple ADSL router has this on by default. What it effectively does is that the sender adds parity to the data-stream in such a way that if anything is corrupted underway it can be re-assembled by the receiving side. This obviously has benefits as bit-flips in that data stream will not cause the entire frame to end up in a CRC error and inevitably be discarded causing an end-device to retry the entire IO. There are however some limitations.

  • It only works on 16G ASICs on 16G or 10G connected ports.
  • When FEC enabled HBA’s are in loop mode and connect as such to a switch.
  • When connected thru DWDM devices that do not support TTS

Once more I need you to refer to the documentation that comes with the attached HBA’s or WDM solution.

Whenever possible enable FEC. “portcfgfec –enable -FEC <slot>/<port>“. Be aware that the deskew value on trunkmembers will show a value that is not correct as the distance calculation differs between FEC-enabled and FEC-disabled ports.

Testing E-port connectivity

Since FOS 7.0 Brocade has introduced the D-port concept. This stands for Diagnostics port. This function puts the port in a state where it will not participate in any fabric related service. It will only turn on link related end-to-end services to try and establish a connection between the two switches and run a series of link-tests by sending a set of data patterns over the link. Furthermore it tests internal and external electrical and optical characteristics and will provide an overview of the findings. Be aware there are limitations on which hardware is supported and in which configuration. For DWDM connected ISL’s there is a “-dwdm” parameter to the “portcfgdport” command which disables optical loop-back tests.

switch:admin> portdporttest --show 10/39
D-Port Information:
===================
Port:                           7
Remote WWNN:                    10:00:00:05:33:81:43:00
Remote port index:              71
Mode:                           Automatic
No. of test frames:             1 Million
Test frame size:                1024 Bytes
FEC (enabled/option/active):    Yes/No/No
CR (enabled/option/active):     Yes/No/No
Start time:                     Sun Dec 7 22:49:05 2014
End time:                       Sun Dec 7 22:49:30 2014
Status:                         PASSED
================================================================================
Test                    Start time      Result          EST(HH:MM:SS) Comments
================================================================================
Electrical loopback     22:49:07        PASSED          --------      ----------
Optical loopback        22:49:21        PASSED          --------      ----------
Link traffic test       22:49:26        PASSED          --------      ----------
================================================================================
Roundtrip link latency:         277 nano-seconds
Estimated cable distance:       3 meters
Buffers required:               1 (for 2112 byte frames at 16Gbps speed)
Egress power:                   Tx: -2.6 dBm, Rx: -3.3 dBm, Diff: 0.7 dBm (Loss is
within tolerable limit)
Ingress power:                  Rx: -2.5 dBm, Tx: -2.7 dBm, Diff: 0.0 dBm (No Loss)

Make sure this feature is used whenever you setup ISL’s irrespective of distance or connectivity methods. Keep the results on archive as this may be of use for reference purposes later on. For example if maintenance has to be carried out on a long-distance dark-fibre connection you can compare the result before and after.

ISL Segmentation

When things don;t work out Brocade is fairly comprehensive in the messaging but you have to know where to find it.

Below some examples.

1. This shows an incompatible long distance mode.

switch0:admin> switchshow
switchName:    switch0
<snip>
Index Slot Port Address Media  Speed        State    Proto
============================================================
   6    1    6   2bfe40   id    N8       Online      FC  LS E-Port  \
segmented,10:00:50:eb:1a:59:0b:04 (Long distance mode incompat)

The problem is the Long-Distance and VC link-init parameter is different between the two switch-ports comprising the ISL:

switch0:admin> portcfgshow

Ports of Slot 1         6   7 
----------------------+---+---
Speed                  AN  AN 
Fill Word(On Active)    1   0 
Fill Word(Current)      1   0  
AL_PA Offset 13        ..  .. 
Trunk Port             ON  ON 
Long Distance          LS  .. 
VC Link Init           ON  .. 

switch1:admin> portcfgshow

Ports of Slot 2        24  25 
----------------------+---+---
Octet Speed Combo       1   1 
Speed                  AN  AN 
AL_PA Offset 13        ..  .. 
Trunk Port             ON  ON 
Long Distance          ..  .. 
VC Link Init           ..  .. 

By correcting this the ISL turns back to normal:

Index Slot Port Address Media  Speed        State    Proto
============================================================
   6    1    6   2bfe40   id    N8       Online      FC  E-Port  \ 
10:00:50:eb:1a:59:0b:04 "" (downstream)(Trunk master)

2. Another reason is incorrect Fabric timers.

Index Slot Port Address Media  Speed        State    Proto
============================================================
   6    1    6   2bfe40   id    N8       Online      FC  E-Port  \
segmented,10:00:50:eb:1a:59:0b:04 (ED TOV incompat)

One switch has a different setting than the other which causes a discrepancy.

On the wire you can see it like this:

Switch one send an ELP with an E_D_TOV

Selection_021

Switch two sends the ELP with 3000ms

Selection_022

 

Obviously causing the SW_RJT being sent with a logical error.

Selection_023

There are two more place you can look for a segmentation problem. The first is the “fabstatsshow” command output:

switch0:FID43:admin> fabstatsshow
Description                  Count   Port    Timestamp
---------------------------  ------  ------  ----------------
Domain ID forcibly changed:       0
E_Port offline transitions:       8       6  Tue Jul 28 11:23:34 2015
Reconfigurations:                15       6  Tue Jul 28 11:32:55 2015
Segmentations due to:
                  Loopback:       0
           Incompatibility:       2 <     6  Tue Jul 28 11:35:40 2015
                   Overlap:       0
                    Zoning:       0
            E_Port Segment:       0
                 Licensing:       0
           Disabled E_Port:       0
               Platform DB:       0
       Sec Incompatibility:       0
             Sec Violation:       0
                 ECP Error:       0
             Duplicate WWN:       0
            Eport Isolated:       0
        AD header conflict:       0
            VF AD conflict:       0
  MSFR/RD H&T WWN conflict:       0
      ETIZ Incompatibility:       0
     ESC detected conflict:       0
       Encryption conflict:       0
      Compression conflict:       0
  Encryp/Comp bw availability:       0
          Defzone conflict:       0
'<'  - Denotes the type of event that occurred last.

and the “fabriclog“. You can align the time seen in the “fabstatsshow” with the one over here:

11:35:40.587754 *ELP ONLINE timer process                   F2,P1  F2,P1  6     0xd21b
11:35:40.595715 ELP negotiated dp_mode=0x0                  F2,P1  F2,P1  6     NA    
11:35:40.595764 *ELP Snd RJT, vndr unq, unknown             F2,P1  F2,P1  6     0xd21b
11:35:40.595943 ELP RJT Rcv - ct prfrm,in prgs,vu=0         F2,P1  F2,P1  6     0x2d13
11:35:40.601683 SCN LR_PORT(1);g=0xa2 LR_OUT                F2,P1  F2,P1  6     NA    
11:35:40.605465 ELP Rcv:rev=2,flow=4,flen=288,size=372 ...  F2,P1  F2,P1  6     0xd21d
11:35:40.605466 op_mode=0x600400                            F2,P1  F2,P1  6     0xd21d
11:35:40.626064 ELP negotiated dp_mode=0x0                  F2,P1  F2,P1  6     NA    
11:35:40.626220 Segment: incompatible time-out va           F2,P1  F2,P1  6     NA    
11:35:40.626221 -lue
11:35:40.626583 *Removing all nodes from port               F2,P1  F2,P1  6     NA    
11:35:40.626916 *ELP Snd RJT, lgcl err, tov msm             F2,P1  F2,P1  6     0xd21d
11:35:41.074834 *ELP ONLINE timer removed                   F2,P1  F2,P1  6     NA    
11:35:42.415248 *ELP has been removed                       F2,ESC F2,ESC 155   NA

As you can see there are quite a few reasons why an ISL may be prevented from becoming active. The three commands above will at least give you some indication where to look.

To finalize the best practices.

Use all options on direct ISL’s and on long-distance dark-fibre connections.

  • Use ARB primitives (portcfgfillword or the vc-link init parameter.)
  • Turn on QoS
  • Turn on Credit Recovery
  • Turn on FEC (on Condor 3 ASICs /16G capable switches)
  • Bypass L and F port negotiation by locking the port to E-port
  • Set fixed port-speed to the highest mutual speed.
  • On medium distances use the “portcfgeportcredit” command to adjust buffer credit requirements.
  • Use the D-port function on all supported hardware whenever possible; including DWDM links.

On DWDM links make sure that you have a list of vendor supported options especially when it comes to the type of equipment, Brocade feature support in relation to fillwords, QoS, FEC, Credit Recovery etc.Timings on OPS, if applicable, and the ability of grouping ingress-ports so that, in the event OPS kicks in, entire groups will be switched.

IT IS NOT ENOUGH JUST HAVING A “YES, WE DO SUPPORT BROCADE FC CONNECTIONS”. !!!!!!

This will only bring you a massive headache when things go wrong and you have to troubleshoot an ISL problem.

 

Print Friendly, PDF & Email

Subscribe to our newsletter to receive updates on products, services and general information around Linux, Storage and Cybersecurity.

The Cybersecurity option is an OPT-OUT selection due to the importance of the category. Modify your choice if needed.

Select list(s):