One of the most complex ways of implementing long distance connectivity is using CWDM or DWDM extension equipment. Reason being is that the technology is often fairly complex and diagnostics in case of troubleshooting is most often done on a layer 1 level which means you need to have extensive knowledge of not only the way Fibre-Channel operates on a FC-PI (Physical Interface) but also how it aligns to the standards described by the ITU.
CWDM/DWDM
As mentioned I’m not going into the details of the WDM technologies but will outline the requirements from a Brocade configuration perspective.
Brocade basically provides two options when connecting to a WDM solution. You can use special SFP’s (from SmartOptics) which are available in different speeds and wavelenghts. These optics work in a passive optical multiplexer and do support all Brocade native primitives (arb(ff), arb(fe), VC_RDY etc…) so it doesn’t really matter which configuration you apply on an ISL. All optical requirements are still to be adhered such as distance and cabling.
When using the normal SFP’s in a switch the wavelength multiplexing and de-multiplexing (MUX/DEMUX) is done in the DWDM equipment. This can be done in two ways. Active and Passive. The Passive way again uses an optical MUX/DEMUX methos via prisms or ROADM’s. (google these for further info.) A generic “white” laser light is filtered at ingress and only the required wavelength is used for that particular channel which is then multiplexed into a single channel. The remote side does the reverse and we’re done.
These are the two easiest paths as everything is done on a layer 0 optical level and does not interfere with any FC signaling layer. It becomes more complex as soon the WDM solution becomes Active. These solutions are most often seen as transponders which have an OEO (Optical – Electrical – Optical) way of mapping the optical wavelengths of the normal 850nm or 1310 nm SFP to a WDM frequency/wavelength. As that now means that it will need to hack into the FC bitstream to align these it also means the WDM transceivers need to use the same characteristics as depicted by the FC-PI standard things become somewhat more…challenging…
Two other options that are often used is TDM (Time Division Multiplexing) which take an incoming signal and map that on a higher bitrate so multiple channel can be pushed onto a same WDM wavelenght. This also requires the MUX to open up the bitstream and align on the FC bitstream in order to determine what goes where. The last solution I mention is FC-SONET/SDH (Synchronous Optical Network and Synchronous Digital Hierarchy) These are failry rare in data-center environments but are generally common in telco infrastructures. SONET/SDH is very suited for FC transport as it is A: connection oriented and B: latency deterministic which means it can be optimized dynamically based on traffic characteristics.
If the solution is 100% passive (ie the first two options I mentioned above) you can configure an ISL as any normal (direct connect) link.Take into account the distance, speed and framelenght and you should be able to simply adjust the number of buffer-credits and you’re sorted.
If any of the other options are used (ie less than 100% passive) there are a few thing you need to know from the WDM solution.
- What solution is used. (Transponder,TDM, SONET/SDH/CWDM)
- Which wavelenghts are supported?
- Which speed is supported.
- Which FC primitives are supported.
- Is FEC used.
Lets start with 5. In most cases in a WDM environment FEC (Forward Error Correction) is already supplied by the transponder, MUX or SONET/SDH solution. Brocade uses a proprietary algorithm for FEC and this will not work on any of these so best to turn this off.
To do this use
portcfgfec –disable -FEC <slot>/<port>
As the transponder needs to align on the incoming bitstream and FC is using a CDR (see here) you need to know which primitive signals the WDM solution supports. If it only supports IDLE fillwords (see here what these mean.) and you need to connect on 8G speed which by definition requires arb(ff) fillwords you already have a problem. The only solution for that is to ask the WDM provide to support the ARB(ff) fillwords or reduce the speed to 4G. Most vendors have already provided firmware upgrades to support these.
As Brocade also uses a specific primitive in order to support virtual channels on the ISL’s this also needs to be recognized and supported. On a normal link the FC protocol uses the R_RDY primitive signal to inform the sender a buffer credit can be replenished. On and ISL with virtual channels this is replaced with a so called “VC_RDY” which has the same purposed but informs the sender to replenish a buffer credit for a specific virtual channel. If the firmware of the WDM equipment does not support this you will need to configure the port to run in R_RDY mode.
portcfgislmode <slot>/<port> 1
This has some implications as with an ISL running in R_RDY mode the QoS and credit-recovery options are also no longer supported and also need to be turned off.
portcfgqos –disable <slot>/<port>
portcfgcreditrecovery –disable <slot>/<port>
This leaves you with a very basic ISL which should be inter-operable with the majority of WDM equipment.
