9.1 – Long Distance configurations – Direct Connect

Switch fabric configurations over dark fibre, direct connect, links are not much different then connecting them when they sit next to each other. There is no physical interference which has an impact on protocol behavior.

There are however guidelines that have to be adhered to.

  1. Distance has an impact on utilization rate and performance.
  2. Frame length also relates to utilization rate and therefore performance.
  3. Optical characteristics have a direct impact on reliability and error behaviour.
  4. ASIC revisions and licenses determine the capabilities of the link and it’s features.

Lets go over each of them.

Obviously the longer the distance the longer it takes for light to travel across. Light is pretty quick: 300.000km/s in a vacuum. The problem is that light in this case doesn’t travel through a vacuum but through glass. Glass in an optical cable has a refractive index of around 1.47 which basically means that despite the purity of it, light through glass is like humans walking through water and the speed will be reduced to around 200.000km/s. (Yes still pretty quick.)

Based on frequency at which light is sent into the cable each frame has a certain physical length. A full FC frame is 2112 bytes and a at 1G capable port the length of each frame is around 4.123 kilometers. A 2G capable port reduces that to 2.061 kilometers and, fast forwarding to 16G, the distance will be around 258 meters. As you can see that, if you have a 40 kilometer link the sending port only requires around 9 to 10 buffer credits on 1G speed to fully be able to fill up that link without running into a buffer credit problem. On 16G however there is a huge amount of space between the frames which you do not utilize.

Below a picture of what I mean.

framelenght_differenceAs you can see there is a lot of space left to accommodate for more frames to run over this, physically the same, cable. This severely impacts performance as the actual speed of light remains the same, the amount of data being sent through the link does not increase proportionally with the increase in frequency.

As mentioned two things are required to solve that. You need to know how long the average frame-length is in bytes and you need to have a license to unlock the additional buffer-credits available in the Brocade ASIC. The average framesize is fairly easy to calculate.

Each port provides two values:

  • The number of 4-byte transmissions characters sent
  • The number of frames sent

Multiple the number of the first by 4 to obtain the total number of bytes and divide that by the number of frames sent. This should yield a result between 64 and 2112 bytes. This then allows you to calculate the total number of frames a particular link can hold.

To configure the port for long distance and the appropriate number of credits Brocade has adjusted its logic so that it now automatically configure the number of credits based on the values you specify as parameters in the “portcfglongdistance” command.

Switch:FID128:admin> portcfglongdistance
Usage:
portcfglongdistance [Slot/]Port Distance_Level
or
portcfglongdistance [Slot/]Port Distance_Level VC_Trans_Link_Init [-distance <Distance> | -buffers <Buffers>] [-framesize <Frame_Size>] [-fecenable | -fecdisable]

Distance_Level: L0 : normal
LE : <= 10km
LD : auto
LS : static
VC_Trans_Link_Init:
0 normal
1 VC translation
Distance Option:
-distance: LD/LS option to preceed desired distance
-buffers : LD/LS option to preceed desired buffers
Frame_Size:
-framesize: LD/LS distance only option to specify average frame size
FEC:
-fecenable: Enables Forward Error Correction on supported ports
-fecdisable: Disables Forward Error Correction

 

As you can see there are also numerous other options but I will highlight these in the following post. (DWDM)

The other command is “portcfgeportcredits“. If you have calculated the number of desired credits yourself you can use:

portcfgeportcredits –enable <slot>/<port> <number of credits>

Do this on both sides of the link.

VC Collapse

The moment you create a long distance link there is another phenomenon that happens in the backgound. The VC’s 3,4 and 5 will collapse into VC2. This is not a bug but was done to be able to allocated sufficient amount of credits to overcome long-distance links. If the normal credit-model would have been use the ISL would sometimes come up in a buffercredit-limited mode if it was connected on a slot/asic with more ISL’s and F-ports.

When a Condor3 based switch (all 16G switches) connects with a normal ISL (default) configuration and QoS enabled (if licensed) The buffer credit alocation is done in this fashion:

Default (QoS enabled)
0    1    2    3    4    5    6    7
bbc_trc                 4    0    2    2    2    2    1    1

8    9   10   11   12   13   14   15
bbc_trc                 2    2    2    2    2    2    2    0
bbc_trc                 0    0    0    0    0    0    0    0

bbc_mbc                 0    0    3    3    3    3    0    0
bbc_mbc                 0    0    3    3    3    3    3    0

The columns represent VC 0 to 7

The bbc_trc is the transmit credits which are fixed on a VC. VC0 (the first column) is F-class traffic which also has the highest priority on the link. No matter how much data-frames you have stacked up if there is an ILS frame (fabric services) it will be sent first and all data frames will need to wait.

VC1 is not used (was reserved for another data type). Then you see VC’s 2-5 have each 2 credits allocated and VC’s 8 to 14 also have each 2 reserved. VC’s 6 and 7 are for multicast and broadcast traffic and each have 1 credit allocated.

The bbc_mbc is the number of credits that can be borrowed by other VC’s in case that VC runs into a shortage momentarily.

You will see that the credit allocation changes when I turn off QoS. The remainder of the port settings stay the same.

bbc_trc                 4    0    5    5    5    5    1    1
bbc_trc                 0    0    0    0    0    0    0    0
bbc_trc                 0    0    0    0    0    0    0    0
bbc_mbc                 0    0    0    0    0    0    0    0
bbc_mbc                 0    0    0    0    0    0    0    0

As you can see the credit allocation has shifted where the total number of reserved credits has reduced from 28 to 26 and VC’s 8 to 15 are no longer used. The VC’s 2 to 5 now have 5 reserved credits each.

When a port is going to be configured for long distance the credit allocation differs again.  The below long distance port was configured for 40 kilometers with a framesize of 2048 bytes.

DCX8510:FID42:root> portbuffershow
User     Port     Lx     Max/Resv  Avg Buffer Usage & FrameSize    Buffer Needed     Link
Port     Type    Mode    Buffers        Tx         Rx              Usage  Buffers   Distance  Buffers
—-     —-    —-    ——-   —————————-    —— ——-   ——— ———-
88       E      LS       326       – (  – )      – (  – )         166     166        40km     4642

bbc_trc                 4    0    160  0    0    0    1    1
bbc_trc                 0    0    0    0    0    0    0    0
bbc_mbc                 0    0    0    0    0    0    0    0
bbc_mbc                 0    0    0    0    0    0    0    0

 

In this case you’ll see that VC’s 3 to 5 are collapsed into VC2 to allow sufficient number of credits being allocated to cover the distance.

On the same link when QoS is enabled the configuration is somewhat a combination of a normal QoS enabled ISL and a long distance ISL with the same QoS setting.

Sydney_ILAB_DCX8510-42:FID42:root> portbuffershow
User     Port     Lx     Max/Resv  Avg Buffer Usage & FrameSize    Buffer Needed     Link
Port     Type    Mode    Buffers        Tx         Rx              Usage  Buffers   Distance  Buffers
—-     —-    —-    ——-   —————————-    —— ——-   ——— ———-
88       E      LS       180       – (  – )      – (  – )         180     180        40km     4788

bbc_trc                 4    0    2    0    0    0    1    1
bbc_trc                 2    2    2    2    2    2    2    0
bbc_mbc                 0    0    158  0    0    0    0    0
bbc_mbc                 158  158  158  158  158  158  158  0

As you can see adding up the credit number on each VC yields in 180 as per the portbuffershow. (Remember the bbc_mbc is a shared credit pool so should be calculated only once.)

So what is a good choice? Depending if you’re using QoS zones or not you’ll need to make the choice between having a QoS enabled or disabled long distance ISL. Since the majority of traffic is on a medium class it will be mapped onto VCs 2 to 5 anyway so if that is your configuration as well you might turn off QoS entirely.

When you turn off QoS you won’t be able to map traffic onto certain VC’s which inherently means that all data traffic will be subject to latency on one or more end-devices.

For example, you have a remote replication solution from your array vendor but you’re also mapping tape and some volumes over this long-distance link to one or more hosts. If this host (or end device in general) observes any form of latency causing credit back-pressure all devices using this long-distance link will see performance issues including your synchronous replication solutions. (see a few other posts here and here amongst a few others as well). When you enable QoS you can map your replication traffic on different VC’s by using QOSHx_zones where x is the VC you want to map it on. If you omit the x it’ll use a predefined algorithm to map it to one of the QOSH VC’s.

Optics

Obviously when you have a long distance connection you need some grunt in laser power to bridge the distance. If no amplifiers are used then long LW lasers are required. Brocade uses a few SFP’s qualified for different distances. On 8G and 16G speeds they come in a 10KM and 25KM variant and on 4G (yes they do still exist) they are orderable in 4, 10 and 30KM ranges. Obviously the price-tags come with the speeds and distance requirements.

Licenses

In order to be able to utilize the link to its full extent as I described above, besides the optics, you also need a license key in order to release the artificial buffer-credit restriction Brocade imposes in FOS. The Extended Fabric license removes this restriction so you can use all available buffers in the respective ASIC’s.

 

Print Friendly, PDF & Email

Subscribe to our newsletter to receive updates on products, services and general information around Linux, Storage and Cybersecurity.

The Cybersecurity option is an OPT-OUT selection due to the importance of the category. Modify your choice if needed.

Select list(s):