So this week I had an interesting case. As you know the Hitachi arrays have a replication functionality called HUR (Hitachi Universal Replicator) which is an advanced a-synchronous replication solution offered for Mainframe and OpenSystems environments. HUR does not use a primary to secondary push method but rather the target system is issuing reads to the primary array after which this one sends the required data. This optimizes the traffic pattern by using batches of data. From a connectivity perspective you will mostly see full 2K FC frames which means on long distance connections (30KM to 100KM) you can very effectively keep a fairly low number of buffer-credits on both sides whilst still maintain optimum link utilization.

Brocade added a new feature into their Condor 3 ASIC (well actually a lot more) called compression. Basically it cracks open the FC frame, takes the payload, compresses it, recalculates the CRC, glues the FC headers and footers back on and sends the frame on its way. This way the amount of data actively being transported over the links can be reduced significantly however there are some gotchas you have to take into account.

Inline vs Buffered.

Obviously a compression algorithm only works on a whole data-set since it needs to calculate which parts of the payload can be reduced in such a way the remote side is able to deduct the original bytes. Nothing new here since that’s what has been done for years. This also means that as soon as you enable compression the concept of cut-through routing goes out the window. If none of the functions are enabled on a normal ISL configuration the FC frame header could already be on it’s way to the destination past the switch E-gress port whilst the FC CRC and footer haven’t even arrived in the ingress port of the same switch yet. Remember a full 2112 byte FC frame on 16G speed is still around 250 meters long. When compression is enabled this entire frame will be buffered in the ASIC compression block on the E-gress side of the switch after which it does its magic and sends the modified frame on its way. Brocade calls this In-flight compression and not Inline. There is a technical difference here. If you have ever worked with a JDSU Xgig analyser/jammer you can flip bits around that are currently on the wire without interfering with buffering and timing. This is what is called In-line. In-flight is different in the sense that it potentially could do anything including buffering, re-routing and changing timing order among other things.

Performance expectation.

In the case I mentioned above the customer had enabled compression and expected a huge performance boost. After all the dataset they used was perfect for compression purposes, mainly text based stuff in databases and files. The majority was not related to any compressed data types like mpeg’s, and highly optimized and compressed database tables so in theory a performance increase could have been expected. They had done some simulations on an IP network by utilizing a WAN optimizer and these results were pretty good. An average of 1:2.8 compression ratio was achievable here. For numerous reasons the customer did not want to use FCIP for replication and they already had a very stable dark fibre link between the datacentres so the choice was made to enable ISL compression. Since FC works a bit different that IP (remember FC uses a buffer-credit mechanism and not a byte-streaming method like IP) the results were a bit disappointing at first.

The analysis.

The link distance between the two datacenters was around 48 kilometers which means that when used by full 2112 byte FC frames the number of frames on the wire at an 8G link speed are around 95 and to fully utilize the link you would need around 189 buffer-credits. This was what had been configured before and the link utilization was optimal. When compression was enabled on the ISL’s we could see the throughput on the array remained the same but the ISL became under utilized. Effectively less bytes were transferred on the wire then previously. The cause was obviously that the ISL hit a buffer credit limitation. Irrespective of how large the original frame is it will always use 1 buffer. Enabling compression does reduce the size of the frame but does not change anything on the buffer-credit handling. The number of credits on this link remained still the same and thus although the frames were smaller the amount of frames the sending side of the link could actually put on the wire remained the same leaving huge gaps between frames. Taking into account the compression ration that was achievable we could then adjust the number of credits to be able to fully utilize the link. So applying a 1:2.8 compression ratio brought down the number of bits to ~7500 which meant we should adjust the number of credits to ~530. I always use a 10% margin on this so we configured the number of credit to be 580. After this the link once more was fully utilized and performance went back to normal.

Caveats

There will always be caveats which restrict the options and possibilities you have on these configurations. You should really read the manuals and release notes of FOS to be able to determine which options you can and cannot apply to your configuration.

Hope this helps.

Cheers

Erwin