This is important!! Now, you would think that when you look at the weather channel and the person tells you there is a wind coming from the North the direction is from North to South. This could be interpreted somewhat different when he or she tells you there is a “northerly” wind. Does the later mean it’s coming from the North or is it going to the North. The same is now true for Brocade 65xx series switches. The 6505/6510 have a different interpretation of how the wind blows than the 6520 series. Is it moving forward or reverse, is it coming from the front or back and on which side is the intake and/or exhaust and which spare-part do you need to order in case one fails, xx-F or xx-R. If you order the wrong one the switch will tell your this after which it will shut down. Whoops. The documentation also shows some confusion around the PS acronym. Is it now related to the airflow, in which case it stands for Port-Side, or does it mean Power Supply when a FRU is meant?
FC Frame header, or is there more to it
The FC frame header has not changed since its inception back in the late 80-ies. This shows the absolute rock-solid backward compatibility towards previous generation platforms and vendors. Obviously the the FC protocol itself has grown and evolved with marked demands to provide an extremely flexible transport mechanism for the most demanding storage environments. When you start to design a new protocol the concept of future proofing is a must and sits very high on the top of the design list. Not only does it require insight into current markets and requirements but also a huge set of extremely smart brains to come up with possible scenario’s of what might be required in years to come. You don’t want it to become obsolete in 5 years time because there were inherent flaws in the protocol. Next to that your have to facilitate options which allows for flexible expansion of functions and features of which none of the above brains had ever thought of. (Did somebody know about VMware back in 1990?) The concepts of NPIV where not on the radar back then. So what is the structure of this protocol that makes it so flexible.
128GFC Fibre-Channel is here
Well, almost… 32GFC is fixed and at the standard has been ratified and the physical specs are moulded in concrete. In addition to this there are functions that enable to combine 4 of these into a single 128Gb link. Yes 128Gb/s. That’s fast….Ohhh wait… thats only ONE WAY which means a single link can push 256Gb/s in total. That allows you to push 25600MB per second through a link. In order to achieve these transmission speeds there has been a somewhat restructuring on the individual requirements w.r.t. quality of the actual link.
Microsofts new CEO. Saviour or the same?
The board of Microsoft has appointed a new CEO. When Steve Balmer announced stepping down from one of the worlds most influential companies for the last 3 decades the search began. You can say from Microsoft what you want but it has been the company that not only brought computing to the home but also created an entire industry around it. I think there is not a company in the world that has created so many jobs in such a vast variety of businesses, governments, education and numerous other market segments than the computer giant from Seattle. The ubiquity of its products is seen everywhere and has been untouchable for a long time. There have been cracks in its structure though. The company hasn’t been able to make tablets a success even though they announced this technology way before Apple has a crack at it. The handheld PC in the form of a phone has never been a success. Windows was able to hook consumers as it had done on the desktop and laptop. They are trying again now that they’ve bought Nokia. (See here).
Comparing different technologies
OK, let me make one this clear, and most of you know this already, I’m not a fan of FCoE and I wrote numerous articles around this topic. I work for a vendor especially focusing on storage connectivity in a engineering support role. Basically it means I need to know this stuff. When I see people struggling with storage from a connectivity perspective and I come to a point I either meet these people in person or have them on the phone I’m pretty often absolutely flabbergasted around the lack of fairly basic knowledge of Fibre-Channel. The basics of checking port and link errors, the generic concept of flow control and operational management is very often on a pretty low level. Don’t get me wrong, I’m not accusing anyone of being stupid however it turns out that education and self-study w.r.t. storage is not seen as a critical item on the day-to-day IT department shopping list and focus is more leaning towards networking, operating systems and applications.
If, however, your sole job is storage and you are tasked to provide a study between two different sorts of technology both touching the same subject and you’re able to screw up the way the Evaluator Group has done last week I seriously doubt the credibility of such an organization.
CRC errors
In Fibre-Channel, and many other network protocols, the use of CRC (Cyclic Redudancy Check) is adopted to detect corruption of frames. Be aware of the word “frames”!! As I explained in previous posts there are two layer of link integrity, an 8b/10b encoding/decoding algorithm (on 10G and 16G FC it has been changed to 64/66) which ensures dc balance plus error detection on the FC1 layer plus CRC which provides an additional check on the FC2 layer. Primitive signals or sequences are not frames and thus are not guarded with a CRC check.
Crc has the benefit that it can calculate on a serial bitstream as opposed to some other methods which require a certain fixed size of data in order to provide an integrity check. (A PKI like infrastructure is something along these lines such as GPG which can cryptographically sign an email message based on the entire content before it is sent). Secondly the calculation and reverse checking is very simple which means it can be build in hardware (ASICs or FPGA’s) without the need for software spending CPU cycles on both ends of the link which would have a serious impact on performance. There is absolutely no integrity check or security mechanism build into crc so any content can easily be modified, the crc recomputed and forwarded without the receiving side knowing it. In my test environments I use such methods to change an FC frame in-flight in order to see the behaviour of the modified content on the HBA or array. This allows me to inject data to test on protocol errors and the subsequent actions. (If I wasn’t able to recompute the crc, the destination port would detect the incorrect crc and just discard the frame.)
Now, going back to Fibre Channel. The CRC is calculated from the Start Of Frame (SOF) until the last word of the payload and is then appended to the frame. The FC2 layer will then add an End Of Frame (EOF) with a status qualifier. (I get back to this later).
Below a screenshot of a FC trace where the host issued a write(10) command to lun 32. The CRC is determined to be correct and the frame is ended with a EOFt (terminate, this doesn’t mean the IO is terminated but this sequence of the FC exchange is completed. I won’t go into this any further)
If for any reason one or more bits in the bitstream between the SOF and last bit of the payload is changed the receiving side will do a reverse crc check which obviously will fail.
Now, I mentioned that the calculation is done inline of the bitstream. These days all fibre-channel implementations from all vendors use cut-through switching which more or less means that as soon as the first word of the FC frame is received (the one that contains the DID Destination ID or FCID) it is immediately forwarded to the out-port of that switch having the route set up according to FSPF. The second word of the frame may not even have arrived in full yet. This ensures optimal performance with next to no latency from a switching algorithm perspective. If you then do some maths and calculate the length of a FC frame it means that the first word of the frame may have already arrived at its final destination before the last word of the frame has even left the source.
In the above picture you see a representation of three switches (pardon my drawing skills). Switch number one on the left has received a frame from an HBA and is sending this out on port 1. The frame is 1KB in size and the link speed is 8Gb/s. This means that the length of the frame is almost 250 meters long. If one or more bits flip at the 512th byte on the link between 1 and 2 the beginning of the frame is already at it’s destination so nowhere in the entire FC path any form of correction can be done. (there is an exception called FEC but I’ll discuss this in a later post). What will happen is that the port at (2) will detect the crc error and it will replace the EOFn or EOFt with a EOFni or EOFti (the i means invalid.) All switches will forward the frame to its destination (unless the DID in the frame-header is the part which is corrupt). As soon as the EOFxi arrives at the destination it can immediately discard the entire frame, clear the buffer and start the recovery procedures. If the intermediate switch detects the crc error and it would have discarded the frame over there both the initiator and target would have no clue what’s going on and would rely on the default FC and SCSI timeout values before is would be able to act on these.
From a troubleshooting perspective if you now look on the error counters on the switch ports (on a Brocade platform) you will see that port (2) will have logged the crc error in two columns of the porterrshow output: the crc_err and crc_g_eof (CRC error with a good EOF). Since the intermediate switch still forwards the frame port (4) will also detect the same crc error however since port (2) changes the EOFx into an EOFxi (invalid) port only the crc_err column on this port is incremented and not the crc_g_eof column. This mechanism allows you to follow upstream paths and determine where these errors originate from.
Hope this brings some insight and gives you better info of how to interpret these kind of errors.
Regards,
Erwin
Fabric design, the good the bad and the ugly.
For numerous years entire bibles have been filled with storage design concepts, pro’s, cons, benefits, cost structures on port-counts vs. performance etc. however whenever I get to see a fabric overview of what is connected to what and how stuff goes back and forth between initiators and targets it always (well around 99% of the time) looks like this.
Obviously this is a, so called, core-edge fabric with . I’ll ask you the following question: Why is this a bad design. On second thought, I’ll spare you making a list of flaws and turn the question around. Why is this a good design? There is only one answer and this is why: