Defaults
On the switches there are some default settings which you should enable before taking it into production.
First we enable virtual fabrics. (Some OEM’s have this enabled by default, others don’t). For the sake of simplicity just enable it.
SW0X:>fosconfig –enable vf
The switch will reboot and you will see a different command prompt showing up outlining the logical switch you’re currently in. As I mentioned before normally you don’t use the “Default switch” and only newly created logical switches are used for SAN connectivity.
Now two commands are used for configuring the individual chassis as well as each individual logical switch. The admin guides show you each option but I want to highlight a couple. Be aware that some options are chassis-wide and others needs to be applied on each logical switch in that chassis.
configurechassis
Config.Index
From the Custom Attributes there is a config Index setting. Do NOT change this. It a used as an OEM qualifier which enables Broade to set OEM specific values both at the factory and on firmware upgrades. You might get very disturbing results if you change this. Don’t do it.
system.blade.bladeFaultOnHwErrMask
On very rare occasions you may see an ASIC parity error. Unless you’re extremely paranoid this parameter will allow you to hard-fault a blade if an ASIC has encountered such a scenario. Once more, it’s extremely rare so better to leave this alone.
system.cpuload
You really should modify this and set it to 10. The reason is that when the CPU is very busy due to external management utilities or scripting the code may misinterpret some info from the SFP’s and flag it as a false positive. It then will initiate corrective action of which one is to drop light from the SFP and turn it back on again. You don’t want this to happen.
system.i2cTurboCnfg
As of FOS 6.3.2d, 6.4.2a and 7.0.0 and higher this is turned on by default on switches shipped by the factory. If you have 8G platform switches with older code turn this on and set it to 1. This is to prevent of application preventing SFP read operations having a negative impact on link behaviour. The common for issues is most often again aggressive polling by external applications and/or custom build scripts. This may overload the CPU and can cause negative side effects. Frames that do depend on CPU interference (such as most ELS frames for fabric services) may even get dropped. This then will result in login issues and hosts unable to see their disks and/or tapes.
Configure
Disable each logical switch and issue the “configure” command. The “configure” command allows you to adjust logical switch characteristics.
Fabric parameters
One of the most important one is the switch domain ID which can be set between 0 and 239 (0x00 and 0xEF). The other domain id’s 0xF0 to 0xFF are reserved for well know fibre-channel services.
256-area limit
Each logical switch by default changes to 10-bit addressing which basically means it steals 2 bits from the port id and adds it to the area id. If you connect a Ficon environment you need to set this back to 8-bit Zero-based address mode so that the area id (The centre nibble of the Fabric ID) can be used in the HCD configuration. (10-bit addressing mode is NOT supported in a Ficon environment.)
Do NOT (!!!!!!!) change ANY of the fabric timers (E_D_TOV, R_A_TOV, WAN_TOV). Unless you REALLY know what you’re doing you should leave these alone.
The next value that needs to be adjusted in this section is the EHT (Edge Hold Time). In our reference architecture you would set this value to 500 on the core switch and 220 on both the edge switches.
The last value that only needs to be modified is the IDID (Insistent Domain ID) mode. If this logical switch is servicing Ficon this needs to be enabled.
Virtual Channel Parameters.
Do not change any of these.
F-port login parameters.
Unless you have extremely dense VM farms and hosts are logging in HBA’s and virtual (NPIV) addresses at a pace you see connectivity issues occurring these are the setting you’ll need to adjust.
“Logins per second” – 10
“Login stage interval” – 100
“Stage F-disc logins with busy rejects” – 10
If you see ports flapping very often make sure you configure a form of portfencing. FLOGI’s (F-port logins) and FDISC (the way VM’s use to login to the fabric) are very CPU intensive and have fabric-wide reach. Adjusting routing tables and subsequent nameserver database distributions can have a profound impact on overall fabric operations. You do not want hundreds of VM’s login in and out of the fabric frequently.
This concludes the standard logical switch and chassis configuration.
In the example I’ve modified the domain ID’s to 225 and 226 on the edge switches and 193 on the core switch. (do a decimal to hex conversion and you’ll see why). I also adjusted the Edge Hold Time to 220 and 500 on the edge and core switches respectively to prevent credit back-pressure having a more than normal adverse effect in the entire fabric.
Before we enable the switches we need to do some non-standard configuration options. See the next chapter.
Pingback: tim_txcrd_z What is it? What does it represent? | Storage & Beyond
Hello Erwin,
thanks for the guide.
One question regarding the system.cpuload setting. Is changing it to 10 still true for FOS v7.x or other fixed versions referenced in TSB 068 and 111?
What I got from Brocade is that in any version containing the fix modifying the cpuload setting is not needed. Do you have more information about that?
Best regards
Michael
Hi Michael,
Indeed there has been improved logic in later code releases however I’ve still seen issues hitting the i2c bus (the same bus that is used when polling the sfp’s). By setting these values you can still prevent a process from hogging that bus. No guarantees if Brocade will hold on to this setting and values. The problem is most apparent when two or more management applications are polling the switches for info. I’ve seen one environment where 5 management applications plus two scripts where constantly polling the switches. This customer had run into an issue and wanted to be notified on multiple levels via various methods. Obviously the way they configured and used it was well beyond what a switch CPU could handle and they ran into even more issues. By killing off all but one management tool and reducing the polling frequency they went back to a more normal state. That management application was then configured via snmp and syslog forwarding to notify these other apps.
Hope this helps.
Regards,
Erwin
This is great material. The only thing I might do differently is matching up the switchnames to the domain IDs. I usually advise to give the switch a name that helps correlate it to the domain. That way they (and us support people) don’t have to remember
SW01 = 225 (e1)
SW02 = 193 (c1)
SW03 = 226 (e2)
I might go with something like SW-225, SW-193 and SW-226. Granted, you can see this mapping in ‘fabricshow’ (and it could complicate domain ID changes), but I just like knowing at a glance which domain I’m dealing with and it might help the customer make sure they’re sending in data from the right switch.
Thanks for all of these posts.
Hello Mike,
Thanks for your comments. Obviously you can use switchnames as to your liking. The reason I use the hex-values of the domain id’s is that I can more easily map each individual FCID to the respective switches/domains. In my case if i have an HBA connected to port 10 on switch 01 that FCID will become E10A00 so that shows me at a single view what I’m looking at. Also for scripting purposes this is very handy if you extract information out of a supportshow and use scripting tools who can calculate hex to dec values and reverse. I always used, or tried to, machined based information that is not susceptible to human interpretation. This prevents confusion if someone needs to take over. Surely there are more options and it’s up to the person who administers the environment what he/she want to use.
Regards,
Erwin