Energy Efficient Fibre Channel and related cost savings

For years many storage environments have used both active-active and active-passive multipath (MPIO) access mechanisms to access storage arrays in a dispersed or linear method. On enterprise class storage arrays with global caches the active-active method is most often used while on modular arrays you’ll see the active-passive scenario often applied. Inherently this means that during absence of IO, whether being the passive path or due to total non-IO operations (ie. there is no application or operating system sending or receiving any data), the actual fibre-channel links are only sending IDLE or ARB(ff) fillwords to maintain bit- and word synchronization. This also means that both the sender and receiver are always up and thus use the same amount of power as where they transmitting data at full line-rate. Obviously this is a waste of scarce resources and this is what has been addressed in the new FC standards that are coming up. The FC framing and signalling standard will be enhanced to have traffic diagnostics determine if an SFP should be in full power operating power or in a power reduced mode. Below are the details including some cost-savings calculations.

Most switches from Brocade and Cisco are able to extract powerusage of a single SFP. Depending on which type of SFP you have (shortwave/longwave/QSFP/mSFP etc) they all use between around 0.8 and 1.2 watts. Now, if you have a fully loaded DCX8510-8 with 64/8-port cards or a Cisco MDS9710 with 384 16G ports you have a significant amount of SFP’s humming and buzzing along each absorbing around 1 watt/h irrespective if they transport your data or are just maintaining synchronization.

no images were found

Port utilisation

Many switches that I see in the field are not 100% utilized from a port-count perspective so your first win could be there. When looking at an enabled port whether it has something plugged in or not you see that it still uses power. The actual current used for an enabled port if obtainable via commands that pull this info from the SFP itself. Brocade uses the sfpshow command and Cisco does it via “show interface transceiver details”.

So when a port is enabled but not used it looks like this:

  77    2   13   8f4d00   id    4G       No_Light    FC
Transceiver: 150c402001000000 1,2,4_Gbps M5,M6 sw Inter_dist
Current:     6.790   mAmps
Voltage:     3283.5  mVolts

The above is obviously a 4G short-wave SFP. When the same port is persistently disabled it looks like this:

  77    2   13   8f4d00   id    4G       No_Light    FC
Transceiver: 150c402001000000 1,2,4_Gbps M5,M6 sw Inter_dist
Current:     0.000   mAmps
Voltage:     3289.7  mVolts

On a Cisco Nexus 5K you see similarities:

Ethernet2/1
transceiver is present
type is 10Gbase-SR
name is CISCO-FINISAR
<snip>
Transceiver Detail Diagnostics Information (internal calibration)
—————————————————————————-
Current              Alarms                  Warnings
Measurement     High        Low         High          Low
—————————————————————————-
Temperature   40.58 C        93.00 C    -14.00 C     88.00 C       -9.00 C
Voltage        3.25 V         3.70 V      2.90 V      3.59 V        3.00 V
Current        7.00 mA       11.80 mA     4.00 mA    10.80 mA       5.00 mA
Tx Power       -2.94 dBm       0.00 dBm   -6.00 dBm   -1.00 dBm     -5.00 dBm
Rx Power       -1.88 dBm       0.00 dBm  -16.02 dBm   -1.00 dBm    -14.08 dBm
—————————————————————————-
Note: ++  high-alarm; +  high-warning; —  low-alarm; –  low-warning

After shutting down the interface it looks like this:

Nexus5548-HNAS(config-if)# do show inte ethernet 2/3 trans de
Ethernet2/3
transceiver is present
type is 10Gbase-SR
name is CISCO-FINISAR
<snip>
Transceiver Detail Diagnostics Information (internal calibration)
—————————————————————————-
Current              Alarms                  Warnings
Measurement     High        Low         High          Low
—————————————————————————-
Temperature   45.58 C        93.00 C    -14.00 C     88.00 C       -9.00 C
Voltage        3.22 V         3.70 V      2.90 V      3.59 V        3.00 V
Current        0.00 mA  —   11.80 mA     4.00 mA    10.80 mA       5.00 mA
Tx Power        N/A     —    0.00 dBm   -6.00 dBm   -1.00 dBm     -5.00 dBm
Rx Power       -2.49 dBm       0.00 dBm  -16.02 dBm   -1.00 dBm    -14.08 dBm
—————————————————————————-
Note: ++  high-alarm; +  high-warning; —  low-alarm; –  low-warning

Obviously if you have multiple high power SFP’s like ELWL these will use more power but in general these do not sit idle anyway and are really connected.

So in short it will not only provide you with a good security practice, a disabled port doesn’t allow any device or switch to be connected, but also saves you power consumption.

Effective link utilisation

This still doesn’t fix the fact that enabled and active ports may have the same characteristics. They do nothing but keeping synchronized whilst still absorbing power. To resolve this EEFC or Energy Efficient Fibre Channel has been added to the FC-FS-4 standard. The technicalities originate from the friends at IEEE who have started on this technology back in 2006 by creating the 802.3az standard. The original intent was first to adopt the technology in home and small office products and evaluate the findings over a few years before expanding this into the large scale datacentre systems.

Basically it allows a transmitter or receiver to change into a so called LPI (Low Power Idle) mode if certain threshold timers allow this. When a transmitter has no data (either yours or frames that regulate fabric or link services) it can shut down certain features of that port or even turn off the transmitter totally. (depending if the SFP supports this obviously)

A port can only change to LPI mode when it is in the Active state. Whenever the PSM (Port State Machine) is not in an Active state all EEFC functions cannot be used. You cannot turn off a transmitter when you’re in the middle of a recovery sequence of course.

An effective link utilization is between 90% and 100% of the maximum link speed capability. As soon as port reach that level you are most likely to see congestion issues whereby conflicting traffic patterns may cause frame drop due to timeouts. It is therefore that in most fabric the utilisation rate is kept at a lower pace but increasing the physical amount of links between hosts and storage arrays so that traffic can be more evenly dispersed and a more reliable traffic flow is obtained. If therefore the effective link utilization rate has dropped to between 40% and 60% during busy times and between 20 and 40% during “quiet” times you can see that the savings on power-consumption are quite substantial. Lets put this to the test then.

The numbers

Let’s say I have a single director class switch with 384 ports and each SFP uses 1 watt of power. All ports are in use. The busy period account for 40% and the quiet period account for 60% of the time and the effective utilization rate is 60% and 30% respectively.

Putting this in a table you’ll see the numbers:

Ports 384 U-rate Raw Usage (watts) Effective usage (Watts)
Busy time 40% 60% 153.6 92.16
Quiet 60% 30% 230.4 69.12
Total effective usage 161.28
Savings (watts) 222.72

That means I can achieve a saving of 58% by just enabling the energy-efficient feature of the FC protocol. That is quite substantial. If you’re running a data-centre with thousands of ports you will notice the difference on your power-bill pretty quickly.

Products

Currently there are no products on the fibre-channel market yet that have these energy efficient capabilities but you will see that in the near future this will become a standard feature and another pathway towards a greener data-centre.

Regards,

Erwin van Londen

Print Friendly, PDF & Email

Subscribe to our newsletter to receive updates on products, services and general information around Linux, Storage and Cybersecurity.

The Cybersecurity option is an OPT-OUT selection due to the importance of the category. Modify your choice if needed.

Select list(s):