![4
1. Introduction
Changing data center environmental conditions are of importance to IT equipment but also to
power equipment, especially where the two types of equipment share the same physical space and
air stream. ASHRAE’s document [1], “Thermal Guidelines for Data Processing Environments–
Fourth Edition” has increased the industry’s awareness of the effect increased operating
temperature can have on IT equipment. In some cases, power equipment can be subjected to higher
temperatures than the IT equipment. Higher temperatures can impact equipment reliability.
Exposure to warmer temperatures, coupled with the fact that usable life cycle of power equipment
is typically longer than IT equipment, increases the importance of this topic.
This paper discusses how changes to the data center thermal environment may affect power
distribution equipment. This paper also provides an overview of data center power distribution [2]
[3] and describes the typical power equipment used for both IT loads and non-IT loads (i.e. lighting
and cooling). Included in this list of equipment is switchgear, uninterruptible power supplies
(UPS), static transfer switches, switchboards, transformers, power distribution units (PDU),
remote power panels (RPP), panelboards, rack PDU, line cords, facility receptacles and IT cable
trays. Note that the order in which the power distribution equipment is discussed is not necessarily
the same order in which it would appear on an electrical diagram. The paper concludes with a set
of recommendations on how to improve power equipment thermal compatibility and reliability.
1.1 Typical Data Center Power Distribution System
An electrical one-line diagram is typically used to communicate specific details of an electrical
distribution design and shows the logical flow of electricity from utility mains to the IT equipment.
A block diagram, shown in Figure 1, provides a higher-level view of a data center’s electrical flow
without the complexity of a one-line diagram. Figure 1 serves as a guide to show where certain
types of equipment are typically found within a data center, both logically and physically. Dashed
lines are used to show in which type of space each piece of power equipment resides. Heavy
dashed lines indicate electrical space, lines with a dash and a single dot delineate mechanical space,
and the line with a dash and two dots defines the IT space. Generators, though part of the data
center power infrastructure, are beyond the scope of this paper.
In general, data center electrical architectures start with a utility supply at medium-voltage (600 to
1000V) which feeds medium-voltage switchgear. Note: the new version of the National Electrical
Code (NFPA 70 [4]), which will be released in 2017, will define medium voltage as 600 to 1000V.
This medium-voltage is “stepped down” to low-voltage (typically 480V and lower) using a
transformer which then feeds low-voltage switchgear. Low-voltage switchgear generally feeds
UPS units which then feed UPS output switchgear or panelboards and then feed room PDUs.
Room PDUs typically provide power to remote power panels (RPPs) which supply branch circuits
to each IT rack which then feed rack PDUs (i.e. power strips).
Working back up the power supply chain to the utility feed, there is a similar chain of power
distribution equipment used for non-IT loads such as lighting and cooling equipment. As you go
higher in the electrical architecture, the more likely it is that the equipment (at the medium voltage
level) feeds both IT and non-IT equipment.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-4-320.jpg)
![5
Figure 1 Overview of a typical data center power distribution system [2]. Used with permission.
1.2 Categorizing Data Center Power Equipment by Location
Data center power distribution equipment can be categorized a number of different ways but for
the purposes of this paper, it is helpful to categorize power equipment by location and by power
path. The location of power equipment (e.g. mechanical space, IT space, cold aisle, hot aisle) will
determine its usage environment. It is also helpful to know whether the equipment is in series with
critical IT equipment (i.e. a UPS) or in parallel (i.e. lighting power panel) since this may influence
the selection of the power equipment. Power equipment generally has a longer usage lifecycle
than IT equipment.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-5-320.jpg)
![10
Structured Cabling (low
voltage IT cabling, fiber and
copper)
Sometimes Hot, cold aisle,
ballroom or under
floor
Varies widely
Servers YES Cold aisle 3 to 8 years
Storage Arrays YES Cold aisle 5 to 8 years
Networking YES Cold aisle 5 to 10 years
*These lifetimes are based on equipment that is properly maintained and used in an appropriate
application and environment.
**These items are usually replaced because of changes in data center power capacity that require
a change in power cable and bus bar sizing, but not because these items wear out.
As shown in the right-hand column of Table 1, most power equipment stays in service longer than
the IT equipment it supports. Power equipment can have a wider variety of physical locations than
IT equipment. The environments for power equipment are also less homogeneous than for IT
equipment and they include hot and cold aisle, separate grey spaces, battery rooms, as well as the
outdoors. Most power equipment is natural convection cooled with the exception of UPS modules
and some transformers, whereas IT equipment is typically cooled by forced air from a set of
variable speed internal fans.
The intent of the ASHRAE thermal guidelines [1] was to provide guidelines for IT equipment.
Power equipment was not a primary consideration in the writing of those thermal guidelines. As
will be discussed later, the way power equipment specifications are written is quite a bit different
than specifications for IT equipment. Ensuring compatibility between the two types of equipment
is not always a simple exercise of comparing ambient air temperature, relative humidity and dew
point values. Both types of equipment, power and IT, need to be compatible with the environment
of the physical space where they are located. Some power equipment located is located in the hot
aisle exhaust of the IT equipment which is a different environmental condition than the classes
described in the ASHRAE thermal guidelines. In general the thermal and environmental
requirements for power equipment are much less stringent than for IT equipment.
2. Changes in Data Center Environments
Changes to data center environmental conditions are being driven by the need to save energy and
reduce operational expenses. One of the largest operational expenses in delivering IT is the cost
of energy. In a traditional data center, cooling costs alone can easily represent 25% or more of
total energy costs [5]. Many data centers now run several degrees warmer compared to 10 or 15
years ago to save on cooling costs [6]. New energy saving technologies such as air-side
economization and water-side economization are growing in adoption, whose hours of beneficial
use increase as computer room temperatures increase.
2.1 ASHRAE Thermal Guideline Classes for IT Equipment Spaces
ASHRAE issued its first thermal guidelines for data centers in 2004 [1]. The original ASHRAE
air temperature recommended envelope for data centers was 20-25°C (68-77°F). This was a
conservative statement, based on data available at the time, on where a data center could be reliably](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-10-320.jpg)
![11
operated. Reliability and uptime were the primary concerns and energy costs were secondary.
Since then, ASHRAE has issued a recommended range of 18-27°C (64-81°F) and, in 2011,
published classes [1] that allow a temperature range of 5 to 45°C (41 to 113°F). The A3 (40°C,
104°F) and A4 (45°C, 113°F) classes were created to support new energy saving technologies such
as economization. A summary of the ASHRAE recommended range and classes are given in the
table below.
Table 2 ASHRAE 2015 Thermal Guidelines [1]
Classa
Equipment Environmental Specifications for Air Cooling
Product Operationsb,c
Product Power Offc,d
Dry-Bulb
Temperaturee,g
˚C
Humidity Range,
Non-Condensingh,i,k,l
Maximum
Dew Pointk ˚C
Maximum
Elevatione,j,m
m
Maximum
Temperature
Changef in an
Hour (˚C)
Dry-Bulb
Temperature
˚C
Relative
Humidityk %
Recommended (Suitable for all 4 classes)
A1 to
Α4
18 to 27
-9˚C DP to 15˚C
DP and 60% RH
Allowable
A1 15 to 32
-12˚C DP & 8%
RH to 17˚C DP
and 80% RHk
17 3050 5/20 5 to 45 8 to 80
A2 10 to 35
-12˚C DP & 8%
RH to 21˚C DP
and 80% RHk
21 3050 5/20 5 to 45 8 to 80
A3 5 to 40
-12˚C DP & 8%
RH to 24˚C DP
and 85% RHk
24 3050 5/20 5 to 45 8 to 80
A4 5 to 45
-12˚C DP & 8%
RH to 24˚C DP
and 90% RHk
24 3050 5/20 5 to 45 8 to 80
B 5 to 35
8% to 28˚C DP
and 80% RHk 28 3050 NA 5 to 45 8 to 80
C 5 to 40
8% to 28˚C DP
and 80% RHk 28 3050 NA 5 to 45 8 to 80
Notes for Table 2, 2015 Thermal Guidelines—SI Version
a. Classes A3, A4, B, and C are identical to those included in the 2011 edition of Thermal Guidelines for Data Processing Environments.
The 2015 version of the A1 and A2 classes have expanded RH levels compared to the 2011 version.
b. Product equipment is powered ON.
c. Tape products require a stable and more restrictive environment (similar to 2011 Class A1). Typical requirements: minimum
temperature is 15°C, maximum temperature is 32°C, minimum RH is 20%, maximum RH is 80%, maximum dew point is 22°C, rate of
change of temperature is less than 5°C/h, rate of change of humidity is less than 5% RH per hour, and no condensation.
d. Product equipment is removed from original shipping container and installed but not in use, e.g., during repair, maintenance, or
upgrade.
e. Classes A1, A2, B, and C — De-rate maximum allowable dry-bulb temperature 1°C/300m above 900m. Above 2400m altitude, the
de-rated dry-bulb temperature takes precedence over the recommended temperature. Class A3 — De-rate maximum allowable dry-bulb
temperature 1°C/175m above 900m. Class A4 — De-rate maximum allowable dry-bulb temperature 1°C/125m above 900m.
f. For tape storage: 5°C in an hour. For all other ITE: 20°C in an hour and no more than 5°C in any 15 minute period of time. The
temperature change of the ITE must meet the limits shown in the table and is calculated to be the maximum air in let temperature minus
the minimum air inlet temperature within the time window specified. The 5°C or 20°C temperature change is considered to be a
temperature change within a specified period of time and not a rate of change. See Appendix K of reference [1] for additional information
and examples.
g. With a diskette in the drive, the minimum temperature is 10°C (not applicable to Classes A1 or A2).](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-11-320.jpg)
![12
h. The minimum humidity level for Classes A1, A2, A3, and A4 is the higher (more moisture) of the 12°C dew point and the 8%RH.
These intersect at approximately 25°C. Below this intersection (~25°C) the dew point (–12°C) represents the minimum moisture level,
while above it, RH (8%) is the minimum.
i. Based on research funded by ASHRAE and performed at low RH, the following are the minimum requirements:
1) Data centers that have non-ESD floors and where people are allowed to wear non-ESD shoes may want to consider increasing
humidity given that the risk of generating 8kV increases slightly from 0.27% at 25% RH to 0.43% at 8% RH (see Appendix D of
reference [1] for more details).
2) All mobile furnishing/equipment is to be made of conductive or static dissipative materials and bonded to ground.
3) During maintenance on any hardware, a properly functioning and grounded wrist strap must be used by any personnel who contacts
ITE.
j. To accommodate rounding when converting between SI and I-P units, the maximum elevation is considered to have a variation of
±0.1%. The impact on ITE thermal performance within this variation range is negligible and enables the use of rounded values of 3050m
(10,000ft).
k. See Appendix L of reference [1] for graphs that illustrate how the maximum and minimum dew-point limits restrict the stated relative
humidity range for each of the classes for both product operations and product power off.
l. For the upper moisture limit, the limit is the minimum absolute moisture of the DP and RH stated. For the lower moisture limit, the
limit is the maximum absolute moisture of the DP and RH stated.
m. Operation above 3050m requires consultation with IT supplier for each specific piece of equipment.
Applying the ASHRAE thermal guideline classes listed in Table 1 looks straightforward but, in
practice, an older data center may have equipment that was designed to different versions of those
guidelines. For example, the recommended envelope from the 2004 ASHRAE guidelines was 20
– 25°C (68-77°F) whereas newer equipment is typically designed to Class A2 (35°C, 95°F) or even
A3 (40°C, 104°F) and A4 (45°C, 113°F). The life of a typical data center is 15 to 20 years so some
of the data center equipment may be original to the time the data center was built and some will
have been recently refreshed. One of the challenges with implementing new energy saving cooling
technologies, such as economization, is identifying an environmental control window that is
compatible with all of the different thermal specifications, types of equipment, and equipment
vintages in the data center.
2.2 Increasing Use of Economization in IT Equipment Spaces
Economization is the use of outdoor air to cool the data center [7].The premise of economization
is simple – why run an air conditioner or chiller if there is already an unlimited supply of cold air
outside? Many world-wide locations can economize for as much as 50% of the hours in a year
within the ASHRAE recommended range of 18-27°C (64-81°F). Larger percentages of
economization (energy savings) require a wider allowable temperature and humidity range.
There are several forms of economization: air-side economization [7], water-side economization
[7], and refrigerant economization. Air-side economization typically accomplishes cooling by
bringing filtered outside air directly into the data center without any air conditioning or humidity
control. In winter months when the outside air is cold, a comfortable working temperature inside
the data center is maintained by mixing the incoming air with hot exhaust air from the IT
equipment. It should be noted there are forms of indirect air-side economization that use a thermal
wheel, heat pipes, or a plate air–to-air heat exchanger [8]. These indirect air-side economization
methods bring only a very small amount of outside air into the IT space of the data center. The
disadvantage of a wheel or plate is the heat exchange step required makes it less efficient in full
economizer cooling mode compared to air-side economization. The disadvantage of direct air-
side economization is, depending on the local air quality, it can bring pollutants into the data center
[9] which can require additional filtration to remove. Another disadvantage of direct air-side
economization is, in some climates, the humidity of the outdoor air is outside the recommended
(or allowable) limits for the IT equipment. This can preclude use of air-side economization, even](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-12-320.jpg)
![13
when the outdoor temperature is favorable. Direct air-side economization can also make data
centers prone to larger rates of change of temperature and humidity. Water-side economization
uses outdoor air for data center cooling, but indirectly. Outdoor air is used to chill liquid and the
liquid is piped into the data center where air-handling units (AHUs) and/or computer room air
handlers (CRAHs) use an air-liquid heat exchangers called cooling coils to cool the data center air.
The advantage of water-side economization is it doesn’t bring outside air into the data center and
it allows tight humidity control similar to an HVAC cooled data center. Refrigerant economization
is a variant of air conditioning in which the refrigerant bypasses the compressor on the way back
to the condenser outside the data center [10] [11]. The energy that would have been used by the
compressor is what is being saved. A purpose-built air conditioning system is required for
refrigerant economization.
The efficiency of both air-side and water side economization can be improved by the addition of
evaporative or adiabatic cooling. For water-side economization, the cooling of the water can be
improved with the use of an open tower where the heat exchange surfaces are wet and the
evaporation of water improves the cooling efficiency. Evaporative cooling is also sometimes used
with direct air-side economization - the air entering the data center is passed through an
evaporative cooling media where the air is cooled by the addition of more moisture. An example
of a direct air-side economized evaporative cooled modular data center is shown below in Figure
5.
Figure 5 Air-side economized modular data center with evaporative cooling module.
2.3 Rising Exhaust Temperature of IT Equipment
In addition to energy saving improvements in data center cooling, significant improvements have
also been made in the energy efficiency of the IT equipment itself. One of the most common
energy saving improvements is in the fan speed algorithm the equipment uses to regulate fan speed.
Most IT equipment now has a control algorithm that reduces the fan speed at low workloads when
very little cooling is needed. As workloads increase these new energy efficient fan algorithms
increase the fan speed only as much as needed to keep component and sub-assembly temperatures
inside the equipment within design limits. The consequence of this design change has been an
increase in exhaust air temperatures of almost 10°C (18°F) over the last 8 years (see Figure 6).](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-13-320.jpg)
![14
Exhaust temperature rise for air inlet temperatures of 35°C (95°F) and 45°C (113°F) are shown in
Figure 7 and Figure 8. As the inlet air temperature rises to 35°C (95°F) and 45°C (95°F), most
servers reach a maximum exhaust air temperature somewhere between 58 and 60°C (140°F). For
power distribution infrastructure (i.e. line cords, facility receptacles, etc.) located in a hot aisle or
exhaust path of IT equipment there may be operating limitations due to: 1) maximum rated
temperature of a component, which may be addressed by using de-rating, or 2) maximum ambient
temperature as referenced in some IEC standards like IEC 60309-1 [12], 60309-2 [13], 60320, etc.
[14]. Many ITE manufacturers provide Unified Extensible Firmware Interface (UEFI)/BIOS
settings to increase fan speeds to increase cooling and reduce exhaust temperatures to alleviate this
concern. Contact your specific IT Equipment manufacturer for more information. In some
applications, a higher temperature rated appliance or interconnection coupler may be required.
In the future more harmonization of product safety standards, such as IEC 60950-1, may be
desirable to embrace maximum rated temperature versus an ambient environment requirement
with temperature rise since in an IT environment, the local temperature of a component is a
combination of ambient, pre-heat, temperature rise, etc.
Figure 6 Server T projection at 25°C (77°F) server inlet temperature [15].](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-14-320.jpg)
![15
Figure 7 Server T projection at 35°C (95°F) server inlet temperature [15].
Figure 8 Server T projection at 45°C (113°F) server inlet temperature [15].
Because of space and air flow constraints, the exhaust air stream is sometimes used to cool power
equipment such as rack PDUs and IT equipment such as top of rack switches. However, with
the >10°C (18°F) rise in exhaust air temperature, the exhaust air stream may no longer be suitable
for cooling or the equipment in the hot exhaust air stream may need to be redesigned to
accommodate the higher exhaust air temperatures. Power equipment needs to tolerate recent
changes in the data center environment such as the increase in exhaust air temperatures. The
industry needs to make sure the thermal specifications of power equipment are aligned to the
specifications of the IT equipment, where both types of equipment share the same physical space
and same air stream.
2.4 Air Temperature Trends in IT Support Equipment Spaces
IT support equipment spaces include areas for power equipment, a battery room for UPS batteries,
and rooms for mechanical equipment such as air handlers. In general, these spaces have an](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-15-320.jpg)
![19
Another important factor that can impact the maximum ambient air temperature for power
equipment is touch temperature. The temperature of handles, receptacles, and flat surfaces a user
might touch needs to stay within comfort and product safety limits. A widely used international
touch temperature standard is IEC60950-1 shown below in Table 3.
Table 3 Touch temperature limits from IEC60950-1 [16].
Object Being
Touched
Touch
Duration
Metal Glass and
Similar
Plastic and
Rubber
Handles Briefly 60°C 70°C 85°C
Handles Longer
Duration
55°C 65°C 75°C
External surfaces and
parts inside that may
be touched
NA 70°C 80°C 95°C
Note: The exact duration of the terms “briefly” and “long duration” are not defined in IEC60950-
1.
For some power equipment, the maximum ambient air temperature rating is a combination of
conductor temperature rise and touch temperature standards.
4. Medium and Low Voltage Switchgear
Medium and low voltage switchgear is typically located after the utility and before the UPS gear
(see Figure 1). In a data center power distribution system, the medium and low voltage switchgear
is a combination of electrical disconnect switches and circuit breakers used to control, protect, and
isolate electrical equipment. Examples of a low voltage switchgear and distribution switchgear
are shown below in Figure 10 and Figure 11.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-19-320.jpg)
![20
Figure 10 Low voltage switchgear
Figure 11 Distribution switchgear
Most data center switchgear equipment is natural convection cooled. Perforated panels or grilles
are strategically located to facilitate natural convection based cooling. The amount of heat
generated inside the switchgear is dependent on loading. The power carrying capability of the
components inside the switchgear is a function of temperature, decreasing as temperature of the
components increases. Switchgear is carefully designed to meet maximum rated loads with natural
convection cooling alone and minimal temperature rise of components.
Per the IEC 61439 standard for low-voltage switchgear and control gear assemblies [17], the
ambient temperature specification for low and medium voltage switchgear is a 35°C (95°F)
maximum air temperature with a 2 hour excursion up to a 40°C (104°F) air temperature allowed.
Switchgear can be susceptible to high levels of humidity near the condensing limit. In some cases,
high humidity levels can cause leakage and even arcing. However, most switchgear is designed
with a hygrostat and a panel heater to avoid condensation. By heating the air inside the switchgear
cabinet the panel heater reduces the relative humidity of the air effectively mitigating the effect of
the high humidity and preventing condensation. Most switchgear has arc protection relays and
sensors as well as a ground fault relay to protect the switch in case it is inadvertently exposed to
very high levels of humidity.
In general, most electrical power distribution equipment, including switchgear, generates very
little heat. The main exceptions to this are transformers and UPS systems. Switchgear rooms
require only a small amount of cooling which can be provided by a number of different means but
typically by traditional air conditioning. Care must be exercised in designing the cooling system if
the switchgear is in the same room or shares an air supply with batteries. Batteries can generate
hydrogen and building safety codes usually require a minimum rate of air exchange with the air](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-20-320.jpg)
![21
exhausted directly to the outdoors. In some northern climates, it may be possible to re-use the
waste heat from some power equipment, such as transformer and UPS spaces, to heat an adjacent
space such as a battery room.
Airborne contamination and resulting corrosion [9] can affect the reliability and the lifetime of
switchgear. High humidity can accelerate the effect of many types of airborne corrosion. High
levels of outdoor air pollution in newly industrialized parts of the world can create slightly
corrosive environments inside data centers and the power spaces of those data centers. Most
switchgear is designed to have a moderate amount of resistance to corrosion. The most common
corrosion design standard is class C1 of ISO 12944 [18]. Switchgear is also tested for airborne
salt dust and most metal parts are protected by special coatings.
5. Uninterruptible Power Supplies
The purpose of an uninterruptible power supply unit is to provide emergency power when the input
power source, typically mains utility, fails. A UPS differs from an auxiliary or emergency power
system or standby generator in that it will provide seamless protection from input power
interruptions, by supplying energy stored in batteries, super-capacitors, or flywheels. The runtime
of a battery based uninterruptible power source is highly dependent on the loading and can range
from 15 minutes at full load to longer times at reduced loading. Some battery based telecom UPS
units are designed to provide 8 hours or more of run time. Flywheel and super-capacitor based
UPS units provide power for a duration of 15 to 30 seconds to a few minutes which is relatively
short, but sufficient to start a standby power source or properly shut down the protected equipment.
UPS units are used to protect data centers against unexpected power disruptions that could result
in service interruptions or even data loss. References [19] and [20] have a good discussion of
different UPS system design configurations and configurations that make the most sense for a
given set of parameters including availability needs, risk tolerance, types of loads in the data center,
budgets, and existing infrastructure.
Uninterruptible power supply units come in a variety of sizes: monolithic, single and multi-rack,
and modular rack mount systems (see Figure 12, Figure 13, and Figure 14) as well as even smaller
units that fit in place of an IT equipment power supply inside the IT equipment.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-21-320.jpg)
![22
Figure 12 Large monolithic data center UPS.
Figure 13 Examples of smaller rack-based UPS units.
Figure 14 Example of a small rack mount UPS.
Uninterruptable power supplies utilize a variety of energy storage technologies including: battery
(see Figure 15), flywheel (see Figure 16), and super-capacitors [21]. One of the important
differences in these energy storage technologies is the time scale over which they can provide
backup power. In general, batteries provide the longest duration of backup power, typically at
least 15 minutes and sometimes hours or more. Flywheels typically provide power on the order
of 15 to 30 seconds at rated load (and in some cases up to several minutes especially at low load
conditions). Most super-capacitors are capable of providing power for much less than a minute
[21].
It should also be noted that energy storage capacity (batteries, flywheels, etc.) are typically sized
to meet a time requirement based on a maximum full rated load. When there are redundant UPS
systems then each system is sized for the rated load and therefore during normal operations where
both systems are supporting half the actual load, the installed capacity is doubled. When the actual
load is less than the full design rated load as is the case in all but the most heavily loaded systems,
the spare capacity can be significantly greater still.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-22-320.jpg)
![24
valve regulated lead acid (VRLA). Lead acid batteries are a proven technology and they are one
of the most cost effective storage technologies available.
The backup power source for large monolithic UPS systems can also be comprised of a number of
flywheel units. Flywheel systems can tolerate a much wider temperature range (0 to 40°C, 32 to
104°F) than batteries (15 to 25°C, 59 to 77°F). As with batteries, flywheel systems have relatively
low power dissipation and require only a small amount of cooling compared to IT equipment.
Flywheels can be a cheaper alternative to batteries when only short term (on the order of a minute)
back up power is required. The disadvantage is flywheel systems provide power only long enough
for a diesel generator to start up and take over providing power to the IT equipment. Most
communities have limits on the number of hours a facility can run diesel generators. Like battery
based UPS systems, large monolithic flywheel UPS systems are also in their own separate
environmentally controlled section of the data center where they are unaffected by the temperature
of the air streams associated with the IT equipment.
The next type of UPS solution is a distributed UPS solution. These UPS units are designed into
data center style racks and, when located in data halls, can share the same space and air streams
with the IT equipment they support. When considering a distributed UPS solution, there are
several factors to weigh, including: battery technology, electrolytic capacitors and semiconductor
junction temperature. Distributed UPS units are typically very compatible with IT equipment –
nearly all distributed UPS units pull air from the cold aisle and they all use variable speed fans that
are controlled via internal thermal sensors. There is a long history of robust thermal design with
this type of UPS. The most common battery technology for distributed UPS units is valve regulated
lead acid (VRLA) batteries. VRLA batteries are different from flooded/vented lead acid batteries
in that they emit little or no hydrogen gas and don’t require a special room or special ventilation
requirements.
The third type of UPS unit is a small battery backup (BBU) that resides inside the IT equipment,
usually with the same form factor as a power supply unit. For BBUs, the most common battery
technology is lead-acid VRLA though some BBUs use lithium ion (LiON) batteries [22]. One of
the reasons for choosing a battery technology, like LiON, is that it is tolerant of higher temperature.
The location of the BBU is typically at the rear of the IT equipment where the cooling air has
already been significantly pre-heated. Air temperatures in the range of 50o
C (122°F) are not
uncommon in the rear chassis of most IT equipment. Temperatures above 50o
C (122°F) will
adversely affect the service life of the battery. Another important aspect of defining battery
temperature is, during discharge mode, the batteries may have as much as a 30°C (54°F) rise in
temperature, depending on the current draw and the discharge time. After a discharge, the batteries
must be allowed to cool before a charge cycle can be initiated. After a discharge event there will
be a time window where the IT equipment is vulnerable to a back-to-back outage event where, for
the second outage, the BBU would not have energy stored energy to provide back-up power. IT
equipment BBUs are inside the data center space and they will experience the same air streams,
temperature and humidity conditions as would a power supply inside a piece of IT equipment.
This type of backup power deployment is not very common but it illustrates the challenges of
placing a batteries inside IT equipment where a significant amount of preheating can take place.
In the case of a BBU inside a piece of IT equipment, VRLA batteries would not be a suitable
choice.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-24-320.jpg)
![25
Discussion of Super-Capacitor Technology for UPS Systems
An important class of charge storage devices is the so called “ultra-capacitors” or “super-capacitors”
[23]. These devices are used in some UPS systems to provide short term power back-up. The
most common type of super-capacitor is an electric double-layer capacitor (EDLC). Super-
capacitors are an attractive alternative to lead-acid batteries because they have: a) a large number
cycle times along with fast charging times, b) higher power density and faster energy release, c)
are smaller and lighter than batteries, d) are environmentally friendly, and e) generally have lower
maintenance costs and higher reliability than batteries. However, super-capacitors have some
significant disadvantages relative to batteries and other types of capacitors including: a) low
operating voltages, b) shorting hazard due to low internal resistance, and c) elevated temperatures
tend to shorten the life of super-capacitors more than batteries or electrolytic capacitors. A
quantitative comparison of super-capacitor performance parameters to lead acid batteries and
electrolytic capacitors is shown below in Table 4. Note how the maximum temperature rating for
super-capacitors is only 70°C (158°F) whereas most electrolytic capacitors are rated to 85 or 105°C
(185 or 221°F). Also, the typical rated super-capacitor lifetime of 1,000 hours sounds like a lot
but it is only 6 weeks (at the maximum rated temperature of 70°C (158°F)).
Table 4 Comparison of super-capacitor performance to lead acid batteries and electrolytic
capacitors [23].
Performance
Parameter
Lead Acid Battery Super-Capacitor Electrolytic
Capacitor
Charge Time 1 to 5 hours 0.3 to 30 s 10-3
to 10-6
s
Discharge Time 0.3 to 3 hours 0.3 to 30 s 10-3
to 10-6
s
Energy Density
(Wh/kg)
10 – 100 1 – 10 <0.1
Power Density
(W/kg)
<1,000 <10,000 <100,000
Cycle Life 1,000 >500,000 >500,000
Charge/discharge
efficiency
0.7 – 0.85 0.85 - 0.98 >0.95
Operating
Temperature Range
15 to 25°C
(60 to 77°F)
-40 to 70°C -40 to 105°C
Typical Lifetime
Specification Rating
3 to 20 years 70°C for 1,000 hours 85 or 105°C for 1,000
or 2,000 hours
Consider, for example, a super-capacitor rated for 1,000 hours (6 weeks) at 70°C (158°F) and 2.7V
[24]. According to the manufacturer, the lifetime of the capacitor doubles for every 10°C below
the maximum rated temperature of 70°C (158°F). Using the capacitor below its maximum rated
voltage increases the lifetime by 1X for 2.7V (maximum rating), 4.5X for 2.25V, and 16X for
1.8V. The adjusted lifetime of the capacitor is the product of: a) the lifetime extension factor from
temperature, b) the lifetime extension factor from reducing the voltage, and c) the base lifetime of
the component, in this case 1,000 hours. Assume, for sake of this example, the UPS unit containing
the super-capacitors, needs to last for at least 10 years. In order to reach a design goal of 10 years,](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-25-320.jpg)
![26
the super-capacitors need to be de-rated, i.e. used at less than their maximum temperature and
voltage rating in order to increase the component lifetime beyond 1,000 hours (6 weeks).
Components inside a UPS chassis can generate some heat so the temperature of the super-capacitor
components may be slightly higher than the temperature of the surrounding air outside the UPS
cabinet. This temperature rise needs to be carefully measured and taken into account for lifetime
calculations. A significant amount of voltage and temperature de-rating needs to be applied to the
super-capacitors to achieve a >10 year lifetime as shown in Figure 17 below. The de-rating
constraints to achieve 10 years are such that the super-capacitors cannot be used at their maximum
rated voltage of 2.7V. For operational voltages of 2.25 and 1.8V a 10 year lifetime is met only if
the maximum super-capacitor component temperature is maintained at or below 27°C (81°F) and
46°C (115°F), respectively.
Figure 17 Illustration of lifetime de-rating trade-offs to meet a 10 year design goal for a super-
capacitor with a 1,000 hour rating at 70°C (158°F) and 2.7V [24].
Another important point to consider for the application of super-capacitors is the non-operational
(both shipping and storage) temperature range of these components. The recommended storage
temperature for some super-capacitors is as low as 35°C (95°F) [25]. This is well below the 45°C
(113°F) temperature listed in the power off conditions of ASHRAE classes A1, A2, A3 and A4.
Some types of super-capacitors are also sensitive to high humidity levels in the range of 70%RH
and above [25]. Thus, under some circumstances, it could be possible for a piece of UPS equipment
to be shipped to a customer, see high temperatures and high humidity levels for a prolonged period
of time during transit, and arrive at a data center site with compromised super-capacitors.
UPS equipment designers should pay close attention to the temperature, voltage and lifetime
ratings of super-capacitor components. Supplier provided component de-rating guidelines should
be carefully followed. Environmental usage conditions (temperature, humidity) in customer facing
user manuals for the UPS unit should align with the data sheet of the super-capacitors and the de-
rating assumptions applied in the equipment design. Both customers and UPS equipment suppliers
should pay close attention to non-operational shipping and storage conditions, especially during
warm summer months. UPS units with super-capacitors should be shipped and stored with a
temperature/humidity data logger to verify that none of the non-operational temperature and](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-26-320.jpg)
![27
humidity limits for the super-capacitors were exceeded. Data center operators should read UPS
user manuals carefully and maintain the temperature and humidity of IT spaces and grey area
spaces well within those published ranges to ensure the full design lifetime of the super-capacitors.
Discussion of Battery Technologies for UPS Systems
Two battery technologies are in widespread use in UPS systems: flooded/vented lead acid, and
valve regulated lead acid (VRLA). Lithium ion (LiON) batteries are not yet in widespread use but
are considered a promising new technology. A high level comparison of the different battery types
is given below in Table 5.
Table 5 Temperature and service life ranges for common UPS battery technologies.
Flooded Vented
Wet Cell Lead Acid
Valve Regulated
Lead Acid (VRLA)
Lithium Ion
(LiON)
Temperature Range 15 to 25°C
(60 to 77°F)
15 to 25°C
(60 to 77°F)
-40 to 60°C
(-40 to 140°F)
Recommended
Service Life Range
15 to 20 years 3 to 10 years* 6 to 20 years
*Newer pure lead VRLA batteries can support an even longer service life.
Most flooded/vented lead acid and valve regulated lead acid batteries (VRLA) are lead-acid based
technologies but with key differences. The electrolyte of flooded/vented battery is liquid whereas
the electrolyte of the VRLA is typically contained in a gel or a glass matt. Flooded batteries are
vented and can evolve hydrogen during use [26] (especially during recharge). These batteries
require a separate room with its own air supply and enough turn-over of air supply to prevent a
build-up of flammable hydrogen gas. By comparison, VRLA batteries evolve very little hydrogen
and can be used in IT equipment spaces without special ventilation.
Significant advances have been made in the design and construction of VRLA batteries. The most
significant advance has been improvements in the purity of the lead used in the plates, the grid,
and the lead paste. Higher purity lead reduces the impact of corrosion which: 1) prolongs battery
life by as much as 20 to 100% compared to non pure-lead VRLA batteries, 2) improves the high
temperature tolerance of pure-lead VRLA batteries, and 3) improves battery shelf life by up to 4X.
In general, there are several important factors that determine the service life of lead acid batteries
used in UPS systems: 1) the number of cycles; 2) the number of deep discharges; 3) amount of
time exposed to a low state of charge; and, 4) the time averaged temperature of the battery over its
life. A discussion of the service life impact of number of cycles, depth of discharge, and periods
of low state charge is available from the battery suppliers and is beyond the scope of this paper.
The impact of temperature alone is fairly well understood. Both types of lead acid batteries require
a fairly narrow air temperature range of 15 to 25°C (59 to 77°F) with 25°C (77°F) considered an
ideal operating temperature. As a general rule of thumb, these batteries experience a lifetime
reduction by half for every 8 to 10°C (14 to 18°F) increase above their ideal rated operating
temperature of 25o
C (77°F) [27] as show below in Figure 18. Just as higher ambient temperatures
can reduce battery lifespans, colder ambient temperatures can reduce battery capacities, so the best](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-27-320.jpg)
![28
practice is to maintain batteries within a fairly narrow range based upon the battery manufacturer’s
recommendations.
Figure 18 Typical lifetime of a lead-acid battery as a function of temperature. Note that 25oC
(77oF) is a typical temperature for 100% of the battery manufacturer’s rated lifetime.
Flooded wet cell batteries located in a separate battery room tend to stay at a fairly uniform ideal
25o
C (77o
F) temperature so temperature control and temperature fluctuations are typically not a
significant factor that would reduce the service life of these batteries. For flooded wet cell batteries
consult IEEE 450-2010 “IEEE Recommended Practice for Maintenance, Testing, and
Replacement of Vented Lead-Acid Batteries for Stationary Applications” [28]. VRLA batteries
and LiON batteries, however, may be located in the IT space and even within IT equipment racks.
In a real-world data center, it is difficult to maintain perfectly uniform temperature control across
the data center, from aisle to aisle, rack to rack, and along the height of a rack. Heat can even be
transmitted to the batteries from IT equipment in adjacent racks. Battery temperature can change
with charging or discharging. All of these factors must be considered when defining VRLA and
LiON battery replacement schedules and service level agreements. Every effort should be made
to maintain lead-acid VRLA batteries in IT areas and inside IT equipment as close to the ideal
25o
C (77o
F) temperature as possible. For information on prolonging the life of lithium ion batteries,
consult references [29], [30].
In practice, the determination of battery life more nuanced than just temperature alone. For
example, a single short (e.g. 100 hours) excursion to warm temperatures (e.g. 40°C (104°F)) won’t
significantly reduce the battery life. However, sustained (7 days a week by 24 hours a day)
operation at warm temperatures will have a significant impact. Actual battery life can be calculated
by taking the histogram of time the battery spent within each range of temperatures (e.g. 20-25,
25-30, 30-35, and 35-40°C) over a year, the battery life curve in Figure 18, and using both sets of
numbers to create a time-averaged battery life.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-28-320.jpg)
![29
The impact of battery temperature on battery lifetime and UPS performance becomes a very
important consideration when a data center is considering economizing over a wide temperature
range, i.e. outside of the ASHRAE recommended range of 18 to 27o
C (64 to 81°F) and the UPS or
DC Power systems will be co-located within the data hall. Prolonged exposure to warmer air
temperatures can decrease battery life and necessitate a more frequent battery replacement
schedule. When considering economization over a wide temperature range, one should make sure
the UPS and associated batteries can provide adequate back up power over the entire range of
expected data center temperatures. Temperatures below 18o
C (64°F) could reduce battery power
output and temperatures warmer than 27o
C (81°F) could reduce battery lifetime. When redundant
UPS systems are both available, the doubling of battery capacity should mitigate the reduced
battery capacity due to colder temperatures. When either redundant system is out-of-service it
may be advisable to curtail economizer operation. Another important consideration for distributed
UPS systems, aside from economization, is to make sure the UPS units can still provide adequate
backup power in the event of an HVAC system failure where air temperatures inside the data center
can rise by as much as 30o
C (54°F) in a matter of minutes. Locating UPS systems and their
respective batteries in separate, grey area spaces with independent HVAC systems alleviates the
concerns regarding economization of data halls.
A complete discussion of battery technologies and their application is beyond the scope of this
paper. However, several good general references on batteries are available [31] [32].
Key Questions for Selection of Battery Based UPS Equipment
The selection of battery backup systems within the temperature variations of a data center facility
environment requires an answer to a number of questions, essentially application criteria, which
must be answered in advance such as:
What type of equipment is to be supported (IT, cooling, distribution, power distribution
and management systems)?
How is this equipment to be supported from backup battery power (momentary, impulse,
or continuous)?
How critical is the load (service level agreement, reliability requirements)?
Is the data center new construction or an existing facility (physical and location limitations)?
What is the load size (usually given in kilowatt hours or megawatt hours)?
What redundancy level is required (Tier 1 through Tier 4)?
Are there any environmental considerations (e.g. California regional air-quality board)?
What is the level of grid reliability?
What is the anticipated operating environment (computer room through ambient outdoor
environment)?
What is the minimum anticipated duration that the battery is required to maintain the load
(plus a safety factor)?
What is the battery environment (temperature, pressure, humidity)?
Is this a new or tested battery technology?
Is this a new installation or retrofit?
What level of business risk is the data center willing to tolerate?
Floor load capacity (batteries are REALLY heavy)](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-29-320.jpg)
![31
Figure 19 Photo of a cabinet PDU of the type used to distribute power to multiple racks of IT
equipment inside a data center.
Figure 20 Location of PDU in the power supply chain.
Most PDUs are designed to be cooled by natural convection. Openings in the raised floor for
power cables should be sealed to prevent the escape of cold air from the underfloor air volume and
a loss of data center cooling efficiency. Many PDUs have some form of built-in intelligent power
monitoring capability. Power monitoring data can be used to calculate the Green Grid’s PUE
(Power Usage Effectiveness) metric [33] [34]. Typical temperature and humidity ratings for floor
mounted PDUs are 0 to 40°C (32 to 104°F) and either 0 to 95%RH or 10 to 95%RH. The typical
PDU temperature and humidity ratings will meet ASHRAE Classes A1, A2, and A3 at maximum
rated power but require de-rating (reduction) of maximum power in order to meet Class A4 which
has a 45°C (113°F) maximum temperature requirement.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-31-320.jpg)
![32
6.2 Transformers
Though not shown on the power distribution diagram of Figure 1, low voltage transformers are an
important element of data center power infrastructure. Transformers are used between the medium
and low voltage switch gear and as part of UPS and PDU systems. Note: the new version of the
National Electrical Code (NFPA 70 [4]), which will be released in 2017, will define medium
voltage as 600 to 1000V. In a data center, a transformer could be feeding either mission critical
equipment or non-critical equipment. The two primary purposes of transformers are: voltage
change and isolation [35]. Transformers are used to step down voltage from the medium supply
voltage to low voltage supply. Transformers are also used to step down to the end utilization
voltage. Within a PDU, for example, transformers are be used to step the voltage down from the
data center distribution voltage of 480/277V to 240/120V or 208/120V (typical for North America).
Isolation transformers are used to transfer power from an AC source to a piece of equipment while
isolating that equipment from the power source, usually for electrical safety reasons. Isolation
transformers provide galvanic isolation which can protect against electric shock, suppresses
electrical noise in sensitive devices, allow for proper operation of over-current protection, and
transfer power between two circuits which must not be connected. A dedicated isolation
transformer is often built with special insulation between primary and secondary and is specified
to withstand a high voltage between windings.
The load at which a transformer will operate at peak efficiency is determined by the transformer
design and varies widely among transformer models. Transformers should be selected so the point
of maximum efficiency lies close to the typical loading. This is important because energy losses
in transformers cause heat generation. High efficiency transformers and transformers that have
been matched for efficiency to typical load conditions will reduce the heat contribution from
transformers and reduce the incremental data center cooling costs from power equipment
containing transformers. An example of a dry type isolation transformer is shown below in Figure
21.
Figure 21 Example of a dry type distribution transformer.
Transformers are not typically rated for a fixed ambient air temperature range or humidity range.
Instead they are rated by the maximum allowable temperature of the transformer coils. The
maximum temperature of the coils due to power is a function of the ambient air temperature and
the temperature rise of the coils from power dissipation. Medium voltage and low voltage](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-32-320.jpg)
![34
In general, minimum and maximum temperature and humidity ranges are not specified for
panelboards. Instead, the maximum ambient temperature is determined by the temperature rating
of conductor wire used and how the current carrying capacity of that wire has been de-rated for
temperature. De-rating means that, at higher temperature, the maximum rated current carrying
capacity of the wire is reduced. Panelboards are cooled by natural convection and typically they
generate very little heat.
Circuit Breakers
Circuit breakers are a very important component used in most types of power equipment including
panelboards. There are a wide range of different types and sizes of circuit breakers. The focus of
this paper is mainly system level power equipment with only limited discussion of key components
inside the equipment. While circuit breakers are a very important component, the level of detail
required to properly discuss circuit breakers is beyond the scope of this paper. There are many
good references on the types of circuit breakers used in data center power equipment. For more
information, please consult the following references [36], [37], [38], and [39].
6.5 Busways
The National Electrical Manufacturers Association (NEMA) defines a busway as a prefabricated
electrical distribution system consisting of bus bars in a protective enclosure along with insulating
and support materials (see Figure 23 below). Busway systems include straight lengths of busway,
fittings, devices, and accessories. A major advantage of busways is how easy the busway sections
are to connect together. Busways are sometimes faster and cheaper to install than a comparable
cable and conduit assembly and they can be installed either overhead or underneath a raised floor.
In some cases, busways are used because they can carry higher current in a smaller footprint than
is possible with cable. The power distribution system of data centers often consists of a
combination of busway and cable and conduit. Figure 1 shows busways as an alternative to RPPs
and panelboards for distributing power to IT equipment racks.
Figure 23 Example of a modular busway system.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-34-320.jpg)
![35
A typical maximum ambient air temperature rating for a busway system is 40°C (104°F) with no
more than 55°C (121°F) hot spot above ambient. A typical range for relative humidity is 0 to
95%RH with an additional stipulation the environment must be non-condensing. Busways are
cooled by natural convection.
6.6 Rack Automatic Transfer Switches (rATS) and rack Static Transfer
Switches (rSTS)
A rack automatic transfer switch (rATS) provides redundant power to single-corded equipment.
The rack ATS has dual input power cords supplying power to the connected load. If the primary
power is unavailable, the rack ATS will seamlessly switch power to the secondary source within
½ of an AC cycle. A piece of single-corded IT equipment can ride through this minimal
interruption via the bulk capacitance inside the power supply. Most rack ATS units have built-in
network connectivity which enables remote management via web, SNMP, or Telnet interfaces.
Another way of explaining the purpose of a rack ATS is to examine a typical IT equipment rack.
The majority of data center IT equipment (e.g., server, storage, and switch) is dual-corded. Dual
corded means that each piece of IT equipment has two power supplies and each power supply is
connected to a different power source. Most racks have two rack PDUs, one on either side of the
rack and each rack PDU draws power from a different leg of the separate and diverse power source.
In the event of a power interruption to one leg of the grid, the IT equipment can run off just one of
its power supplies without interrupting the functionality of the IT equipment. By comparison,
single-corded equipment can draw its power from only one leg of the power grid. If that leg is
affected by an interruption or an outage, then the single-corded equipment would be affected. The
rack ATS takes its power from two legs of the power grid and gives the single-corded equipment
a similar level of power redundancy as the dual-corded equipment. A good discussion of options
for powering single-corded equipment in a dual-corded environment can be found in reference
[36]. A photo of a rack ATS is shown in Figure 24.
Figure 24 Photo of rack automatic transfer switch (rack ATS).
Rack ATS units are typically mounted in the front of the rack so the switch and the power leg
indicator panel is visible from the cold aisle. However, in some cases, rack ATS units may also](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-35-320.jpg)
![37
Table 2 for ASHRAE thermal guideline classes). Rack PDUs are generally located in the rear of
the rack where the air temperature is much higher. Thus, for rack PDUs, thermal compatibility
with the IT equipment means compatibility with the exhaust air temperature of the IT equipment,
not the inlet air temperature specification.
As mentioned in Section 2.3, not only are IT equipment temperatures rising, they level out at a
maximum of between 58 and 60°C (136 and 140°F) for high inlet temperatures of the allowable
ranges of Classes A3 and A4. Given that these high exhaust temperatures could be encountered
in a real data center situation, ASHRAE recommends all new rack PDU products be designed to
at least 60°C (140°F).
Rack power distribution units and other equipment covered by product safety standards such as
IEC 60950-1 [16] are typically rated by the manufacturer based on a maximum allowable air
temperature. This makes sense because it provides a design point for selecting components inside
the rack PDU. Components internal to the rack PDU must stay at or below their rated maximum
temperatures including the air temperature outside the rack PDU and any temperature rise from
heating due to power dissipation of the components themselves or other components within the
rack PDU. Manufacturers of rack PDUs often employ component de-rating. De-rating is the
practice of designing components to be used at temperatures, voltages, and currents less than their
maximum rating as a way of reducing failure rate of the components and improving the reliability
of the rack PDU. In an economized data center, rack PDUs must be carefully selected so their
overall manufacturer design rating meets or exceeds the maximum exhaust temperature of the IT
equipment during the highest inlet air temperature excursion and workload conditions expected.
It is worth noting that a rack PDU with a higher maximum temperature rating has additional
benefits beyond making it capable of economization. In a traditionally cooled data center that uses
a chiller or direct expansion units, an HVAC failure can cause data center temperatures to soar in
a matter of minutes [37]. Higher maximum temperature ratings of rack PDUs and IT equipment
give a data center manager more ride-through time to restart their chiller or air conditioning unit
before equipment reaches its thermal trip limits and starts shutting down. Additional ride-through
time also gives data center operators more time to gracefully shut down IT equipment and avoid a
potential loss of data. Other important rack PDU selection considerations include life expectancy,
projected failure rate, over-current protection (OCP) design limits, natural cooling versus active
cooling designs, mounting configuration, outlet cable ratings, and environmental self-monitoring.
State of the Industry
Intelligent or smart rack PDUs have become the norm for most data centers. The need to
continually monitor power usage on a per-cabinet, or even per-device basis has become a data
center and facilities management best practice. The monitoring capability of rack PDUs provides
the information needed to make data driven decisions. Intelligent rack PDUs contain very
sophisticated electronics and require a level of design rigor and care similar to that of the IT
equipment itself. Data center loads are no longer as stable and static as in the past due to
virtualization, right-sizing of power supplies, and remote-power-off and other IT equipment](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-37-320.jpg)
![39
Figure 27 Server reliability x-factor as a function of inlet air temperature (data from [1]).
In the case of power equipment, if that equipment was made with components, materials, and
manufacturing processes that are similar to the IT equipment then one would expect a similar
failure rate trend with temperature. However, power equipment tends to be much simpler and have
far fewer active components than IT equipment so it is unclear whether power equipment will
follow the same trend as IT equipment. One would still expect a higher failure rate of power
equipment at elevated temperatures but maybe not to the same degree as for the servers shown in
Figure 27.
Table 1 shows the typical usage time for a rack PDU as 8 to 12 years. One of the concerns with
any piece of equipment is whether that equipment will last for the amount of time one intends to
use it or before it is replaced due to obsolescence. The term “lifetime” refers to how long a piece
of equipment will last before we can expect it to wear out and fail. In practice, one would like to
see “lifetimes” much longer than the length of time the equipment is intended to be used. If this
is the case, there should be no wear-out or associated high failure rate during the usage duration of
the product. Most modern component manufacturing and selection processes are carefully
engineered to avoid wear-out failure mechanisms which could cause high failure rates.
A word of caution is in order when using the term “lifetime”. Lifetime is only meaningful when
a cumulative failure criterion is defined along with it. A valid use of the term lifetime would be
to say, for example, a piece of equipment has a lifetime of 15.7 years to reach 1% cumulative
failures. That is, by 15.7 years 1% of the equipment would have failed. Another important term
often used incorrectly as a proxy for lifetime is mean time between failures (MTBF). MTBF is
the average amount of time between failures and is not a lifetime. To better illustrate why MTBF
can be a misleading indicator of lifetime, consider an example where a data center manager selects
a rack PDU with a 10 year MTBF. At first glance, 10 years looks like it would be satisfactory for
a piece of equipment one expects to use for 8 to 12 years. However, if you assume a constant
failure rate in which the failure of a piece of equipment is just as likely in any hour as it is in the
next hour, only 36.8% of the rack PDUs will still be functional at 10 years. In other words, 63.2%](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-39-320.jpg)
![41
prevent leakage of air through the rack. As with the vertical rack PDU, this creates a situation
where the horizontal PDU doesn’t receive any cooling air from the front of the rack and must be
designed to operate in the high temperature environment of the IT equipment exhaust (see Figure
29). For horizontal rack PDUs, there are some server rack fans that can be installed in place of a
blanking panel in the front of the rack that will direct some cooling air to the horizontal rack PDU.
While these fan units do consume a small amount of energy, they can significantly reduce the air
temperature around a horizontal rack PDU and provide a modest improvement in reliability.
Server rack fans are not recommended for vertically installed rack PDUs. A single set of rack fans
will not provide cooling air along the height of the PDU nor will they reduce IT equipment exhaust
temperatures.
Figure 29 Side view of rack showing heat build-up around installation of a horizontal rack PDU.
Along with the considerations for localized temperatures around the rack PDU, the power cabling
from the outputs of that rack PDU to the servers and other devices needs to be investigated. Be
sure that the C13 and C19 couplers and associated cables are properly sized to allow for full current
carrying capability over the entire temperature range. C13 and C19 refer to standard types of
couplers defined in IEC 60320 [14]. Also consider that tightly bound bundles of cabling will cause
localized heating within that bundle, which could exceed the ratings of the cables.
Environmental Monitoring
Many intelligent rack PDUs with network connectivity have the capability to provide temperature
and humidity monitoring inside the rack. These rack PDUs typically have multiple probes that
can be positioned independently. With network connectivity, this data can be reported in real-time
and used for:
Mapping the distribution of air temperature across the data center air to better optimize
cooling
Making sure air temperatures are within the design specifications of the equipment
Identifying localized hot spots that may need additional cooling
Providing alert capability for temperature excursions that could cause a thermal shutdown
of IT equipment and compromise data center uptime](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-41-320.jpg)
![43
at lower
temperatures.
ASHRAE Class A2 10 to 35°C -12˚C DP & 8% RH
to27˚C DP and 85%
RH
Cold aisle and
ballroom
ASHRAE Class A3 5 to 40°C -12˚C DP & 8% RH
to27˚C DP and 85%
RH
Cold aisle and
ballroom
ASHRAE Class A4 5 to 45°C -12˚C DP & 8% RH
to27˚C DP and 90%
RH
Cold aisle and
ballroom
*Note: the maximum ambient air temperature rating of most power equipment is flexible. Higher
ambient air temperature ratings are possible if the maximum power rating is reduced.
In the context of this discussion, thermal compatibility is defined as the power equipment
specifications meeting or exceeding those of a given ASHRAE class plus appropriate preheating
based on location. A summary of how power distribution specifications align to the ASHRAE
thermal guidelines for IT equipment is given below in Table 7.
Table 7 Cold aisle and hot aisle compatibility of power distribution equipment.
Equipment Type or
Standard
Cold Aisle Thermal
Compatibility at Maximum
Power Rating
Hot Aisle Thermal
Compatibility at Maximum
Power Rating
Floor Mounted Power
Distribution Unit (PDU)
Compatible with ASHRAE
Class A3
Compatible only with power
de-rating
Remote Power Panel
(RPP)
Compatible with ASHRAE
Class A3
Compatible only with power
de-rating
Panelboards N/A
Busways Compatible with ASHRAE
Class A3
Compatible only with power
de-rating
Rack ATS Compatible with ASHRAE
Class A3/A4
Depends on Model
Rack PDU
Compatible with ASHRAE
Class A4
Depends on Model
IEC 61439 Standard [38] Compatible with ASHRAE
Class A2
Not Compatible
Nearly all power equipment is cold aisle compatible up to ASHRAE Class A3 at maximum rated
power. Most power equipment is hot aisle compatible but only with de-rating (reduction) of
maximum power to limit conductor temperature rise and stay within maximum conductor
temperature limits. Rack PDU and ATS units are all cold aisle compatible and many models, rated
at >60°C (140°F), are also hot aisle compatible. Unlike other types of power equipment, rack
PDUs, rack ATS units, and some line or power cords have a fixed maximum ambient air
temperature rating and power de-rating flexibility is not provided in their specifications. The
maximum ambient air temperature of rack PDUs is primarily limited by touch temperature. The
receptacles have a 70°C (158°F) touch temperature limit per IEC60320-1 [14] and most metal](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-43-320.jpg)
![44
surfaces also have a 70°C (158°F) touch temperature limit per IEC60950-1 [16]. If 10°C (18°F)
is allowed for conductor temperature rise, the maximum ambient air temperature rating needs to
be 60°C (140°F) which is the most common rating for rack PDUs. Higher maximum air
temperature ratings for rack PDUs may be needed in the future if the trend of rising IT equipment
exhaust temperatures continues. However, equipment designers will need to find a way of
overcoming touch temperature limitations.
The IEC 61439 family of standards “Low-Voltage Switchgear and Control Gear Assemblies” [38]
listed in Table 7, is a commonly accepted international standard for power and power distribution
equipment. IEC 61439 covers low voltage switch gear, power switchgear, control gear assemblies,
distribution boards, and busways. IEC 61439 aligns to ASHRAE Class A2 (35°C, 95°F)).
However, the most common design point for power equipment is 40°C (104°F) which corresponds
to ASHRAE Class A3. ASHRAE TC9.9 recommends designing to IEC 61439, at a minimum, and
to the temperature requirements of Class A3, where possible. IEC 61439 and Class A3 differ in
their humidity requirements: IEC 61439 calls out a requirement of 50%RH at 40°C (104°F) which
is a 27.6°C (81.7°F) maximum dew point versus only a 24°C (75°F) maximum dew point for
ASHRAE class A3 and A4. However, little or no design work or component selection is typically
done on the basis of meeting a humidity requirement. Any product that meets the IEC humidity
requirements should also meet or exceed the ASHRAE A3 and A4 humidity requirements.
A summary of the ASHRAE TC9.9 recommendations for rack power distribution equipment is as
follows:
Rack PDUs should have a maximum ambient air temperature rating of at least 60°C
(140°F).
Openings in the raised floor for power cables should be sealed to prevent the escape of cold
air from the underfloor air volume and a loss of data center cooling efficiency.
The maximum power rating of power distribution equipment should be sized to account
for variations and excursions in air temperature. High air temperature excursions can occur
for a number of reasons such as: hot weather where an air-side economized data center
takes its cooling air from outdoors, an HVAC failure that causes temperatures to rise
quickly in the data center, or a localized data center hot spot when cooling is less than ideal.
Locate power equipment in ballroom, cold aisles, or grey areas where possible. Higher hot
aisle temperatures necessitate de-rating of maximum power whereas cold aisle
temperatures typically don’t require de-rating. Hot aisle temperatures have risen over time
and may continue to rise.
Ideally, the power distribution infrastructure should not be a weak link that causes data center
equipment to shut down. The power equipment should allow the IT equipment to run as long as
possible to avoid a service outage or data loss and give the equipment time to save data and perform
a graceful shutdown, if needed.
7. Higher Voltage DC (HVDC) Power
Both AC and HVDC voltage can be dangerous depending on the amperage flowing in a circuit.
In general, there are two electrical hazards with either AC or HVDC: shock and arc flash.
However, since AC voltage is sinusoidal, continuously changing direction and HVDC voltage is](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-44-320.jpg)
![47
277 VAC
The advantage of using 277 VAC rather than 200-240 VAC in data centers is the Capex saving
from eliminating the final step down transformer (see Figure 30). Not only is there a savings in
hardware, but there is also a savings in copper wiring and maintenance. There is also an Opex
savings in the form of less heat released. A transformer with a nameplate capacity of 100 kW that
is 96% efficient at rated capacity has a loss of 4 kW that would require an equivalent amount of
cooling. New 2016 DOE efficiency standards for transformers [39] are shown below in Table 8
compared to the former 2002 NEMA TP-1 standard [40]. Improvements in transformer efficiency,
especially in three-phase transformers, should necessitate less cooling. Note: the efficiency
requirements for single phase transformers have not changed except that that new 2016 DOE
standard requires they be tested and measured to two decimal places instead of one.
Table 8 New 2016 DOE standard transformer efficiencies compared to former 2002 NEMA TP-
1 efficiency levels.
Supplemental
Two of the emerging electrical power schemes used in data centers distribute voltages higher than
the traditional 200-240 VAC. An operating voltage higher than 250V requires design changes to
IT power supplies, line cords, and rack PDUs. The majority of 19” rack-mount IT equipment in
the marketplace uses the IEC 60320 standard appliance inlets on power supplies. Above 250V,
these couplers can no longer be used because of voltage spacing requirements and electric shock
protection. Moving to a different set of couplers on the power supplies affects the power jumper
cords between rack-mount drawer and PDU as well as the PDU receptacles.
The thermal compatibility of DC power equipment isn’t much different than AC power equipment
as their placement and usage in the data center is the same. Reducing voltage conversions will
reduce the heat rejected into the environment, but ever increasing AC conversion efficiencies of
UPSs and IT equipment power supplies at the 200-240 VAC level are also reducing heat output.
2002 NEMA TP-1 2016 DOE Standard 2002 NEMA TP-1 2016 DOE Standard
15 97.7% 97.70% 15 97.0% 97.89%
25 98.0% 98.00% 30 97.5% 98.23%
37.5 98.2% 98.20% 45 97.7% 98.40%
50 98.3% 98.30% 75 98.0% 98.60%
75 98.5% 98.50% 112.5 98.2% 98.74%
100 98.6% 98.60% 150 98.3% 98.83%
167 98.7% 98.70% 225 98.5% 98.94%
250 98.8% 98.80% 300 98.6% 99.02%
333 98.9% 98.90% 500 98.7% 99.14%
750 98.8% 99.23%
1000 98.9% 99.28%
Single-Phase Three-Phase
Efficiency %Efficiency %
KVA KVA](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-47-320.jpg)
![50
different material (e.g. plastic instead of metal) may need to be used in the equipment
construction.
11. Power equipment that is part of a row of racks containing IT equipment (such as floor
mounted PDUs) should be compatible with a hot aisle cold aisle air flow configuration.
If the power equipment requires a cooling air flow, it should take air flow from the cold
aisle and exhaust the air to the hot aisle in a manner similar to the IT equipment [41].
Having power equipment in the same row as IT equipment is more common in colocation
data centers. Front to back air flow configuration is important for implementing energy
efficiency strategies such as aisle containment.
12. Ensure power equipment is installed in such a way as to prevent condensation. Precise
humidity control is not critical for most power equipment, however, prevention of
condensation is. Simple heaters, such as panel heaters, should be installed where needed
to keep conductors slightly warmer than the surrounding air and prevent condensation from
forming.
13. Most VRLA batteries may not be suitable for IT equipment spaces where the data
center is economizing over a temperature range wider than 18 to 27°C (64°F to 81°F).
The lifetime of most lead-acid based VRLA batteries is very temperature dependent and
operating these batteries at temperatures much above the ideal temperature of 25°C (77°F)
may not practical or cost effective.
14. Seal openings for power cables in the raised floor to prevent escape of cold air from
the plenum and a loss of data center cooling efficiency. One solution is to use
commercially available grommet and brush kits. A supply of cooling air from the plenum
under the raised floor is usually not necessary for power equipment as most power
equipment is natural convection cooled. This recommendation applies primarily to power
equipment located in ballroom areas.
15. Monitor data center air temperature at multiple locations including locations with
power equipment and line cords. Air temperatures can vary widely across a data center
even within the length of a meter. Real-time monitoring and reporting of air temperature
will prevent accidental equipment overheating and shutdown and it is a tool the data center
operator can use to optimize cooling.](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-50-320.jpg)
![51
9. Summary
Most power equipment is rated by the temperature of the electrical conductors and not the
surrounding air. Rack PDUs are a notable exception, though the maximum air temp is calculated
based on conductor temperature rise. This flexible rating system allows power equipment to be
used over a wide range of temperature, provided the ampacity is de-rated per the manufacturer’s
guidelines. Compatibility of thermal specifications is needed, where power equipment shares the
same space or air streams with IT equipment. Compatibility does not necessarily mean thermal
specifications (power vs. IT equipment) are defined the same way. It does mean the power
equipment meets or exceeds the thermal specifications of the IT equipment including any
conductor temperature rise. ASHRAE recommends power equipment be designed for the
temperature and humidity requirements of IEC61439 at a minimum and preferably to the 40°C
(104°F) upper air temperature limit of ASHRAE class A3. This recommendation is important
because data center environmental conditions are slowly changing to save money on cooling costs.
Many data centers now operate at slightly warmer air temperatures and some are even economized
using outside air. A set of recommendations has been proposed for the major types of power
equipment to improve their thermal compatibility with IT equipment and their reliability.
10. References
[1] ASHRAE TC9.9, Thermal Guidelines for Data Processing Environments, Fourth
Edition, ASHRAE, Atlanta Georgia, 2015.
[2] P. Hu, "Electrical Distribution Equipment in Data Center Environments, White Paper
61, Revision 1," 2015. [Online]. Available:
http://www.apcmedia.com/salestools/VAVR-8W4MEX/VAVR-
8W4MEX_R1_EN.pdf?sdirect=true. [Accessed 9 July 2015].
[3] D. Loucks, "Power Equipment and Data Center Design, The Green Grid White Paper
#51," 2012. [Online]. Available:
http://www.thegreengrid.org/~/media/WhitePapers/WP51-
PowerEquipmentandDataCenterDesign_v1.pdf?lang=en. [Accessed 9 July 2015].
[4] National Fire Protection Association, NFPA 70 National Electrical Code, Quincy, MA:
National Fire Protection Association, 2014.
[5] Emerson Network Power, " Five Strategies for Cutting Data CenterEnergy Costs
Through Enhanced Cooling Efficiency, WP151-47," 2007. [Online]. Available:
http://www.emersonnetworkpower.com/documentation/en-
us/brands/liebert/documents/white%20papers/data-center-energy-efficiency_151-
47.pdf. [Accessed 10 June 2015].
[6] P. Thibodeau, "It's getting warmer in some data centers," 15 July 2013. [Online].
Available: http://www.computerworld.com/article/2483971/data-center/it-s-getting-
warmer-in-some-data-centers.html. [Accessed 9 July 2015].
[7] J. Kaiser, "Survey Results: Data Center Economizer Use, White Paper #41," 2011.
[Online]. Available:
http://www.thegreengrid.org/sitecore/content/Global/Content/white-papers/WP41-
SurveyResultsDataCenterEconomizerUse.aspx. [Accessed 10 June 2015].](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-51-320.jpg)
![52
[8] Air Enterprises, "Thermowheel - the Energy Recovery Leader," [Online]. Available:
http://airenterprises.com/products/energy-recovery-thermowheel/. [Accessed 10 June
2015].
[9] ASHRAE, Particulate and Gaseous Contamination in Datacom Environments, Atlanta,
Georgia: American Society of Heating, Refrigerating, and Air-Conditioning Engineers,
2014.
[10] R. Miller, "APC EcoBreeze Offers Free Cooling Options," 14 December 2010.
[Online]. Available: http://www.datacenterknowledge.com/archives/2010/12/14/apc-
ecobreeze-offers-free-cooling-options/. [Accessed 20 August 2015].
[11] LIebert, "Liebert® DSE™ with EconoPhase™: Highest Efficiency DX Cooling with
pumped refrigerant economizer, Product Brief," 2015. [Online]. Available:
http://www.emersonnetworkpower.com/documentation/en-
us/brands/liebert/documents/white%20papers/liebert_dse_product_brief-sl-18930.pdf.
[Accessed 20 August 2015].
[12] International Electrotechnical Commission (IEC), IEC 60309-1 Plugs, socket-outlets
and couplers for industrial purposes – Part 1 General Requirements, Geneva
Switzerland: IEC, 2005.
[13] International Electrotechnical Commission (IEC), IEC 60309-2 lugs, socket-outlets and
couplers for industrial purposes - Part 2: Dimensional interchangeability requirements
for pin and contact-tube accessories, Geneva Switzerland: IEC, 2005.
[14] International Electrotechnical Commission (IEC), IEC 60320-1:2015 Appliance
couplers for household and similar general purposes - Part 1: General requirements,
International Electrotechnical Commission (IEC), 2015.
[15] American Society of Heating Refrigerating and Air-Conditioning Engineers
(ASHRAE), IT Equipment Design Impact on Data Center Solutions, Atlanta Georgia:
ASHRAE, 2016.
[16] International Electrotechnical Commission (IEC), IEC 60950-1 Information technology
equipment – Safety, Part 1: General Requirements, 2nd Edition, Geneva, Switzerland:
IEC, 2005.
[17] IEC 61439 Low-voltage switchgear and controlgear assemblies, International
Electrotechnical Commission (IEC), 2011.
[18] ISO 12944: Paints and varnishes -- Corrosion protection of steel structures by
protective paint systems, International Organization for Standardization, 2007.
[19] N. Rasmussen, "The Different Types of UPS Systems, APC Schneider Electric White
Paper #1, Revision 7," 2011. [Online]. Available:
http://www.apcmedia.com/salestools/SADE-5TNM3Y/SADE-
5TNM3Y_R7_EN.pdf?sdirect=true. [Accessed 9 July 2015].
[20] K. McCarthy, "Comparing UPS System Design Configurations, APC Schneider
Electric White Paper #75, Revision 3," [Online]. Available:
http://www.apcmedia.com/salestools/SADE-5TPL8X/SADE-
5TPL8X_R3_EN.pdf?sdirect=true. [Accessed 9 July 2015].
[21] S. McCluer, "Comparing Data Center Batteries, Flywheels and Ultracapacitors, APC
White Paper #65," 2009. [Online]. Available: http://www.apcdistributors.com/white-](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-52-320.jpg)
![53
papers/Power/WP-
65%20Comparing%20Data%20Center%20Batteries,%20Flywheels,%20and%20Ultrac
apacitors.pdf. [Accessed 10 June 2015].
[22] Underwriter's Laboratories, Standard for Lithium Batteries UL 1642, Underwriter's
Laboratories, September, 2012.
[23] M. Azarian, "Overview of Electrical Double Layer Capacitors," University of
Maryland, Center for Advanced Life Cycle Engineering, College Park, Maryland, 2013.
[24] Nichicon UM Series EDLC Capacitor Specification.
[25] Nichicon, Application Guidelines For Electric Double Layer Capacitors "EVerCAP"
CAT.8100A.
[26] S. Mcluer, "Battery Technology for Data Centers and Network Rooms: Ventilation,
APC Schneider White Paper #30," 2003. [Online]. Available:
http://www.apcdistributors.com/white-papers/Power/WP-
34%20Battery%20Technology%20for%20Data%20Centers%20and%20Network%20r
ooms%20Ventilation.pdf. [Accessed 10 June 2015].
[27] Eaton, "The Large UPS Battery Handbook," 2012. [Online]. Available:
http://www.capitolpower.com/pdf/Batteryhandbk.pdf. [Accessed July 2015].
[28] IEEE, IEEE 450-2010, IEEE Recommended Practice for Maintenance, Testing, and
Replacement of Vented Lead-Acid Batteries for Stationary Applications, New York:
IEEE, 2011.
[29] Battery University, "How to Prolong Lithium Based Batteries," 20 January 2016.
[Online]. Available:
http://batteryuniversity.com/learn/article/how_to_prolong_lithium_based_batteries.
[Accessed 9 February 2016].
[30] F. Leng, "Effect of Temperature on the Aging rate of Li Ion Battery Operating above
Room Temperature," Scientific Reports, vol. 5, no. Article number: 12967, 2015.
[31] T. Reddy, Linden's Handbook of Batteries, New York: McGraw-Hill, 2011.
[32] ASHRAE TC9.9, "Data Center Storage Equipment - Thermal Guidelines, Issues and
Best Practices," 2014. [Online]. Available:
http://tc99.ashraetcs.org/documents/ASHRAE_Storage_White_Paper_2015.pdf.
[Accessed 6 July 2015].
[33] V. Avelar, "The Green Grid White Paper #49 PUE™: A Comprehensive Examination
of the Metric," The Green Grid, 2012.
[34] K.-D. Lange, Server Efficiency--Metrics For Computer Serverse And Storage, Atlanta
Georgia: ASHRAE, 2015.
[35] N. Rasmussen, "The Role of Isolation Transformers in Data Center UPS Systems, APC
Schneider Electric White Paper #98, Revision 0," 2011. [Online]. Available:
http://www.apcmedia.com/salestools/NRAN-7NB2FG/NRAN-
7NB2FG_R0_EN.pdf?sdirect=true. [Accessed 10 July 2015].
[36] R. Tajali, "LOW VOLTAGE CIRCUIT BREAKER GUIDELINES FOR DATA
CENTERS, Rev 2," 13 May 2002. [Online]. Available:
http://www.powerlogic.com/literature/DataCenterCBApplications_05132002.pdf.
[Accessed 3 May 2016].](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-53-320.jpg)
![54
[37] Eaton, "DATA CENTER – UNIQUE CHARACTERISTICS AND THE ROLE OF
MEDIUM VOLTAGE VACUUM CIRCUIT BREAKERS," [Online]. Available:
https://powerquality.eaton.com/About-Us/News-Events/whitepapers/Data-center-
characteristics.asp?id=&key=&Quest_user_id=&leadg_Q_QRequired=&site=&menu=
&cx=3&x=18&y=7. [Accessed 3 May 2016].
[38] Raritan, "Data Center Power Overload Protection:," 2009. [Online]. Available:
http://www.gocsc.com/UserFiles/File/Raritan/TA2011/DCpoweroverloadprotection.pdf
. [Accessed 3 May 2016].
[39] J. Anderson and M. Jansma, "Circuit Breakers in Data Centers: The Hidden Danger,"
[Online]. Available:
http://www.on365.co.uk/files//The%20Ugly%20Truth%20About%20Circuit%20Break
ers%20In%20Data%20Centers.pdf. [Accessed 3 May 2016].
[40] V. Avelar, "Powering Single Corded Equipment in a Dual Corded Environment, APC
Schneider White Paper #62," [Online]. Available:
http://www.apcmedia.com/salestools/SADE-5TNRLE/SADE-
5TNRLE_R1_EN.pdf?sdirect=true. [Accessed 14 July 2015].
[41] P. Lin, "Data Center Temperature Rise During a Cooling System Outage, APC White
Paper 179," 2014. [Online]. Available: http://www.apcmedia.com/salestools/DBOY-
7CDJNW/DBOY-7CDJNW_R1_EN.pdf. [Accessed 27 August 2015].
[42] International Electrotechnical Commission, IEC 61439 Low-Voltage Switchgear and
Control Gear Assemblies, Geneva Switzerland: IEC, 2011.
[43] United States Department of Energy (DOE), Code of Federal Regulations at 10 CFR
431.196, Washington DC: United States Department of Energy (DOE), 2016.
[44] National Electrical Manufacturers Association (NEMA), NEMA TP-1 Guide for
Determining Energy Efficiency for Distribution Transformers, Rosslyn Virginia:
NEMA, 2002.
[45] P. Lin, "Implementing Hot and Cold Air Containment in Data Centers, APC White
Paper #153," 2013. [Online]. Available: http://www.apcmedia.com/salestools/VAVR-
8K6P9G/VAVR-8K6P9G_R0_EN.pdf?sdirect=true. [Accessed 10 June 2015].](https://image.slidesharecdn.com/ashraetc0909powerwhitepaper22june2016revised-160830024455/85/ASHRAE-TC9-9-DATA-CENTER-STANDARD-AND-BEST-PRACTICES-54-320.jpg)
ASHRAE TC9.9 - DATA CENTER STANDARD AND BEST PRACTICES
- 1. 1 ASHRAE TC9.9 Data Center Power Equipment Thermal Guidelines and Best Practices Whitepaper created by ASHRAE Technical Committee (TC) 9.9 Mission Critical Facilities, Data Centers, Technology Spaces, and Electronic Equipment © ASHRAE 2016
- 2. 2 Data Center Power Equipment Thermal Guidelines and Best Practices Table of Contents 1. INTRODUCTION.................................................................................................................4 1.1 TYPICAL DATA CENTER POWER DISTRIBUTION SYSTEM ................................4 1.2 CATEGORIZING DATA CENTER POWER EQUIPMENT BY LOCATION.............5 2. CHANGES IN DATA CENTER ENVIRONMENTS .....................................................10 2.1 ASHRAE THERMAL GUIDELINE CLASSES FOR IT EQUIPMENT SPACES ......10 2.2 INCREASING USE OF ECONOMIZATION IN IT EQUIPMENT SPACES .............12 2.3 RISING EXHAUST TEMPERATURE OF IT EQUIPMENT ......................................13 2.4 AIR TEMPERATURE TRENDS IN IT SUPPORT EQUIPMENT SPACES...............15 2.5 THERMAL TRANSIENTS AND EXCURSIONS IN IT EQUIPMENT SPACES ......16 3. TEMPERATURE RATINGS FOR POWER EQUIPMENT.........................................17 4. MEDIUM AND LOW VOLTAGE SWITCHGEAR.......................................................19 5. UNINTERRUPTIBLE POWER SUPPLIES ...................................................................21 6. ELECTRICAL POWER DISTRIBUTION .....................................................................30 6.1 ROOM POWER DISTRIBUTION UNIT (PDU) ..........................................................30 6.2 TRANSFORMERS.........................................................................................................32 6.3 REMOTE POWER PANEL (RPP) ................................................................................33 6.4 PANELBOARDS ...........................................................................................................33 6.5 BUSWAYS.....................................................................................................................34 6.6 RACK AUTOMATIC TRANSFER SWITCHES (RATS) AND RACK STATIC TRANSFER SWITCHES (RSTS)..................................................................................35 6.7 RACK POWER DISTRIBUTION UNITS.....................................................................36 6.8 RECOMMENDATIONS FOR POWER DISTRIBUTION EQUIPMENT...................42 7. HIGHER VOLTAGE DC (HVDC) POWER...................................................................44 8. ASHRAE TC9.9 RECOMMENDATIONS ......................................................................48 9. SUMMARY .........................................................................................................................51 10. REFERENCES....................................................................................................................51 APPENDIX A – DEFINITIONS.................................................................................................55
- 3. 3 Acknowledgements The ASHRAE TC9.9 committee would like to thank the following persons for their groundbreaking work and willingness to share their subject matter knowledge in order to further the understanding of the entire data center industry: Chuck Rabe – Hewlett Packard Enterprise Darrel Gaston – Hewlett Packard Enterprise Mark Lewis – Hewlett Packard Enterprise David Mohr – Hewlett Packard Enterprise Dave Rotheroe – Hewlett Packard Enterprise Dave Kelley – Emerson Network Power Eric Wilcox – Emerson Network Power Kyle Wessels – Emerson Network Power Jon Fitch – Dell Al Dutra - Dell John W. Collins – Eaton Marc H. Hollingsworth – Eaton Sturges Wheeler – Eaton Phillip J. Fischer – Eaton John Bean – Schneider Electric Victor Avelar - Schneider Electric Jay Taylor – Schneider Electric Marc Cram – Server Technology Robert Faulkner – Server Technology Joe Prisco – IBM Roger Schmidt – IBM William Brodsky – IBM Paul Estilow – DLB Associates These persons invested a significant amount of their time in conference calls, writing drafts, drawing figures, and editing and reviewing text. Thanks also to Jon Fitch (Dell) for leading the white paper team and making final edits to the paper. Special thanks to Roger Schmidt for his support on the white paper and for his leadership of the ASHRAE TC9.9 IT sub-committee. Special thanks also to Dave Kelley (Emerson), Paul Artman (Lenovo), John Groenewold (Chase), William Brodsky (IBM), Roger Schmidt (IBM), Terry Rodgers (Primary Integration Solutions), Tom Davidson (DLB Associates), Jason Matteson (Lenovo), Joe Prisco (IBM), and Dustin Demetriou (IBM) for taking the time to do an in-depth review of the draft and for providing detailed and insightful feedback. Thanks also to Ian Bitterland, Bob Landstrom, and Harry Handlin from The Green Grid for providing helpful review comments.
- 4. 4 1. Introduction Changing data center environmental conditions are of importance to IT equipment but also to power equipment, especially where the two types of equipment share the same physical space and air stream. ASHRAE’s document [1], “Thermal Guidelines for Data Processing Environments– Fourth Edition” has increased the industry’s awareness of the effect increased operating temperature can have on IT equipment. In some cases, power equipment can be subjected to higher temperatures than the IT equipment. Higher temperatures can impact equipment reliability. Exposure to warmer temperatures, coupled with the fact that usable life cycle of power equipment is typically longer than IT equipment, increases the importance of this topic. This paper discusses how changes to the data center thermal environment may affect power distribution equipment. This paper also provides an overview of data center power distribution [2] [3] and describes the typical power equipment used for both IT loads and non-IT loads (i.e. lighting and cooling). Included in this list of equipment is switchgear, uninterruptible power supplies (UPS), static transfer switches, switchboards, transformers, power distribution units (PDU), remote power panels (RPP), panelboards, rack PDU, line cords, facility receptacles and IT cable trays. Note that the order in which the power distribution equipment is discussed is not necessarily the same order in which it would appear on an electrical diagram. The paper concludes with a set of recommendations on how to improve power equipment thermal compatibility and reliability. 1.1 Typical Data Center Power Distribution System An electrical one-line diagram is typically used to communicate specific details of an electrical distribution design and shows the logical flow of electricity from utility mains to the IT equipment. A block diagram, shown in Figure 1, provides a higher-level view of a data center’s electrical flow without the complexity of a one-line diagram. Figure 1 serves as a guide to show where certain types of equipment are typically found within a data center, both logically and physically. Dashed lines are used to show in which type of space each piece of power equipment resides. Heavy dashed lines indicate electrical space, lines with a dash and a single dot delineate mechanical space, and the line with a dash and two dots defines the IT space. Generators, though part of the data center power infrastructure, are beyond the scope of this paper. In general, data center electrical architectures start with a utility supply at medium-voltage (600 to 1000V) which feeds medium-voltage switchgear. Note: the new version of the National Electrical Code (NFPA 70 [4]), which will be released in 2017, will define medium voltage as 600 to 1000V. This medium-voltage is “stepped down” to low-voltage (typically 480V and lower) using a transformer which then feeds low-voltage switchgear. Low-voltage switchgear generally feeds UPS units which then feed UPS output switchgear or panelboards and then feed room PDUs. Room PDUs typically provide power to remote power panels (RPPs) which supply branch circuits to each IT rack which then feed rack PDUs (i.e. power strips). Working back up the power supply chain to the utility feed, there is a similar chain of power distribution equipment used for non-IT loads such as lighting and cooling equipment. As you go higher in the electrical architecture, the more likely it is that the equipment (at the medium voltage level) feeds both IT and non-IT equipment.
- 5. 5 Figure 1 Overview of a typical data center power distribution system [2]. Used with permission. 1.2 Categorizing Data Center Power Equipment by Location Data center power distribution equipment can be categorized a number of different ways but for the purposes of this paper, it is helpful to categorize power equipment by location and by power path. The location of power equipment (e.g. mechanical space, IT space, cold aisle, hot aisle) will determine its usage environment. It is also helpful to know whether the equipment is in series with critical IT equipment (i.e. a UPS) or in parallel (i.e. lighting power panel) since this may influence the selection of the power equipment. Power equipment generally has a longer usage lifecycle than IT equipment.
- 6. 6 Figure 2 General classification of data center spaces: ballroom, hot aisles, cold aisles, and grey areas. Figure 2 above shows an example of a typical data center facility space plan. Most data centers have four types of environmental areas: ballroom spaces, hot aisles, cold aisles, and grey areas. Many data center designs have computer rooms where cold air is distributed through a raised floor system that uses the under floor space as a supply air plenum formed by the raised floor. Cold aisles are formed by the space between the front faces of two rows of IT equipment racks. The front face of the rack is where IT equipment takes in its cooling air. Cold air is distributed through an under floor air volume formed by the raised floor. Perforated floor tiles in front of the racks direct the cooling air upwards to the face of each rack. Many new data centers as well as legacy Telco facilities use an alternative design that deploys IT racks directly on the floor slab. Regardless, both design strategies orient the IT racks in what is called a hot aisle/cold aisle layout. Cold aisles are formed by the space between the front faces of two rows of IT equipment racks. The front face of the rack is where IT equipment takes in its cooling air. As the name would imply, the coolest air temperatures in the data center are found in the cold aisle. Hot aisles are formed by the space between two rows of back to back racks where the hot exhaust air from both racks is directed into a common aisle. The highest air temperatures in the data center are typically found in the hot aisle. Ballroom spaces are a multi-purpose area of the data center IT equipment space. Ballroom spaces are the areas that aren’t contained within a hot aisle or a cold aisle. The temperature of the ballroom spaces can be similar to the cold aisles, the hot aisles or something in between depending on the
- 7. 7 aisle containment strategy being used. A lower volume of cooling air is usually provided to ballroom areas compared to cold aisles because the equipment located in ballroom spaces usually generates much less heat than IT equipment. Grey areas are dedicated to IT support equipment. The air temperature is similar to or slightly warmer than the cold aisles or ballroom areas. A much smaller volume of cooling air is provided to these areas, compared to a cold aisle or ballroom, because the IT support equipment generates much less heat than IT equipment. Grey areas are dedicated to IT support equipment. The space temperature in most grey areas is controlled to a broader range of environmental conditions than computer rooms allowing for both higher and lower temperature and humidity extremes than the cold aisles or ballroom areas. A much smaller volume of cooling air is provided to these areas, compared to a cold aisle or ballroom, because the IT support equipment generates much less heat than IT equipment and has a far greater tolerance for ambient environmental conditions. Figure 3 and Figure 4 are Venn diagrams that illustrate the location of power equipment within a data center. Figure 3 shows where the major types of power equipment are typically located: inside the data hall (i.e. IT equipment space) and outside of the data hall. The equipment shown in the overlapped area can be located either inside or outside the data hall. Figure 4 applies to power equipment located within the data hall, and specifically breaks down equipment location in the hot aisle and in the cold aisle. The equipment shown in the overlapped area of Figure 4 is sometimes located in the hot aisle, cold aisle, or ballroom areas. Power equipment including line cords, receptacles, terminations, etc. placed in the hot aisle must be specified to operate in a warmer environment. Note that equipment located in certain areas of the data center such as end of rows and back of racks or cabinets may also require a higher operating temperature specification. Figure 3 Venn diagram illustrating the physical location of data center power equipment.
- 8. 8 Figure 4 Venn diagram of power equipment located within the data hall (IT equipment space). Table 1 summarizes the information in Figure 3 and Figure 4 but also adds the estimated range of usage lifetimes for various equipment types.
- 9. 9 Table 1 Comparison of physical location and typical usage lifetimes for power and IT equipment. Type of Equipment Shares Space & Air with IT Equip? Location Typical Usage Lifetime (years)* Generators NO Outdoor enclosure or separate room 20+ years Electrical Switch Gear Sometimes Ballroom or separate grey area 20+ years Transformers Sometimes Ballroom or separate grey area 20+ years UPS – Rack mount YES Ballroom or Cold aisle 4 to 8 years UPS - floor standing and row-based 3-phase UPS from ~20kW up to 1.2MW YES Cold aisle, ballroom or separate grey area 10 to 20 years UPS – Central Enterprise - 3,000+ KVA Sometimes Separate grey area or ballroom 15 to 20 years Flooded Vented Wet Cell Battery Strings NO Typically grey area - separate battery room 15 to 20 years Sealed batteries Sometimes Ballroom, cold aisle, or grey area 3 to 10 years Flywheel UPS Sometimes Ballroom, cold aisle, hot aisle, or grey area 15 to 20 years Panelboard Sometimes Ballroom or separate grey area 20+ years Static Transfer Switch Sometimes Ballroom, cold aisle, hot aisle, or grey area 20 to 25 years Computer Room PDU Sometimes Ballroom, cold aisle, hot aisle, or grey area 8 to 20 years Remote Power Panel (RPP) YES Ballroom, cold aisle, hot aisle, or grey area 8 to 20 years Rack PDU YES Hot aisle 8 to 12 years Rack ATS YES Hot or cold aisle 8 to 12 years Power Cable (individual cables) YES Hot, cold aisle, ballroom or under floor 8 to 20 years** Power Bus Way (copper bus bar, plug modules into to tap power) YES Hot, cold aisle, ballroom or under floor 8 to 20 years**
- 10. 10 Structured Cabling (low voltage IT cabling, fiber and copper) Sometimes Hot, cold aisle, ballroom or under floor Varies widely Servers YES Cold aisle 3 to 8 years Storage Arrays YES Cold aisle 5 to 8 years Networking YES Cold aisle 5 to 10 years *These lifetimes are based on equipment that is properly maintained and used in an appropriate application and environment. **These items are usually replaced because of changes in data center power capacity that require a change in power cable and bus bar sizing, but not because these items wear out. As shown in the right-hand column of Table 1, most power equipment stays in service longer than the IT equipment it supports. Power equipment can have a wider variety of physical locations than IT equipment. The environments for power equipment are also less homogeneous than for IT equipment and they include hot and cold aisle, separate grey spaces, battery rooms, as well as the outdoors. Most power equipment is natural convection cooled with the exception of UPS modules and some transformers, whereas IT equipment is typically cooled by forced air from a set of variable speed internal fans. The intent of the ASHRAE thermal guidelines [1] was to provide guidelines for IT equipment. Power equipment was not a primary consideration in the writing of those thermal guidelines. As will be discussed later, the way power equipment specifications are written is quite a bit different than specifications for IT equipment. Ensuring compatibility between the two types of equipment is not always a simple exercise of comparing ambient air temperature, relative humidity and dew point values. Both types of equipment, power and IT, need to be compatible with the environment of the physical space where they are located. Some power equipment located is located in the hot aisle exhaust of the IT equipment which is a different environmental condition than the classes described in the ASHRAE thermal guidelines. In general the thermal and environmental requirements for power equipment are much less stringent than for IT equipment. 2. Changes in Data Center Environments Changes to data center environmental conditions are being driven by the need to save energy and reduce operational expenses. One of the largest operational expenses in delivering IT is the cost of energy. In a traditional data center, cooling costs alone can easily represent 25% or more of total energy costs [5]. Many data centers now run several degrees warmer compared to 10 or 15 years ago to save on cooling costs [6]. New energy saving technologies such as air-side economization and water-side economization are growing in adoption, whose hours of beneficial use increase as computer room temperatures increase. 2.1 ASHRAE Thermal Guideline Classes for IT Equipment Spaces ASHRAE issued its first thermal guidelines for data centers in 2004 [1]. The original ASHRAE air temperature recommended envelope for data centers was 20-25°C (68-77°F). This was a conservative statement, based on data available at the time, on where a data center could be reliably
- 11. 11 operated. Reliability and uptime were the primary concerns and energy costs were secondary. Since then, ASHRAE has issued a recommended range of 18-27°C (64-81°F) and, in 2011, published classes [1] that allow a temperature range of 5 to 45°C (41 to 113°F). The A3 (40°C, 104°F) and A4 (45°C, 113°F) classes were created to support new energy saving technologies such as economization. A summary of the ASHRAE recommended range and classes are given in the table below. Table 2 ASHRAE 2015 Thermal Guidelines [1] Classa Equipment Environmental Specifications for Air Cooling Product Operationsb,c Product Power Offc,d Dry-Bulb Temperaturee,g ˚C Humidity Range, Non-Condensingh,i,k,l Maximum Dew Pointk ˚C Maximum Elevatione,j,m m Maximum Temperature Changef in an Hour (˚C) Dry-Bulb Temperature ˚C Relative Humidityk % Recommended (Suitable for all 4 classes) A1 to Α4 18 to 27 -9˚C DP to 15˚C DP and 60% RH Allowable A1 15 to 32 -12˚C DP & 8% RH to 17˚C DP and 80% RHk 17 3050 5/20 5 to 45 8 to 80 A2 10 to 35 -12˚C DP & 8% RH to 21˚C DP and 80% RHk 21 3050 5/20 5 to 45 8 to 80 A3 5 to 40 -12˚C DP & 8% RH to 24˚C DP and 85% RHk 24 3050 5/20 5 to 45 8 to 80 A4 5 to 45 -12˚C DP & 8% RH to 24˚C DP and 90% RHk 24 3050 5/20 5 to 45 8 to 80 B 5 to 35 8% to 28˚C DP and 80% RHk 28 3050 NA 5 to 45 8 to 80 C 5 to 40 8% to 28˚C DP and 80% RHk 28 3050 NA 5 to 45 8 to 80 Notes for Table 2, 2015 Thermal Guidelines—SI Version a. Classes A3, A4, B, and C are identical to those included in the 2011 edition of Thermal Guidelines for Data Processing Environments. The 2015 version of the A1 and A2 classes have expanded RH levels compared to the 2011 version. b. Product equipment is powered ON. c. Tape products require a stable and more restrictive environment (similar to 2011 Class A1). Typical requirements: minimum temperature is 15°C, maximum temperature is 32°C, minimum RH is 20%, maximum RH is 80%, maximum dew point is 22°C, rate of change of temperature is less than 5°C/h, rate of change of humidity is less than 5% RH per hour, and no condensation. d. Product equipment is removed from original shipping container and installed but not in use, e.g., during repair, maintenance, or upgrade. e. Classes A1, A2, B, and C — De-rate maximum allowable dry-bulb temperature 1°C/300m above 900m. Above 2400m altitude, the de-rated dry-bulb temperature takes precedence over the recommended temperature. Class A3 — De-rate maximum allowable dry-bulb temperature 1°C/175m above 900m. Class A4 — De-rate maximum allowable dry-bulb temperature 1°C/125m above 900m. f. For tape storage: 5°C in an hour. For all other ITE: 20°C in an hour and no more than 5°C in any 15 minute period of time. The temperature change of the ITE must meet the limits shown in the table and is calculated to be the maximum air in let temperature minus the minimum air inlet temperature within the time window specified. The 5°C or 20°C temperature change is considered to be a temperature change within a specified period of time and not a rate of change. See Appendix K of reference [1] for additional information and examples. g. With a diskette in the drive, the minimum temperature is 10°C (not applicable to Classes A1 or A2).
- 12. 12 h. The minimum humidity level for Classes A1, A2, A3, and A4 is the higher (more moisture) of the 12°C dew point and the 8%RH. These intersect at approximately 25°C. Below this intersection (~25°C) the dew point (–12°C) represents the minimum moisture level, while above it, RH (8%) is the minimum. i. Based on research funded by ASHRAE and performed at low RH, the following are the minimum requirements: 1) Data centers that have non-ESD floors and where people are allowed to wear non-ESD shoes may want to consider increasing humidity given that the risk of generating 8kV increases slightly from 0.27% at 25% RH to 0.43% at 8% RH (see Appendix D of reference [1] for more details). 2) All mobile furnishing/equipment is to be made of conductive or static dissipative materials and bonded to ground. 3) During maintenance on any hardware, a properly functioning and grounded wrist strap must be used by any personnel who contacts ITE. j. To accommodate rounding when converting between SI and I-P units, the maximum elevation is considered to have a variation of ±0.1%. The impact on ITE thermal performance within this variation range is negligible and enables the use of rounded values of 3050m (10,000ft). k. See Appendix L of reference [1] for graphs that illustrate how the maximum and minimum dew-point limits restrict the stated relative humidity range for each of the classes for both product operations and product power off. l. For the upper moisture limit, the limit is the minimum absolute moisture of the DP and RH stated. For the lower moisture limit, the limit is the maximum absolute moisture of the DP and RH stated. m. Operation above 3050m requires consultation with IT supplier for each specific piece of equipment. Applying the ASHRAE thermal guideline classes listed in Table 1 looks straightforward but, in practice, an older data center may have equipment that was designed to different versions of those guidelines. For example, the recommended envelope from the 2004 ASHRAE guidelines was 20 – 25°C (68-77°F) whereas newer equipment is typically designed to Class A2 (35°C, 95°F) or even A3 (40°C, 104°F) and A4 (45°C, 113°F). The life of a typical data center is 15 to 20 years so some of the data center equipment may be original to the time the data center was built and some will have been recently refreshed. One of the challenges with implementing new energy saving cooling technologies, such as economization, is identifying an environmental control window that is compatible with all of the different thermal specifications, types of equipment, and equipment vintages in the data center. 2.2 Increasing Use of Economization in IT Equipment Spaces Economization is the use of outdoor air to cool the data center [7].The premise of economization is simple – why run an air conditioner or chiller if there is already an unlimited supply of cold air outside? Many world-wide locations can economize for as much as 50% of the hours in a year within the ASHRAE recommended range of 18-27°C (64-81°F). Larger percentages of economization (energy savings) require a wider allowable temperature and humidity range. There are several forms of economization: air-side economization [7], water-side economization [7], and refrigerant economization. Air-side economization typically accomplishes cooling by bringing filtered outside air directly into the data center without any air conditioning or humidity control. In winter months when the outside air is cold, a comfortable working temperature inside the data center is maintained by mixing the incoming air with hot exhaust air from the IT equipment. It should be noted there are forms of indirect air-side economization that use a thermal wheel, heat pipes, or a plate air–to-air heat exchanger [8]. These indirect air-side economization methods bring only a very small amount of outside air into the IT space of the data center. The disadvantage of a wheel or plate is the heat exchange step required makes it less efficient in full economizer cooling mode compared to air-side economization. The disadvantage of direct air- side economization is, depending on the local air quality, it can bring pollutants into the data center [9] which can require additional filtration to remove. Another disadvantage of direct air-side economization is, in some climates, the humidity of the outdoor air is outside the recommended (or allowable) limits for the IT equipment. This can preclude use of air-side economization, even
- 13. 13 when the outdoor temperature is favorable. Direct air-side economization can also make data centers prone to larger rates of change of temperature and humidity. Water-side economization uses outdoor air for data center cooling, but indirectly. Outdoor air is used to chill liquid and the liquid is piped into the data center where air-handling units (AHUs) and/or computer room air handlers (CRAHs) use an air-liquid heat exchangers called cooling coils to cool the data center air. The advantage of water-side economization is it doesn’t bring outside air into the data center and it allows tight humidity control similar to an HVAC cooled data center. Refrigerant economization is a variant of air conditioning in which the refrigerant bypasses the compressor on the way back to the condenser outside the data center [10] [11]. The energy that would have been used by the compressor is what is being saved. A purpose-built air conditioning system is required for refrigerant economization. The efficiency of both air-side and water side economization can be improved by the addition of evaporative or adiabatic cooling. For water-side economization, the cooling of the water can be improved with the use of an open tower where the heat exchange surfaces are wet and the evaporation of water improves the cooling efficiency. Evaporative cooling is also sometimes used with direct air-side economization - the air entering the data center is passed through an evaporative cooling media where the air is cooled by the addition of more moisture. An example of a direct air-side economized evaporative cooled modular data center is shown below in Figure 5. Figure 5 Air-side economized modular data center with evaporative cooling module. 2.3 Rising Exhaust Temperature of IT Equipment In addition to energy saving improvements in data center cooling, significant improvements have also been made in the energy efficiency of the IT equipment itself. One of the most common energy saving improvements is in the fan speed algorithm the equipment uses to regulate fan speed. Most IT equipment now has a control algorithm that reduces the fan speed at low workloads when very little cooling is needed. As workloads increase these new energy efficient fan algorithms increase the fan speed only as much as needed to keep component and sub-assembly temperatures inside the equipment within design limits. The consequence of this design change has been an increase in exhaust air temperatures of almost 10°C (18°F) over the last 8 years (see Figure 6).
- 14. 14 Exhaust temperature rise for air inlet temperatures of 35°C (95°F) and 45°C (113°F) are shown in Figure 7 and Figure 8. As the inlet air temperature rises to 35°C (95°F) and 45°C (95°F), most servers reach a maximum exhaust air temperature somewhere between 58 and 60°C (140°F). For power distribution infrastructure (i.e. line cords, facility receptacles, etc.) located in a hot aisle or exhaust path of IT equipment there may be operating limitations due to: 1) maximum rated temperature of a component, which may be addressed by using de-rating, or 2) maximum ambient temperature as referenced in some IEC standards like IEC 60309-1 [12], 60309-2 [13], 60320, etc. [14]. Many ITE manufacturers provide Unified Extensible Firmware Interface (UEFI)/BIOS settings to increase fan speeds to increase cooling and reduce exhaust temperatures to alleviate this concern. Contact your specific IT Equipment manufacturer for more information. In some applications, a higher temperature rated appliance or interconnection coupler may be required. In the future more harmonization of product safety standards, such as IEC 60950-1, may be desirable to embrace maximum rated temperature versus an ambient environment requirement with temperature rise since in an IT environment, the local temperature of a component is a combination of ambient, pre-heat, temperature rise, etc. Figure 6 Server T projection at 25°C (77°F) server inlet temperature [15].
- 15. 15 Figure 7 Server T projection at 35°C (95°F) server inlet temperature [15]. Figure 8 Server T projection at 45°C (113°F) server inlet temperature [15]. Because of space and air flow constraints, the exhaust air stream is sometimes used to cool power equipment such as rack PDUs and IT equipment such as top of rack switches. However, with the >10°C (18°F) rise in exhaust air temperature, the exhaust air stream may no longer be suitable for cooling or the equipment in the hot exhaust air stream may need to be redesigned to accommodate the higher exhaust air temperatures. Power equipment needs to tolerate recent changes in the data center environment such as the increase in exhaust air temperatures. The industry needs to make sure the thermal specifications of power equipment are aligned to the specifications of the IT equipment, where both types of equipment share the same physical space and same air stream. 2.4 Air Temperature Trends in IT Support Equipment Spaces IT support equipment spaces include areas for power equipment, a battery room for UPS batteries, and rooms for mechanical equipment such as air handlers. In general, these spaces have an
- 16. 16 environmental control system that is separate from the IT equipment portions of the data center. Most power equipment in IT support spaces produces only a small amount of heat compared to a similar square footage of IT equipment (Note: UPS units can generate quite a bit of heat and do require a significant amount of cooling). Most power equipment in IT support spaces produces so little heat that the majority of the equipment designs are natural convection cooled. Thus, the energy savings of running the IT support equipment spaces at a higher temperature or with economization is relatively small. In the case of an IT support space such as a UPS battery room, the lifetime of the back-up batteries is very sensitive to temperature and the battery room needs to be maintained within a narrow temperature range specified by the battery manufacturer. For a number of reasons, the air temperature of IT support equipment spaces has remained relatively constant and will probably stay that way for the near future. 2.5 Thermal Transients and Excursions in IT Equipment Spaces Thermal transients and temperature excursions are an important design consideration for both IT equipment and for power equipment that shares the same physical space. Thermal transients can be caused by a number of factors including: Changes in IT equipment work load Use of economization An HVAC failure Power failure (will cause HVAC shut down) The 2015 ASHRAE thermal guidelines in Table 2 contain an important change in the way temperature rate of change is specified – the new guideline calls out a maximum temperature change in an hour rather than an absolute rate of temperature change. In a traditional data center cooling system the cold-aisle air temperature is typically controlled within a narrow range of 1- 3°C (2-5°F). Chiller stages or direct expansion (DX) air conditioning units are cycled on and off to maintain this narrow air temperature range. However, chiller/DX cycling can create very high instantaneous rates of air temperature change, though over a very small range of temperature, i.e. 1-3°C (2-5°F) at most. Thermal studies carried out by ASHRAE show most components inside data center equipment, such as hard drives, have enough thermal mass that they are almost unaffected by these high instantaneous rates of air temperature change. The time lag for these components to respond to a temperature change is much longer than the duration of the change. Thus, the new ASHRAE temperature change guideline takes thermal lag into account and is intended to keep critical components, such as hard disk drives, within the supplier’s specified range for temperature change. Traditional chiller and direct expansion based cooling technologies have worked well in data centers for years without causing equipment problems due to rapid temperature rate of change. The new ASHRAE guideline change makes it clear that expensive changes to existing data center cooling infrastructure to meet an instantaneous rate of temperature rate of change requirement are not necessary. Changes in IT work load will change the amount of heat the IT equipment is generating affecting the temperature of the exhaust air in the hot aisle. For example, this will have a direct impact ambient air temperature surrounding rack PDUs, which are usually located in the back of the IT equipment racks. Work load changes will also change the loading on associated power equipment, changing the temperature of the conductors inside that power equipment. Workloads can vary
- 17. 17 with the time of day and with seasons (e.g. major shopping holidays, stock trading days with heavy volume, etc. depending on the type of customer the IT equipment supports). Economization is a common cause of thermal transients in some data centers. In an economized data center, the ambient air temperature floats within a predetermined range that can be much wider than the control range of a typical HVAC data center. HVAC data centers are usually controlled within a fairly narrow range of about 2°C (3.6°F) whereas the control range for an economized data center could be 18 to 27°C (64°F to 81°F) or even wider. In an economized data center the cold aisle ambient air temperature will change depending on the conditions outside the data center. The temperature of most economized data centers will show a daily sinusoidal variation over time as warm day time temperatures give way to cooler night time temperatures and so forth as well as a seasonal variation. Depending on how the economization is implemented in the data center environmental control system, it may be possible to have cold aisle air temperature excursions as high as 40 or even 45°C (104°C or 113°C) (ASHRAE Class A3 or A4). It is important to note these are typically short term (hours) excursions and not continuous operation at high temperature. The impact of short high temperature excursions to the time weighted average failure rate over a long period of time, such as a year, is very small. For example, in an economized data center, rack PDUs must be carefully selected so their rating meets or exceeds the maximum exhaust temperature of the IT equipment during the highest inlet air temperature excursion and workload conditions expected. Chiller or HVAC failures are unusual but they can occur and, when they do, a temperature rise in the cold aisle air temperature of 30°C (54°F) in as little as 5 minutes is not uncommon. The amount of time one has before IT equipment starts going into thermal shutdown is called ride through time. During an HVAC failure event the power draw of the IT equipment will go up as fans inside the IT equipment speed up to try to cool the equipment. This will cause an increased power demand which will cause a conductor temperature rise inside the power equipment. The power infrastructure of a data center needs to be designed to handle the increased air temperature and power demand from an HVAC failure for a long enough period of time that back-up power and cooling could be brought online. It should be noted that some facilities have thermal storage, such as a tank of ice or chilled water, to provide additional ride through time until the chillers or HVAC units come back on line. 3. Temperature Ratings for Power Equipment Temperature ratings for most power equipment are determined differently and defined differently than for IT equipment. For IT equipment, maximum temperature ratings are typically determined by detailed thermal analyses. The maximum temperature rating is usually a fixed limit at which the cooling capability of the IT equipment reaches the maximum temperature specification limits of key components such as the processor, DIMMs, hard disk drives, or power supply. Most power equipment has a maximum temperature rating at full power load. However, unlike IT equipment, this maximum temperature rating is flexible and it is a function of several factors including the maximum temperature rating of the conductors, the temperature rise of the conductors from the power loading (ampacity), the ambient air temperature, and other electronic components in the equipment such as electronic trip units and ATS controllers. If the maximum ambient air temperature is determined from the conductor temperature alone then the calculation is given by
- 18. 18 Maximum Tc = Tc + Ta Where Maximum Tc is the maximum conductor temperature rating, Ta is the ambient air temperature and Tc is the temperature rise of the conductor caused by the power loading. If the terms in the equation above are re-arranged, the allowable ambient air temperature is simply the maximum conductor temperature rating minus the temperature rise of the conductor from the power loading. Ta = maximum Tc - Tc For example, consider a power busway with an ambient air temperature rating of 40°C (104°F) at full power load. The busway can be used at temperatures above 40°C (104°F) if the power loading is reduced so the conductor temperature rise is also reduced and the conductor stays at or below its maximum temperature rating, Tc. Figure 9 below is an example of a set of de-rating multipliers for a busway. The multiplier value on the y-axis is used to calculate a reduced maximum power load for a given ambient air temperature shown on the x-axis. Figure 9 Example of power multipliers showing how power is de-rated (reduced) at ambient air temperatures above the full power capacity rating of 40°C (104°F). While power de-rating is an important consideration, it is also important to consider the temperature capability of other electronic components inside power equipment such as electronic trip units and ATS controllers. High operating temperatures could cause these components to lose functionality. Sustained high operating temperatures could also shorten the lifetime of these components. While power de-rating is a very important factor in determining operating ambient air temperature, the air temperature determined from a power de-rating analysis must be balanced against the temperature capability of all of the components that make up a given piece of power equipment. 0.5 0.6 0.7 0.8 0.9 1 1.1 30 35 40 45 50 55 60 65 70 75 80 PowerMultiplier Ambient Air Temperature (°C)
- 19. 19 Another important factor that can impact the maximum ambient air temperature for power equipment is touch temperature. The temperature of handles, receptacles, and flat surfaces a user might touch needs to stay within comfort and product safety limits. A widely used international touch temperature standard is IEC60950-1 shown below in Table 3. Table 3 Touch temperature limits from IEC60950-1 [16]. Object Being Touched Touch Duration Metal Glass and Similar Plastic and Rubber Handles Briefly 60°C 70°C 85°C Handles Longer Duration 55°C 65°C 75°C External surfaces and parts inside that may be touched NA 70°C 80°C 95°C Note: The exact duration of the terms “briefly” and “long duration” are not defined in IEC60950- 1. For some power equipment, the maximum ambient air temperature rating is a combination of conductor temperature rise and touch temperature standards. 4. Medium and Low Voltage Switchgear Medium and low voltage switchgear is typically located after the utility and before the UPS gear (see Figure 1). In a data center power distribution system, the medium and low voltage switchgear is a combination of electrical disconnect switches and circuit breakers used to control, protect, and isolate electrical equipment. Examples of a low voltage switchgear and distribution switchgear are shown below in Figure 10 and Figure 11.
- 20. 20 Figure 10 Low voltage switchgear Figure 11 Distribution switchgear Most data center switchgear equipment is natural convection cooled. Perforated panels or grilles are strategically located to facilitate natural convection based cooling. The amount of heat generated inside the switchgear is dependent on loading. The power carrying capability of the components inside the switchgear is a function of temperature, decreasing as temperature of the components increases. Switchgear is carefully designed to meet maximum rated loads with natural convection cooling alone and minimal temperature rise of components. Per the IEC 61439 standard for low-voltage switchgear and control gear assemblies [17], the ambient temperature specification for low and medium voltage switchgear is a 35°C (95°F) maximum air temperature with a 2 hour excursion up to a 40°C (104°F) air temperature allowed. Switchgear can be susceptible to high levels of humidity near the condensing limit. In some cases, high humidity levels can cause leakage and even arcing. However, most switchgear is designed with a hygrostat and a panel heater to avoid condensation. By heating the air inside the switchgear cabinet the panel heater reduces the relative humidity of the air effectively mitigating the effect of the high humidity and preventing condensation. Most switchgear has arc protection relays and sensors as well as a ground fault relay to protect the switch in case it is inadvertently exposed to very high levels of humidity. In general, most electrical power distribution equipment, including switchgear, generates very little heat. The main exceptions to this are transformers and UPS systems. Switchgear rooms require only a small amount of cooling which can be provided by a number of different means but typically by traditional air conditioning. Care must be exercised in designing the cooling system if the switchgear is in the same room or shares an air supply with batteries. Batteries can generate hydrogen and building safety codes usually require a minimum rate of air exchange with the air
- 21. 21 exhausted directly to the outdoors. In some northern climates, it may be possible to re-use the waste heat from some power equipment, such as transformer and UPS spaces, to heat an adjacent space such as a battery room. Airborne contamination and resulting corrosion [9] can affect the reliability and the lifetime of switchgear. High humidity can accelerate the effect of many types of airborne corrosion. High levels of outdoor air pollution in newly industrialized parts of the world can create slightly corrosive environments inside data centers and the power spaces of those data centers. Most switchgear is designed to have a moderate amount of resistance to corrosion. The most common corrosion design standard is class C1 of ISO 12944 [18]. Switchgear is also tested for airborne salt dust and most metal parts are protected by special coatings. 5. Uninterruptible Power Supplies The purpose of an uninterruptible power supply unit is to provide emergency power when the input power source, typically mains utility, fails. A UPS differs from an auxiliary or emergency power system or standby generator in that it will provide seamless protection from input power interruptions, by supplying energy stored in batteries, super-capacitors, or flywheels. The runtime of a battery based uninterruptible power source is highly dependent on the loading and can range from 15 minutes at full load to longer times at reduced loading. Some battery based telecom UPS units are designed to provide 8 hours or more of run time. Flywheel and super-capacitor based UPS units provide power for a duration of 15 to 30 seconds to a few minutes which is relatively short, but sufficient to start a standby power source or properly shut down the protected equipment. UPS units are used to protect data centers against unexpected power disruptions that could result in service interruptions or even data loss. References [19] and [20] have a good discussion of different UPS system design configurations and configurations that make the most sense for a given set of parameters including availability needs, risk tolerance, types of loads in the data center, budgets, and existing infrastructure. Uninterruptible power supply units come in a variety of sizes: monolithic, single and multi-rack, and modular rack mount systems (see Figure 12, Figure 13, and Figure 14) as well as even smaller units that fit in place of an IT equipment power supply inside the IT equipment.
- 22. 22 Figure 12 Large monolithic data center UPS. Figure 13 Examples of smaller rack-based UPS units. Figure 14 Example of a small rack mount UPS. Uninterruptable power supplies utilize a variety of energy storage technologies including: battery (see Figure 15), flywheel (see Figure 16), and super-capacitors [21]. One of the important differences in these energy storage technologies is the time scale over which they can provide backup power. In general, batteries provide the longest duration of backup power, typically at least 15 minutes and sometimes hours or more. Flywheels typically provide power on the order of 15 to 30 seconds at rated load (and in some cases up to several minutes especially at low load conditions). Most super-capacitors are capable of providing power for much less than a minute [21]. It should also be noted that energy storage capacity (batteries, flywheels, etc.) are typically sized to meet a time requirement based on a maximum full rated load. When there are redundant UPS systems then each system is sized for the rated load and therefore during normal operations where both systems are supporting half the actual load, the installed capacity is doubled. When the actual load is less than the full design rated load as is the case in all but the most heavily loaded systems, the spare capacity can be significantly greater still.
- 23. 23 Figure 15 Data center UPS battery room with flooded cell type batteries. Figure 16 Example of flywheel technology UPS with flywheel shown in front. For purposes of this paper, the attribute we are most interested in is the thermal compatibility of power equipment with the IT equipment when the two types of equipment share the same physical space and air supply. For thermal compatibility, UPS systems can be grouped into three broad categories: 1) monolithic data center UPS systems or large flywheel systems; 2) distributed UPS, and 3) IT equipment level battery backup unit (BBU). The first case of large monolithic data center UPS units is simple because these systems and their associated battery strings or flywheels, and other infrastructure are nearly always contained in their own independently cooled grey area of the data center. That is, the air temperature of these systems is completely independent and won’t be affected by the temperature of the IT equipment. However, the temperature of batteries in the UPS systems does need to be closely controlled. The most common battery technology is lead-acid of which there are two types: vented lead acid (VLA) and
- 24. 24 valve regulated lead acid (VRLA). Lead acid batteries are a proven technology and they are one of the most cost effective storage technologies available. The backup power source for large monolithic UPS systems can also be comprised of a number of flywheel units. Flywheel systems can tolerate a much wider temperature range (0 to 40°C, 32 to 104°F) than batteries (15 to 25°C, 59 to 77°F). As with batteries, flywheel systems have relatively low power dissipation and require only a small amount of cooling compared to IT equipment. Flywheels can be a cheaper alternative to batteries when only short term (on the order of a minute) back up power is required. The disadvantage is flywheel systems provide power only long enough for a diesel generator to start up and take over providing power to the IT equipment. Most communities have limits on the number of hours a facility can run diesel generators. Like battery based UPS systems, large monolithic flywheel UPS systems are also in their own separate environmentally controlled section of the data center where they are unaffected by the temperature of the air streams associated with the IT equipment. The next type of UPS solution is a distributed UPS solution. These UPS units are designed into data center style racks and, when located in data halls, can share the same space and air streams with the IT equipment they support. When considering a distributed UPS solution, there are several factors to weigh, including: battery technology, electrolytic capacitors and semiconductor junction temperature. Distributed UPS units are typically very compatible with IT equipment – nearly all distributed UPS units pull air from the cold aisle and they all use variable speed fans that are controlled via internal thermal sensors. There is a long history of robust thermal design with this type of UPS. The most common battery technology for distributed UPS units is valve regulated lead acid (VRLA) batteries. VRLA batteries are different from flooded/vented lead acid batteries in that they emit little or no hydrogen gas and don’t require a special room or special ventilation requirements. The third type of UPS unit is a small battery backup (BBU) that resides inside the IT equipment, usually with the same form factor as a power supply unit. For BBUs, the most common battery technology is lead-acid VRLA though some BBUs use lithium ion (LiON) batteries [22]. One of the reasons for choosing a battery technology, like LiON, is that it is tolerant of higher temperature. The location of the BBU is typically at the rear of the IT equipment where the cooling air has already been significantly pre-heated. Air temperatures in the range of 50o C (122°F) are not uncommon in the rear chassis of most IT equipment. Temperatures above 50o C (122°F) will adversely affect the service life of the battery. Another important aspect of defining battery temperature is, during discharge mode, the batteries may have as much as a 30°C (54°F) rise in temperature, depending on the current draw and the discharge time. After a discharge, the batteries must be allowed to cool before a charge cycle can be initiated. After a discharge event there will be a time window where the IT equipment is vulnerable to a back-to-back outage event where, for the second outage, the BBU would not have energy stored energy to provide back-up power. IT equipment BBUs are inside the data center space and they will experience the same air streams, temperature and humidity conditions as would a power supply inside a piece of IT equipment. This type of backup power deployment is not very common but it illustrates the challenges of placing a batteries inside IT equipment where a significant amount of preheating can take place. In the case of a BBU inside a piece of IT equipment, VRLA batteries would not be a suitable choice.
- 25. 25 Discussion of Super-Capacitor Technology for UPS Systems An important class of charge storage devices is the so called “ultra-capacitors” or “super-capacitors” [23]. These devices are used in some UPS systems to provide short term power back-up. The most common type of super-capacitor is an electric double-layer capacitor (EDLC). Super- capacitors are an attractive alternative to lead-acid batteries because they have: a) a large number cycle times along with fast charging times, b) higher power density and faster energy release, c) are smaller and lighter than batteries, d) are environmentally friendly, and e) generally have lower maintenance costs and higher reliability than batteries. However, super-capacitors have some significant disadvantages relative to batteries and other types of capacitors including: a) low operating voltages, b) shorting hazard due to low internal resistance, and c) elevated temperatures tend to shorten the life of super-capacitors more than batteries or electrolytic capacitors. A quantitative comparison of super-capacitor performance parameters to lead acid batteries and electrolytic capacitors is shown below in Table 4. Note how the maximum temperature rating for super-capacitors is only 70°C (158°F) whereas most electrolytic capacitors are rated to 85 or 105°C (185 or 221°F). Also, the typical rated super-capacitor lifetime of 1,000 hours sounds like a lot but it is only 6 weeks (at the maximum rated temperature of 70°C (158°F)). Table 4 Comparison of super-capacitor performance to lead acid batteries and electrolytic capacitors [23]. Performance Parameter Lead Acid Battery Super-Capacitor Electrolytic Capacitor Charge Time 1 to 5 hours 0.3 to 30 s 10-3 to 10-6 s Discharge Time 0.3 to 3 hours 0.3 to 30 s 10-3 to 10-6 s Energy Density (Wh/kg) 10 – 100 1 – 10 <0.1 Power Density (W/kg) <1,000 <10,000 <100,000 Cycle Life 1,000 >500,000 >500,000 Charge/discharge efficiency 0.7 – 0.85 0.85 - 0.98 >0.95 Operating Temperature Range 15 to 25°C (60 to 77°F) -40 to 70°C -40 to 105°C Typical Lifetime Specification Rating 3 to 20 years 70°C for 1,000 hours 85 or 105°C for 1,000 or 2,000 hours Consider, for example, a super-capacitor rated for 1,000 hours (6 weeks) at 70°C (158°F) and 2.7V [24]. According to the manufacturer, the lifetime of the capacitor doubles for every 10°C below the maximum rated temperature of 70°C (158°F). Using the capacitor below its maximum rated voltage increases the lifetime by 1X for 2.7V (maximum rating), 4.5X for 2.25V, and 16X for 1.8V. The adjusted lifetime of the capacitor is the product of: a) the lifetime extension factor from temperature, b) the lifetime extension factor from reducing the voltage, and c) the base lifetime of the component, in this case 1,000 hours. Assume, for sake of this example, the UPS unit containing the super-capacitors, needs to last for at least 10 years. In order to reach a design goal of 10 years,
- 26. 26 the super-capacitors need to be de-rated, i.e. used at less than their maximum temperature and voltage rating in order to increase the component lifetime beyond 1,000 hours (6 weeks). Components inside a UPS chassis can generate some heat so the temperature of the super-capacitor components may be slightly higher than the temperature of the surrounding air outside the UPS cabinet. This temperature rise needs to be carefully measured and taken into account for lifetime calculations. A significant amount of voltage and temperature de-rating needs to be applied to the super-capacitors to achieve a >10 year lifetime as shown in Figure 17 below. The de-rating constraints to achieve 10 years are such that the super-capacitors cannot be used at their maximum rated voltage of 2.7V. For operational voltages of 2.25 and 1.8V a 10 year lifetime is met only if the maximum super-capacitor component temperature is maintained at or below 27°C (81°F) and 46°C (115°F), respectively. Figure 17 Illustration of lifetime de-rating trade-offs to meet a 10 year design goal for a super- capacitor with a 1,000 hour rating at 70°C (158°F) and 2.7V [24]. Another important point to consider for the application of super-capacitors is the non-operational (both shipping and storage) temperature range of these components. The recommended storage temperature for some super-capacitors is as low as 35°C (95°F) [25]. This is well below the 45°C (113°F) temperature listed in the power off conditions of ASHRAE classes A1, A2, A3 and A4. Some types of super-capacitors are also sensitive to high humidity levels in the range of 70%RH and above [25]. Thus, under some circumstances, it could be possible for a piece of UPS equipment to be shipped to a customer, see high temperatures and high humidity levels for a prolonged period of time during transit, and arrive at a data center site with compromised super-capacitors. UPS equipment designers should pay close attention to the temperature, voltage and lifetime ratings of super-capacitor components. Supplier provided component de-rating guidelines should be carefully followed. Environmental usage conditions (temperature, humidity) in customer facing user manuals for the UPS unit should align with the data sheet of the super-capacitors and the de- rating assumptions applied in the equipment design. Both customers and UPS equipment suppliers should pay close attention to non-operational shipping and storage conditions, especially during warm summer months. UPS units with super-capacitors should be shipped and stored with a temperature/humidity data logger to verify that none of the non-operational temperature and
- 27. 27 humidity limits for the super-capacitors were exceeded. Data center operators should read UPS user manuals carefully and maintain the temperature and humidity of IT spaces and grey area spaces well within those published ranges to ensure the full design lifetime of the super-capacitors. Discussion of Battery Technologies for UPS Systems Two battery technologies are in widespread use in UPS systems: flooded/vented lead acid, and valve regulated lead acid (VRLA). Lithium ion (LiON) batteries are not yet in widespread use but are considered a promising new technology. A high level comparison of the different battery types is given below in Table 5. Table 5 Temperature and service life ranges for common UPS battery technologies. Flooded Vented Wet Cell Lead Acid Valve Regulated Lead Acid (VRLA) Lithium Ion (LiON) Temperature Range 15 to 25°C (60 to 77°F) 15 to 25°C (60 to 77°F) -40 to 60°C (-40 to 140°F) Recommended Service Life Range 15 to 20 years 3 to 10 years* 6 to 20 years *Newer pure lead VRLA batteries can support an even longer service life. Most flooded/vented lead acid and valve regulated lead acid batteries (VRLA) are lead-acid based technologies but with key differences. The electrolyte of flooded/vented battery is liquid whereas the electrolyte of the VRLA is typically contained in a gel or a glass matt. Flooded batteries are vented and can evolve hydrogen during use [26] (especially during recharge). These batteries require a separate room with its own air supply and enough turn-over of air supply to prevent a build-up of flammable hydrogen gas. By comparison, VRLA batteries evolve very little hydrogen and can be used in IT equipment spaces without special ventilation. Significant advances have been made in the design and construction of VRLA batteries. The most significant advance has been improvements in the purity of the lead used in the plates, the grid, and the lead paste. Higher purity lead reduces the impact of corrosion which: 1) prolongs battery life by as much as 20 to 100% compared to non pure-lead VRLA batteries, 2) improves the high temperature tolerance of pure-lead VRLA batteries, and 3) improves battery shelf life by up to 4X. In general, there are several important factors that determine the service life of lead acid batteries used in UPS systems: 1) the number of cycles; 2) the number of deep discharges; 3) amount of time exposed to a low state of charge; and, 4) the time averaged temperature of the battery over its life. A discussion of the service life impact of number of cycles, depth of discharge, and periods of low state charge is available from the battery suppliers and is beyond the scope of this paper. The impact of temperature alone is fairly well understood. Both types of lead acid batteries require a fairly narrow air temperature range of 15 to 25°C (59 to 77°F) with 25°C (77°F) considered an ideal operating temperature. As a general rule of thumb, these batteries experience a lifetime reduction by half for every 8 to 10°C (14 to 18°F) increase above their ideal rated operating temperature of 25o C (77°F) [27] as show below in Figure 18. Just as higher ambient temperatures can reduce battery lifespans, colder ambient temperatures can reduce battery capacities, so the best
- 28. 28 practice is to maintain batteries within a fairly narrow range based upon the battery manufacturer’s recommendations. Figure 18 Typical lifetime of a lead-acid battery as a function of temperature. Note that 25oC (77oF) is a typical temperature for 100% of the battery manufacturer’s rated lifetime. Flooded wet cell batteries located in a separate battery room tend to stay at a fairly uniform ideal 25o C (77o F) temperature so temperature control and temperature fluctuations are typically not a significant factor that would reduce the service life of these batteries. For flooded wet cell batteries consult IEEE 450-2010 “IEEE Recommended Practice for Maintenance, Testing, and Replacement of Vented Lead-Acid Batteries for Stationary Applications” [28]. VRLA batteries and LiON batteries, however, may be located in the IT space and even within IT equipment racks. In a real-world data center, it is difficult to maintain perfectly uniform temperature control across the data center, from aisle to aisle, rack to rack, and along the height of a rack. Heat can even be transmitted to the batteries from IT equipment in adjacent racks. Battery temperature can change with charging or discharging. All of these factors must be considered when defining VRLA and LiON battery replacement schedules and service level agreements. Every effort should be made to maintain lead-acid VRLA batteries in IT areas and inside IT equipment as close to the ideal 25o C (77o F) temperature as possible. For information on prolonging the life of lithium ion batteries, consult references [29], [30]. In practice, the determination of battery life more nuanced than just temperature alone. For example, a single short (e.g. 100 hours) excursion to warm temperatures (e.g. 40°C (104°F)) won’t significantly reduce the battery life. However, sustained (7 days a week by 24 hours a day) operation at warm temperatures will have a significant impact. Actual battery life can be calculated by taking the histogram of time the battery spent within each range of temperatures (e.g. 20-25, 25-30, 30-35, and 35-40°C) over a year, the battery life curve in Figure 18, and using both sets of numbers to create a time-averaged battery life.
- 29. 29 The impact of battery temperature on battery lifetime and UPS performance becomes a very important consideration when a data center is considering economizing over a wide temperature range, i.e. outside of the ASHRAE recommended range of 18 to 27o C (64 to 81°F) and the UPS or DC Power systems will be co-located within the data hall. Prolonged exposure to warmer air temperatures can decrease battery life and necessitate a more frequent battery replacement schedule. When considering economization over a wide temperature range, one should make sure the UPS and associated batteries can provide adequate back up power over the entire range of expected data center temperatures. Temperatures below 18o C (64°F) could reduce battery power output and temperatures warmer than 27o C (81°F) could reduce battery lifetime. When redundant UPS systems are both available, the doubling of battery capacity should mitigate the reduced battery capacity due to colder temperatures. When either redundant system is out-of-service it may be advisable to curtail economizer operation. Another important consideration for distributed UPS systems, aside from economization, is to make sure the UPS units can still provide adequate backup power in the event of an HVAC system failure where air temperatures inside the data center can rise by as much as 30o C (54°F) in a matter of minutes. Locating UPS systems and their respective batteries in separate, grey area spaces with independent HVAC systems alleviates the concerns regarding economization of data halls. A complete discussion of battery technologies and their application is beyond the scope of this paper. However, several good general references on batteries are available [31] [32]. Key Questions for Selection of Battery Based UPS Equipment The selection of battery backup systems within the temperature variations of a data center facility environment requires an answer to a number of questions, essentially application criteria, which must be answered in advance such as: What type of equipment is to be supported (IT, cooling, distribution, power distribution and management systems)? How is this equipment to be supported from backup battery power (momentary, impulse, or continuous)? How critical is the load (service level agreement, reliability requirements)? Is the data center new construction or an existing facility (physical and location limitations)? What is the load size (usually given in kilowatt hours or megawatt hours)? What redundancy level is required (Tier 1 through Tier 4)? Are there any environmental considerations (e.g. California regional air-quality board)? What is the level of grid reliability? What is the anticipated operating environment (computer room through ambient outdoor environment)? What is the minimum anticipated duration that the battery is required to maintain the load (plus a safety factor)? What is the battery environment (temperature, pressure, humidity)? Is this a new or tested battery technology? Is this a new installation or retrofit? What level of business risk is the data center willing to tolerate? Floor load capacity (batteries are REALLY heavy)
- 30. 30 Code requirements for installing emergency eyewashes and showers in close proximity to battery systems (do you want a shower in your data hall?) The questions above take into account only physical installation criteria – they do not include environmental sustainability considerations which are also important. 6. Electrical Power Distribution This section details the equipment in the power path between the low voltage switch gear and the IT equipment with the exception of UPS systems which have been discussed in the previous section. The order of discussion below will generally follow the flow of the power distribution diagram in Figure 1. Although Figure 1 is a fairly typical power distribution configuration, data center power distribution systems may and often do differ significantly. 6.1 Room Power Distribution Unit (PDU) In the context of this discussion, a power distribution unit (PDU) is a device used in data centers to distribute AC power to multiple racks of IT equipment. These PDUs are typically located inside the IT equipment space of the data center. A floor-mounted PDU (see Figure 19 below), sometimes called a main distribution unit (MDU), provides an effective means for bringing power from upstream switchgear (either utility or UPS power) to the IT equipment space where it can be distributed to various equipment racks within a data hall, main MDF/IDF (main distribution facility/independent distribution facility) room, network operations center (NOC), or other IT space. PDUs typically bring power in at 480V (but depending on model can accept input voltages of 600V, 480V, 415V, 400V or 380V), have integral transformers to drop the voltage down to voltages acceptable to IT equipment, and then provide multiple output breakers to feed remote power panels (RPPs), wall panels, or rack PDUs. Each PDU can handle larger amounts of energy (300 kilovolt-amps and higher depending on the manufacturer and model) than an ordinary power strip (typically 1.4 to 15 kilovolt-amps) and typically provide power to multiple equipment racks. Standalone cabinet PDUs are self-contained units that include main breakers, individual circuit breakers, and power monitoring panels. These PDUs are sometimes combined with integral static transfer switches (STSs) that allow two diverse power sources to feed the PDU. The PDU cabinet provides internal bus bars for neutral and grounding. Per the block diagram power distribution system diagram in Figure 1, the PDU is downstream from the UPS and it is sometimes connected to a RPP and rPDU where power is fed to individual pieces of IT equipment (see Figure 20). In most cases, the PDU distributes power directly to the rack PDU. Cables are fed to the PDU either underneath through the floor or from overhead.
- 31. 31 Figure 19 Photo of a cabinet PDU of the type used to distribute power to multiple racks of IT equipment inside a data center. Figure 20 Location of PDU in the power supply chain. Most PDUs are designed to be cooled by natural convection. Openings in the raised floor for power cables should be sealed to prevent the escape of cold air from the underfloor air volume and a loss of data center cooling efficiency. Many PDUs have some form of built-in intelligent power monitoring capability. Power monitoring data can be used to calculate the Green Grid’s PUE (Power Usage Effectiveness) metric [33] [34]. Typical temperature and humidity ratings for floor mounted PDUs are 0 to 40°C (32 to 104°F) and either 0 to 95%RH or 10 to 95%RH. The typical PDU temperature and humidity ratings will meet ASHRAE Classes A1, A2, and A3 at maximum rated power but require de-rating (reduction) of maximum power in order to meet Class A4 which has a 45°C (113°F) maximum temperature requirement.
- 32. 32 6.2 Transformers Though not shown on the power distribution diagram of Figure 1, low voltage transformers are an important element of data center power infrastructure. Transformers are used between the medium and low voltage switch gear and as part of UPS and PDU systems. Note: the new version of the National Electrical Code (NFPA 70 [4]), which will be released in 2017, will define medium voltage as 600 to 1000V. In a data center, a transformer could be feeding either mission critical equipment or non-critical equipment. The two primary purposes of transformers are: voltage change and isolation [35]. Transformers are used to step down voltage from the medium supply voltage to low voltage supply. Transformers are also used to step down to the end utilization voltage. Within a PDU, for example, transformers are be used to step the voltage down from the data center distribution voltage of 480/277V to 240/120V or 208/120V (typical for North America). Isolation transformers are used to transfer power from an AC source to a piece of equipment while isolating that equipment from the power source, usually for electrical safety reasons. Isolation transformers provide galvanic isolation which can protect against electric shock, suppresses electrical noise in sensitive devices, allow for proper operation of over-current protection, and transfer power between two circuits which must not be connected. A dedicated isolation transformer is often built with special insulation between primary and secondary and is specified to withstand a high voltage between windings. The load at which a transformer will operate at peak efficiency is determined by the transformer design and varies widely among transformer models. Transformers should be selected so the point of maximum efficiency lies close to the typical loading. This is important because energy losses in transformers cause heat generation. High efficiency transformers and transformers that have been matched for efficiency to typical load conditions will reduce the heat contribution from transformers and reduce the incremental data center cooling costs from power equipment containing transformers. An example of a dry type isolation transformer is shown below in Figure 21. Figure 21 Example of a dry type distribution transformer. Transformers are not typically rated for a fixed ambient air temperature range or humidity range. Instead they are rated by the maximum allowable temperature of the transformer coils. The maximum temperature of the coils due to power is a function of the ambient air temperature and the temperature rise of the coils from power dissipation. Medium voltage and low voltage
- 33. 33 transformers can even have multiple ratings if fans are included instead of relying on natural convection cooling. 6.3 Remote Power Panel (RPP) A remote power panel (RPP) is a distribution panel that is downstream of and fed by a floor-mount PDU or transformer (see Figure 20). The reason for placing the PDU inside the IT space is to locate the point of distribution of moderately high current capacity close to the IT equipment loads. Remote power panels are like PDUs but without a transformer. Therefore, they are smaller having a footprint about the size of a standard raised floor tile. RPPs can contain as many as four panelboards and a monitoring system. RPPs distribute power to the rack PDUs inside the IT equipment racks. Remote power panels typically have a temperature rating of 0 to 40°C (32 to 104°F) and a humidity range of either 0 to 95%RH or 10 to 95%RH. This typical RPP temperature and humidity range is compatible with ASHRAE Classes A1, A2, and A3 at maximum rated power but requires de- rating (reduction) of maximum power to meet Class A4 which is a 45°C (113°F) maximum temperature. RPPs are nearly always natural convection cooled. 6.4 Panelboards Panelboards are typically a metal cabinet that houses the main electrical bussing and the terminals on which circuit breakers, neutral wires, and ground wires are installed. Panelboards can be found in mechanical and electrical spaces and also in IT spaces. Panelboards are used in place of RPPs to distribute power to utility/convenience outlets as well as lighting and security devices. HVAC equipment is typically 480V and is fed from motor control centers (MCCs). Panelboards are typically mounted against a wall or on steel bracings so they are only accessible from the front. Panelboards are essentially wall-mounted low voltage switchboards. Panelboards are typically the location for the last circuit breaker before the load in the electrical distribution system and all branch circuits are fed out of them to the load. A typical panelboard is shown below in Figure 22. Figure 22 Photo of typical data center wall-mounted panelboard.
- 34. 34 In general, minimum and maximum temperature and humidity ranges are not specified for panelboards. Instead, the maximum ambient temperature is determined by the temperature rating of conductor wire used and how the current carrying capacity of that wire has been de-rated for temperature. De-rating means that, at higher temperature, the maximum rated current carrying capacity of the wire is reduced. Panelboards are cooled by natural convection and typically they generate very little heat. Circuit Breakers Circuit breakers are a very important component used in most types of power equipment including panelboards. There are a wide range of different types and sizes of circuit breakers. The focus of this paper is mainly system level power equipment with only limited discussion of key components inside the equipment. While circuit breakers are a very important component, the level of detail required to properly discuss circuit breakers is beyond the scope of this paper. There are many good references on the types of circuit breakers used in data center power equipment. For more information, please consult the following references [36], [37], [38], and [39]. 6.5 Busways The National Electrical Manufacturers Association (NEMA) defines a busway as a prefabricated electrical distribution system consisting of bus bars in a protective enclosure along with insulating and support materials (see Figure 23 below). Busway systems include straight lengths of busway, fittings, devices, and accessories. A major advantage of busways is how easy the busway sections are to connect together. Busways are sometimes faster and cheaper to install than a comparable cable and conduit assembly and they can be installed either overhead or underneath a raised floor. In some cases, busways are used because they can carry higher current in a smaller footprint than is possible with cable. The power distribution system of data centers often consists of a combination of busway and cable and conduit. Figure 1 shows busways as an alternative to RPPs and panelboards for distributing power to IT equipment racks. Figure 23 Example of a modular busway system.
- 35. 35 A typical maximum ambient air temperature rating for a busway system is 40°C (104°F) with no more than 55°C (121°F) hot spot above ambient. A typical range for relative humidity is 0 to 95%RH with an additional stipulation the environment must be non-condensing. Busways are cooled by natural convection. 6.6 Rack Automatic Transfer Switches (rATS) and rack Static Transfer Switches (rSTS) A rack automatic transfer switch (rATS) provides redundant power to single-corded equipment. The rack ATS has dual input power cords supplying power to the connected load. If the primary power is unavailable, the rack ATS will seamlessly switch power to the secondary source within ½ of an AC cycle. A piece of single-corded IT equipment can ride through this minimal interruption via the bulk capacitance inside the power supply. Most rack ATS units have built-in network connectivity which enables remote management via web, SNMP, or Telnet interfaces. Another way of explaining the purpose of a rack ATS is to examine a typical IT equipment rack. The majority of data center IT equipment (e.g., server, storage, and switch) is dual-corded. Dual corded means that each piece of IT equipment has two power supplies and each power supply is connected to a different power source. Most racks have two rack PDUs, one on either side of the rack and each rack PDU draws power from a different leg of the separate and diverse power source. In the event of a power interruption to one leg of the grid, the IT equipment can run off just one of its power supplies without interrupting the functionality of the IT equipment. By comparison, single-corded equipment can draw its power from only one leg of the power grid. If that leg is affected by an interruption or an outage, then the single-corded equipment would be affected. The rack ATS takes its power from two legs of the power grid and gives the single-corded equipment a similar level of power redundancy as the dual-corded equipment. A good discussion of options for powering single-corded equipment in a dual-corded environment can be found in reference [36]. A photo of a rack ATS is shown in Figure 24. Figure 24 Photo of rack automatic transfer switch (rack ATS). Rack ATS units are typically mounted in the front of the rack so the switch and the power leg indicator panel is visible from the cold aisle. However, in some cases, rack ATS units may also
- 36. 36 be mounted in the back of the rack in the hot aisle. Maximum temperature ratings for rack ATS vary. Typical ratings are 45°C (113°F), 50°C (122°F) (Conformité Européenne, CE), and 60°C (140°F) (Underwriters Laboratories, UL) depending on the regulatory certification agency. Maximum humidity ratings are either 85 or 95%RH. These typical ratings meet ASHRAE classes A2 and A3 for cold aisle installations. Units with a 95%RH maximum relative humidity rating will also meet ASHRAE class A4. Rack ATS are natural convection cooled. 6.7 Rack Power Distribution Units A rack power distribution unit (rPDU) is a device with multiple outlets for distributing electrical power to IT equipment inside a rack and is essentially an outlet strip designed for use in IT racks with additional monitoring and capacity. Rack PDUs are also commonly called Cabinet Distribution Units (CDUs). A rack PDU is usually the last piece of equipment in the data center power distribution system and it is where the plugs of the individual pieces of IT equipment are connected to data center power. A rack PDU is usually located in the back of the IT equipment rack so it is in close proximity to the IT equipment power supplies. Rack PDUs are sometimes also called cabinet or enclosure PDUs because they are located inside the cabinet or enclosure of the rack. A photo of a typical rack PDU is shown in Figure 25. Rack PDUs can be used in either a vertical installation configuration (most common) or a horizontal installation configuration. Figure 25 Photos of vertical rack PDU and horizontal rack PDU shown installed in a rack. Some rack PDUs have “intelligence”, i.e. the ability to meter, monitor, and control the rack PDU through a network interface. Rack PDUs vary widely in the level of built-in intelligence. The most sophisticated rack PDUs enable remote monitoring of current, voltage, power, power factor, temperature, humidity and even the ability to switch individual outlets on and off remotely. What Rack PDU and Corresponding Power Cord Temperature Rating Is Needed? The key consideration for defining a rack PDU maximum temperature rating is it must be compatible with the IT equipment in the rack. However, thermal compatibility does not necessarily mean identical specification limits, i.e. a maximum temperature rating identical to the IT equipment (see
- 37. 37 Table 2 for ASHRAE thermal guideline classes). Rack PDUs are generally located in the rear of the rack where the air temperature is much higher. Thus, for rack PDUs, thermal compatibility with the IT equipment means compatibility with the exhaust air temperature of the IT equipment, not the inlet air temperature specification. As mentioned in Section 2.3, not only are IT equipment temperatures rising, they level out at a maximum of between 58 and 60°C (136 and 140°F) for high inlet temperatures of the allowable ranges of Classes A3 and A4. Given that these high exhaust temperatures could be encountered in a real data center situation, ASHRAE recommends all new rack PDU products be designed to at least 60°C (140°F). Rack power distribution units and other equipment covered by product safety standards such as IEC 60950-1 [16] are typically rated by the manufacturer based on a maximum allowable air temperature. This makes sense because it provides a design point for selecting components inside the rack PDU. Components internal to the rack PDU must stay at or below their rated maximum temperatures including the air temperature outside the rack PDU and any temperature rise from heating due to power dissipation of the components themselves or other components within the rack PDU. Manufacturers of rack PDUs often employ component de-rating. De-rating is the practice of designing components to be used at temperatures, voltages, and currents less than their maximum rating as a way of reducing failure rate of the components and improving the reliability of the rack PDU. In an economized data center, rack PDUs must be carefully selected so their overall manufacturer design rating meets or exceeds the maximum exhaust temperature of the IT equipment during the highest inlet air temperature excursion and workload conditions expected. It is worth noting that a rack PDU with a higher maximum temperature rating has additional benefits beyond making it capable of economization. In a traditionally cooled data center that uses a chiller or direct expansion units, an HVAC failure can cause data center temperatures to soar in a matter of minutes [37]. Higher maximum temperature ratings of rack PDUs and IT equipment give a data center manager more ride-through time to restart their chiller or air conditioning unit before equipment reaches its thermal trip limits and starts shutting down. Additional ride-through time also gives data center operators more time to gracefully shut down IT equipment and avoid a potential loss of data. Other important rack PDU selection considerations include life expectancy, projected failure rate, over-current protection (OCP) design limits, natural cooling versus active cooling designs, mounting configuration, outlet cable ratings, and environmental self-monitoring. State of the Industry Intelligent or smart rack PDUs have become the norm for most data centers. The need to continually monitor power usage on a per-cabinet, or even per-device basis has become a data center and facilities management best practice. The monitoring capability of rack PDUs provides the information needed to make data driven decisions. Intelligent rack PDUs contain very sophisticated electronics and require a level of design rigor and care similar to that of the IT equipment itself. Data center loads are no longer as stable and static as in the past due to virtualization, right-sizing of power supplies, and remote-power-off and other IT equipment
- 38. 38 firmware energy saving strategies. The monitoring capability of rack PDUs and other monitoring data can be integrated with data center infrastructure management (DCIM) software to better control data center temperature and more efficiently manage data center power consumption. A survey of rack PDU maximum temperature ratings from six leading rack PDU suppliers is shown below in Figure 26. The most common design points were 45 (113°F), 60 (140°F) and 65°C (149°F). Figure 26 Benchmarking of rack PDU maximum temperature ratings across multiple suppliers. Reliability and Life Expectancy of Rack PDUs Most of the failure mechanisms for electronics are thermally activated. That is, the failure rate increases at higher temperatures. A plot of server failure rate as a function of temperature is shown below in Figure 27. Failure rate is shown on the y-axis as an x-factor. The failure rate data in the graph is normalized to the failure rate at a steady state operating temperature of 20°C (68°F). IT equipment operated continuously at 30°C (86°F) would have an x-factor of 1.4 or a 40% higher failure rate than would the equipment run at 20°C (68°F), and so forth. When estimating the failure rate of a piece of IT equipment in an economized environment, where the air temperature may vary widely over time, one needs to do a time weighted average failure rate. The failure rate of the equipment is not equal to the highest air temperature excursion.
- 39. 39 Figure 27 Server reliability x-factor as a function of inlet air temperature (data from [1]). In the case of power equipment, if that equipment was made with components, materials, and manufacturing processes that are similar to the IT equipment then one would expect a similar failure rate trend with temperature. However, power equipment tends to be much simpler and have far fewer active components than IT equipment so it is unclear whether power equipment will follow the same trend as IT equipment. One would still expect a higher failure rate of power equipment at elevated temperatures but maybe not to the same degree as for the servers shown in Figure 27. Table 1 shows the typical usage time for a rack PDU as 8 to 12 years. One of the concerns with any piece of equipment is whether that equipment will last for the amount of time one intends to use it or before it is replaced due to obsolescence. The term “lifetime” refers to how long a piece of equipment will last before we can expect it to wear out and fail. In practice, one would like to see “lifetimes” much longer than the length of time the equipment is intended to be used. If this is the case, there should be no wear-out or associated high failure rate during the usage duration of the product. Most modern component manufacturing and selection processes are carefully engineered to avoid wear-out failure mechanisms which could cause high failure rates. A word of caution is in order when using the term “lifetime”. Lifetime is only meaningful when a cumulative failure criterion is defined along with it. A valid use of the term lifetime would be to say, for example, a piece of equipment has a lifetime of 15.7 years to reach 1% cumulative failures. That is, by 15.7 years 1% of the equipment would have failed. Another important term often used incorrectly as a proxy for lifetime is mean time between failures (MTBF). MTBF is the average amount of time between failures and is not a lifetime. To better illustrate why MTBF can be a misleading indicator of lifetime, consider an example where a data center manager selects a rack PDU with a 10 year MTBF. At first glance, 10 years looks like it would be satisfactory for a piece of equipment one expects to use for 8 to 12 years. However, if you assume a constant failure rate in which the failure of a piece of equipment is just as likely in any hour as it is in the next hour, only 36.8% of the rack PDUs will still be functional at 10 years. In other words, 63.2%
- 40. 40 of the rack PDUs would have already failed by 10 years. In this example, a 10 year MTBF does not ensure an acceptable failure rate during a 10 year deployment. Great care should be exercised when selecting a rack PDU using MTBF values. Calculations should be carried out to provide an estimate of the cumulative percent failing at the end of the deployment time (e.g. 12 years) and the cumulative percentage failing are within an amount the IT equipment operator finds acceptable, usually a couple of percent. Internal heat generation is a factor that needs to be accounted for in the design of a rack PDU and for defining a specification for the maximum external air temperature. As shown in Figure 28, there can be a significant amount of internal heat generation in a rack PDU. All of the wiring, connectors, contacts, and over-current protection (OCP) devices in the rack PDU are resistive in nature and will add heat to the system. Consider the OCP devices, in particular, which may be circuit breakers or fuses. Not only will they add heat to the system, but they are also sensitive to the temperature they are exposed to. Most magnetic-hydraulic type circuit breakers are rated for use at 85°C (185°F); however, some less-expensive thermal-magnetic breakers can be rated to only 40°C (104°F). Since nuisance tripping of the rack PDU internal breakers can cause costly service interruptions, it is worth investigating this point with the manufacturer when selecting rack PDUs. Figure 28 Close-up infrared photo of rack PDU showing internal heat generation. One of the thermal challenges rack PDUs have is that the location of the IT equipment power cords and the architecture of a typical data center rack makes it difficult or impossible for a rack PDU to access air from the cold aisle in front of the rack for cooling. Most rack PDUs are passively cooled and don’t have fans or air moving devices inside them. Thus, rack PDUs have to be designed for much higher temperature operation than the IT equipment that connects to them. The most common form factor for rack PDUs is known as the vertical rack PDU which often mounts to dedicated panels toward the rear of enclosed racks or on brackets extending from 2-post racks. The natural convection-cooled rack PDUs are likely to have ventilation holes that should remain free from obstruction. In some cases, horizontal rack PDUs are mounted within the rack unit (RU) space in the back of the rack. A blanking panel is often used in front of the rack PDU to
- 41. 41 prevent leakage of air through the rack. As with the vertical rack PDU, this creates a situation where the horizontal PDU doesn’t receive any cooling air from the front of the rack and must be designed to operate in the high temperature environment of the IT equipment exhaust (see Figure 29). For horizontal rack PDUs, there are some server rack fans that can be installed in place of a blanking panel in the front of the rack that will direct some cooling air to the horizontal rack PDU. While these fan units do consume a small amount of energy, they can significantly reduce the air temperature around a horizontal rack PDU and provide a modest improvement in reliability. Server rack fans are not recommended for vertically installed rack PDUs. A single set of rack fans will not provide cooling air along the height of the PDU nor will they reduce IT equipment exhaust temperatures. Figure 29 Side view of rack showing heat build-up around installation of a horizontal rack PDU. Along with the considerations for localized temperatures around the rack PDU, the power cabling from the outputs of that rack PDU to the servers and other devices needs to be investigated. Be sure that the C13 and C19 couplers and associated cables are properly sized to allow for full current carrying capability over the entire temperature range. C13 and C19 refer to standard types of couplers defined in IEC 60320 [14]. Also consider that tightly bound bundles of cabling will cause localized heating within that bundle, which could exceed the ratings of the cables. Environmental Monitoring Many intelligent rack PDUs with network connectivity have the capability to provide temperature and humidity monitoring inside the rack. These rack PDUs typically have multiple probes that can be positioned independently. With network connectivity, this data can be reported in real-time and used for: Mapping the distribution of air temperature across the data center air to better optimize cooling Making sure air temperatures are within the design specifications of the equipment Identifying localized hot spots that may need additional cooling Providing alert capability for temperature excursions that could cause a thermal shutdown of IT equipment and compromise data center uptime
- 42. 42 Improving overall data center energy efficiency 6.8 Recommendations for Power Distribution Equipment The purpose of Section 6 is to review the functionality of power distribution equipment and make recommendations to foster better thermal compatibility with IT equipment. In an ideal world, the end user shouldn’t have to do any re-engineering and, as long as the user follows installation best practices, the power distribution equipment should have “plug and play” thermal compatibility with the IT equipment. The power distribution equipment would have temperature and humidity specifications that meet or exceed the common IT environmental specifications and it would be obvious how the two sets of equipment environmental specifications align. A summary of typical temperature and humidity ratings for different types of power distribution equipment is shown below in Table 6. The IEC 61439 standard and several ASHRAE classes are included for comparison. Table 6 Typical environmental ratings for power distribution equipment at maximum rated power along with aisle location. Equipment Type or Industry Standard Operational Temperature Range* Operational Humidity Range Location Floor Mounted Power Distribution Unit (PDU) 0 to 40°C (32 to 104°F) 0 to 95%RH or 10 to 95%RH Ballroom, end of aisle, or cold aisle Remote Power Panel (RPP) 0 to 40°C (32 to 104°F) 0 to 95%RH or 10 to 95%RH Ballroom, end of aisle, cold aisle, or separate grey area Panelboards Determined by circuit breakers and conductors inside Non-condensing Ballroom or separate grey area Busways 40°C (104°F) with no more than 55°C (131°F) hot spot rise above ambient 0% to 95% Non-condensing Ballroom, cold aisle, hot aisle, underfloor, or separate grey area Rack ATS Varies: 45°C (113°F), 50°C (122°F) (CE), 60°C (140°F) (UL) Varies: 85 or 95% RH maximum Cold or hot aisle Rack PDU Varies: 45°C (113°F), 60°C (140°F), 65°C (149°F) 5 to 95%RH typical Hot aisle IEC 61439 Standard -5 to 35°C (23 to 95°F), 40°C (104°F) for 2 hours 50% maximum at 40°C (104°F) (27.6°C, 81.7°F dew point), higher relative humidity is allowed N/A
- 43. 43 at lower temperatures. ASHRAE Class A2 10 to 35°C -12˚C DP & 8% RH to27˚C DP and 85% RH Cold aisle and ballroom ASHRAE Class A3 5 to 40°C -12˚C DP & 8% RH to27˚C DP and 85% RH Cold aisle and ballroom ASHRAE Class A4 5 to 45°C -12˚C DP & 8% RH to27˚C DP and 90% RH Cold aisle and ballroom *Note: the maximum ambient air temperature rating of most power equipment is flexible. Higher ambient air temperature ratings are possible if the maximum power rating is reduced. In the context of this discussion, thermal compatibility is defined as the power equipment specifications meeting or exceeding those of a given ASHRAE class plus appropriate preheating based on location. A summary of how power distribution specifications align to the ASHRAE thermal guidelines for IT equipment is given below in Table 7. Table 7 Cold aisle and hot aisle compatibility of power distribution equipment. Equipment Type or Standard Cold Aisle Thermal Compatibility at Maximum Power Rating Hot Aisle Thermal Compatibility at Maximum Power Rating Floor Mounted Power Distribution Unit (PDU) Compatible with ASHRAE Class A3 Compatible only with power de-rating Remote Power Panel (RPP) Compatible with ASHRAE Class A3 Compatible only with power de-rating Panelboards N/A Busways Compatible with ASHRAE Class A3 Compatible only with power de-rating Rack ATS Compatible with ASHRAE Class A3/A4 Depends on Model Rack PDU Compatible with ASHRAE Class A4 Depends on Model IEC 61439 Standard [38] Compatible with ASHRAE Class A2 Not Compatible Nearly all power equipment is cold aisle compatible up to ASHRAE Class A3 at maximum rated power. Most power equipment is hot aisle compatible but only with de-rating (reduction) of maximum power to limit conductor temperature rise and stay within maximum conductor temperature limits. Rack PDU and ATS units are all cold aisle compatible and many models, rated at >60°C (140°F), are also hot aisle compatible. Unlike other types of power equipment, rack PDUs, rack ATS units, and some line or power cords have a fixed maximum ambient air temperature rating and power de-rating flexibility is not provided in their specifications. The maximum ambient air temperature of rack PDUs is primarily limited by touch temperature. The receptacles have a 70°C (158°F) touch temperature limit per IEC60320-1 [14] and most metal
- 44. 44 surfaces also have a 70°C (158°F) touch temperature limit per IEC60950-1 [16]. If 10°C (18°F) is allowed for conductor temperature rise, the maximum ambient air temperature rating needs to be 60°C (140°F) which is the most common rating for rack PDUs. Higher maximum air temperature ratings for rack PDUs may be needed in the future if the trend of rising IT equipment exhaust temperatures continues. However, equipment designers will need to find a way of overcoming touch temperature limitations. The IEC 61439 family of standards “Low-Voltage Switchgear and Control Gear Assemblies” [38] listed in Table 7, is a commonly accepted international standard for power and power distribution equipment. IEC 61439 covers low voltage switch gear, power switchgear, control gear assemblies, distribution boards, and busways. IEC 61439 aligns to ASHRAE Class A2 (35°C, 95°F)). However, the most common design point for power equipment is 40°C (104°F) which corresponds to ASHRAE Class A3. ASHRAE TC9.9 recommends designing to IEC 61439, at a minimum, and to the temperature requirements of Class A3, where possible. IEC 61439 and Class A3 differ in their humidity requirements: IEC 61439 calls out a requirement of 50%RH at 40°C (104°F) which is a 27.6°C (81.7°F) maximum dew point versus only a 24°C (75°F) maximum dew point for ASHRAE class A3 and A4. However, little or no design work or component selection is typically done on the basis of meeting a humidity requirement. Any product that meets the IEC humidity requirements should also meet or exceed the ASHRAE A3 and A4 humidity requirements. A summary of the ASHRAE TC9.9 recommendations for rack power distribution equipment is as follows: Rack PDUs should have a maximum ambient air temperature rating of at least 60°C (140°F). Openings in the raised floor for power cables should be sealed to prevent the escape of cold air from the underfloor air volume and a loss of data center cooling efficiency. The maximum power rating of power distribution equipment should be sized to account for variations and excursions in air temperature. High air temperature excursions can occur for a number of reasons such as: hot weather where an air-side economized data center takes its cooling air from outdoors, an HVAC failure that causes temperatures to rise quickly in the data center, or a localized data center hot spot when cooling is less than ideal. Locate power equipment in ballroom, cold aisles, or grey areas where possible. Higher hot aisle temperatures necessitate de-rating of maximum power whereas cold aisle temperatures typically don’t require de-rating. Hot aisle temperatures have risen over time and may continue to rise. Ideally, the power distribution infrastructure should not be a weak link that causes data center equipment to shut down. The power equipment should allow the IT equipment to run as long as possible to avoid a service outage or data loss and give the equipment time to save data and perform a graceful shutdown, if needed. 7. Higher Voltage DC (HVDC) Power Both AC and HVDC voltage can be dangerous depending on the amperage flowing in a circuit. In general, there are two electrical hazards with either AC or HVDC: shock and arc flash. However, since AC voltage is sinusoidal, continuously changing direction and HVDC voltage is
- 45. 45 constant, there are a different set of calculations used to quantify the hazards. One important difference is that HVDC power flow must be controlled by a circuit breaker when applying or removing power from IT equipment. The absence of zero crossings in HVDC current flow means no intrinsic arc suppression. Other Electrical Power Distribution Schemes Used in Data Centers The typical power distribution infrastructure transforms the utility voltage electricity supply to a value within the range of 200-240 VAC depending on geographical location. For example, North America is either 208 VAC or 240 VAC whereas Europe is 230 VAC. In recent years, the data center electrical delivery path efficiency has been studied and new schemes are emerging which can reduce capital expenditure (Capex) and operational expenditure (Opex). Figure 30 Most Common US Data Center Electrical Distribution Topologies Figure 30 shows that the HVDC (380 VDC) and 277 VAC use a higher supply voltage compared to the 200-240 VAC traditional data center topology. For a product that is at constant power, increasing the electricity supply voltage means decreasing the amperage. Amperage is inversely proportional to voltage per Ohm’s Law equation P=VI. In general, less amperage equates to smaller cross-sectional area (larger gauge) copper cables to deliver the power. Even though larger gauge cables have slightly higher impedance, the heating losses (I2 R) associated with the copper cables are dominated by the amperage. Higher Voltage DC (HVDC) DC power has been around for a long time. Telecom central offices (COs) use -48 VDC for a couple reasons: 1) compatibility with valve-regulated lead-acid (VRLA) battery strings and 2) minimize corrosion by keeping the positive terminal at ground potential. The general principle behind high or higher voltage direct current (HVDC) is to eliminate or improve the efficiency of voltage conversions in the data center. The HVDC topology in Figure 30 replaces the traditional voltage step down transformer with a more efficient AC-to-DC rectifier. 380 VDC is a natural selection for IT equipment operation since an AC power supply goes through several stages before arriving at the output 380 VDC from the boost stage. See Figure 31. Topology Traditional HVDC 277 Vac StepDown Transformer StepDown Transformer StepDown Transformer StepDown Transformer Rectifier 200-240 VAC 277 VAC380 VDC
- 46. 46 Figure 31 IT Equipment Power Supply Train In theory, the electromagnetic interference (EMI) filter, AC-to-DC rectifier, and DC boost stage can be eliminated from a traditional AC power supply. Not only are there less components inside the IT equipment DC power supply when using HVDC, but also less components in the data center power train. Figure 32 compares the AC power train in a typical data center with double conversion UPS against a DC distribution scheme. Figure 32 AC and HVDC Power Trains Compatibility with photovoltaic panel and wind turbine energy generation is also another driving force behind HVDC. Renewable energy is generated at DC voltage levels and used in smaller scale DC micro-grids that can be easily to implement. 208/120 V UPS AC/DC DC/AC PDU Bypass 480/277 V13.8 kV Battery cabinet(s) Chrg Isolation TransformerIsolation Transformer 480/277 V13.8 kV Isolation Transformer 380 Vdc Battery cabinet(s) Rectifier PDU AC HVDC
- 47. 47 277 VAC The advantage of using 277 VAC rather than 200-240 VAC in data centers is the Capex saving from eliminating the final step down transformer (see Figure 30). Not only is there a savings in hardware, but there is also a savings in copper wiring and maintenance. There is also an Opex savings in the form of less heat released. A transformer with a nameplate capacity of 100 kW that is 96% efficient at rated capacity has a loss of 4 kW that would require an equivalent amount of cooling. New 2016 DOE efficiency standards for transformers [39] are shown below in Table 8 compared to the former 2002 NEMA TP-1 standard [40]. Improvements in transformer efficiency, especially in three-phase transformers, should necessitate less cooling. Note: the efficiency requirements for single phase transformers have not changed except that that new 2016 DOE standard requires they be tested and measured to two decimal places instead of one. Table 8 New 2016 DOE standard transformer efficiencies compared to former 2002 NEMA TP- 1 efficiency levels. Supplemental Two of the emerging electrical power schemes used in data centers distribute voltages higher than the traditional 200-240 VAC. An operating voltage higher than 250V requires design changes to IT power supplies, line cords, and rack PDUs. The majority of 19” rack-mount IT equipment in the marketplace uses the IEC 60320 standard appliance inlets on power supplies. Above 250V, these couplers can no longer be used because of voltage spacing requirements and electric shock protection. Moving to a different set of couplers on the power supplies affects the power jumper cords between rack-mount drawer and PDU as well as the PDU receptacles. The thermal compatibility of DC power equipment isn’t much different than AC power equipment as their placement and usage in the data center is the same. Reducing voltage conversions will reduce the heat rejected into the environment, but ever increasing AC conversion efficiencies of UPSs and IT equipment power supplies at the 200-240 VAC level are also reducing heat output. 2002 NEMA TP-1 2016 DOE Standard 2002 NEMA TP-1 2016 DOE Standard 15 97.7% 97.70% 15 97.0% 97.89% 25 98.0% 98.00% 30 97.5% 98.23% 37.5 98.2% 98.20% 45 97.7% 98.40% 50 98.3% 98.30% 75 98.0% 98.60% 75 98.5% 98.50% 112.5 98.2% 98.74% 100 98.6% 98.60% 150 98.3% 98.83% 167 98.7% 98.70% 225 98.5% 98.94% 250 98.8% 98.80% 300 98.6% 99.02% 333 98.9% 98.90% 500 98.7% 99.14% 750 98.8% 99.23% 1000 98.9% 99.28% Single-Phase Three-Phase Efficiency %Efficiency % KVA KVA
- 48. 48 Time will tell the amount of HVDC or 277 VAC market penetrations given the legacy and rich history of 200-240 VAC as the electricity supply voltage. A more complete discussion of HVDC and 277 VAC is beyond the scope of this paper. 8. ASHRAE TC9.9 Recommendations The ASHRAE TC9.9 committee has the following recommendations for power equipment (including line cords, power cords, and receptacles) that share the same physical space and air streams in the data center as the IT equipment: 1. Design power equipment to meet the temperature and humidity requirements of IEC61439 at a minimum and preferably to the 40°C (104°F) temperature requirement of ASHRAE Class A3. 40°C (104°F) is already a de-facto industry standard design point for most power equipment. This recommendation applies to power equipment residing in the ballroom, cold aisle, and grey areas of the data center. 2. Ensure power equipment environmental specifications are compatible with IT equipment specifications where the two types of equipment share the same physical space and air stream. Power equipment temperature specifications are typically based on conductor temperature whereas IT equipment specifications are based on air temperature. The term compatibility does not imply the two sets of environmental specifications are identical, only that they are similar enough the two types of equipment can coexist in the same space and air stream. 3. Design rack PDUs for a maximum ambient air temperature rating of at least 60°C (140°F). This is needed to align with rising IT equipment exhaust temperatures in the hot aisle. If IT equipment exhaust temperatures continue to increase in the future, ratings above 60°C (140°F) may be needed. 4. Size the maximum power rating of power distribution equipment to account for variations and excursions in air temperature. Warm air temperatures, above the typical temperature for a data center, can occur at hot spots where the local cooling is less than ideal. High air temperature excursions can occur for a number of reasons such as an HVAC failure that causes temperatures to rise quickly in the data center or where an air-side or water-side economized data center takes its cooling air from outdoors and there is a rapid change in the weather such as a change from cool night time conditions to hot afternoon conditions on a summer day. 5. Locate power equipment in ballroom, cold aisle areas, or grey areas where possible to minimize ampacity de-rating for temperature. Avoid locating power equipment in hot aisles. The temperature of the ambient air in a hot aisle is not directly controlled, rather it is a byproduct of the IT equipment and associated work load on the IT equipment. Higher hot aisle temperatures necessitate de-rating of maximum power whereas ballroom and cold aisle temperatures typically don’t require de-rating. Hot aisle temperatures have risen over time and may continue to rise. Be sure to follow individual equipment manufacturers’ recommendations on temperature and equipment placement.
- 49. 49 6. Make sure battery replacement intervals for distributed UPS units are determined from the manufacturer’s guidelines and are based on the actual battery temperature and usage not just data center air temperature. UPS equipment manufacturers should provide guidelines on how to determine battery replacement interval based on battery temperature and usage, such as the number of charge and discharge cycles. Other factors can influence battery temperature such as charge and discharge rate, heating of the batteries from adjacent IT equipment, and data center air temperature. Replacement intervals should be determined so that a UPS unit will always be able to provide its rated back-up power for the intended duration in the event of a power outage. 7. Power equipment using super-capacitors should be carefully designed using the component manufacturer’s temperature and voltage de-rating guidelines and the temperature of the super-capacitor component itself not the ambient air temperature. Power equipment manufacturers should use the super-capacitor supplier provided de-rating data and make sure the de-rating assumptions used support the intended lifetime and the published temperature and humidity operating specifications of the equipment. De-rating calculations should be done using the super-capacitor component temperature and not the temperature of the surrounding air outside of the equipment cabinet. Most super-capacitors have a maximum non-operational storage temperature of 45°C (113°F) and some even as low as 35°C (95°F). Temperature conditions during shipping and storage, especially in summer, can compromise the lifetime of super-capacitors. ASHRAE recommends including a temperature/humidity monitor during shipping and storage to detect any conditions that could adversely affect the super-capacitors. IT spaces and grey areas where super-capacitor based power equipment are used should have temperature and humidity control that is well within the equipment specification limits. 8. Design small battery back-up (BBU) units that reside inside IT equipment in place of a power supply to the same thermal conditions as a power supply. A 50°C (122°F) air temperature should be the minimum design point and this temperature should be measured at the same location as the inlet air of the power supply unit (PSU) the BBU is replacing. The battery replacement interval should be determined from data provided by the battery supplier and the temperature of the battery, not the temperature of the surrounding air. 9. Cords from the IT equipment to the rack PDU should be properly sized to allow for full current carrying capability over the ambient air temperature range around the cord for the power equipment the cord is connected to. A 60°C (140°F) ambient air temperature is a recommended design point for IT equipment power cords and rack PDU power cords that reside in the back of an IT equipment rack. 10. For both equipment design and installation, pay close attention to touch temperature limit guidelines such as IEC60950-1 and others. This recommendation applies to both power equipment designers and data center operators. Rising data center temperatures, especially IT equipment exhaust temperatures, are pushing many surfaces closer to touch temperature specification limits. If specification touch temperature limits are exceeded, a
- 50. 50 different material (e.g. plastic instead of metal) may need to be used in the equipment construction. 11. Power equipment that is part of a row of racks containing IT equipment (such as floor mounted PDUs) should be compatible with a hot aisle cold aisle air flow configuration. If the power equipment requires a cooling air flow, it should take air flow from the cold aisle and exhaust the air to the hot aisle in a manner similar to the IT equipment [41]. Having power equipment in the same row as IT equipment is more common in colocation data centers. Front to back air flow configuration is important for implementing energy efficiency strategies such as aisle containment. 12. Ensure power equipment is installed in such a way as to prevent condensation. Precise humidity control is not critical for most power equipment, however, prevention of condensation is. Simple heaters, such as panel heaters, should be installed where needed to keep conductors slightly warmer than the surrounding air and prevent condensation from forming. 13. Most VRLA batteries may not be suitable for IT equipment spaces where the data center is economizing over a temperature range wider than 18 to 27°C (64°F to 81°F). The lifetime of most lead-acid based VRLA batteries is very temperature dependent and operating these batteries at temperatures much above the ideal temperature of 25°C (77°F) may not practical or cost effective. 14. Seal openings for power cables in the raised floor to prevent escape of cold air from the plenum and a loss of data center cooling efficiency. One solution is to use commercially available grommet and brush kits. A supply of cooling air from the plenum under the raised floor is usually not necessary for power equipment as most power equipment is natural convection cooled. This recommendation applies primarily to power equipment located in ballroom areas. 15. Monitor data center air temperature at multiple locations including locations with power equipment and line cords. Air temperatures can vary widely across a data center even within the length of a meter. Real-time monitoring and reporting of air temperature will prevent accidental equipment overheating and shutdown and it is a tool the data center operator can use to optimize cooling.
- 51. 51 9. Summary Most power equipment is rated by the temperature of the electrical conductors and not the surrounding air. Rack PDUs are a notable exception, though the maximum air temp is calculated based on conductor temperature rise. This flexible rating system allows power equipment to be used over a wide range of temperature, provided the ampacity is de-rated per the manufacturer’s guidelines. Compatibility of thermal specifications is needed, where power equipment shares the same space or air streams with IT equipment. Compatibility does not necessarily mean thermal specifications (power vs. IT equipment) are defined the same way. It does mean the power equipment meets or exceeds the thermal specifications of the IT equipment including any conductor temperature rise. ASHRAE recommends power equipment be designed for the temperature and humidity requirements of IEC61439 at a minimum and preferably to the 40°C (104°F) upper air temperature limit of ASHRAE class A3. This recommendation is important because data center environmental conditions are slowly changing to save money on cooling costs. Many data centers now operate at slightly warmer air temperatures and some are even economized using outside air. A set of recommendations has been proposed for the major types of power equipment to improve their thermal compatibility with IT equipment and their reliability. 10. References [1] ASHRAE TC9.9, Thermal Guidelines for Data Processing Environments, Fourth Edition, ASHRAE, Atlanta Georgia, 2015. [2] P. Hu, "Electrical Distribution Equipment in Data Center Environments, White Paper 61, Revision 1," 2015. [Online]. Available: http://www.apcmedia.com/salestools/VAVR-8W4MEX/VAVR- 8W4MEX_R1_EN.pdf?sdirect=true. [Accessed 9 July 2015]. [3] D. Loucks, "Power Equipment and Data Center Design, The Green Grid White Paper #51," 2012. [Online]. Available: http://www.thegreengrid.org/~/media/WhitePapers/WP51- PowerEquipmentandDataCenterDesign_v1.pdf?lang=en. [Accessed 9 July 2015]. [4] National Fire Protection Association, NFPA 70 National Electrical Code, Quincy, MA: National Fire Protection Association, 2014. [5] Emerson Network Power, " Five Strategies for Cutting Data CenterEnergy Costs Through Enhanced Cooling Efficiency, WP151-47," 2007. [Online]. Available: http://www.emersonnetworkpower.com/documentation/en- us/brands/liebert/documents/white%20papers/data-center-energy-efficiency_151- 47.pdf. [Accessed 10 June 2015]. [6] P. Thibodeau, "It's getting warmer in some data centers," 15 July 2013. [Online]. Available: http://www.computerworld.com/article/2483971/data-center/it-s-getting- warmer-in-some-data-centers.html. [Accessed 9 July 2015]. [7] J. Kaiser, "Survey Results: Data Center Economizer Use, White Paper #41," 2011. [Online]. Available: http://www.thegreengrid.org/sitecore/content/Global/Content/white-papers/WP41- SurveyResultsDataCenterEconomizerUse.aspx. [Accessed 10 June 2015].
- 52. 52 [8] Air Enterprises, "Thermowheel - the Energy Recovery Leader," [Online]. Available: http://airenterprises.com/products/energy-recovery-thermowheel/. [Accessed 10 June 2015]. [9] ASHRAE, Particulate and Gaseous Contamination in Datacom Environments, Atlanta, Georgia: American Society of Heating, Refrigerating, and Air-Conditioning Engineers, 2014. [10] R. Miller, "APC EcoBreeze Offers Free Cooling Options," 14 December 2010. [Online]. Available: http://www.datacenterknowledge.com/archives/2010/12/14/apc- ecobreeze-offers-free-cooling-options/. [Accessed 20 August 2015]. [11] LIebert, "Liebert® DSE™ with EconoPhase™: Highest Efficiency DX Cooling with pumped refrigerant economizer, Product Brief," 2015. [Online]. Available: http://www.emersonnetworkpower.com/documentation/en- us/brands/liebert/documents/white%20papers/liebert_dse_product_brief-sl-18930.pdf. [Accessed 20 August 2015]. [12] International Electrotechnical Commission (IEC), IEC 60309-1 Plugs, socket-outlets and couplers for industrial purposes – Part 1 General Requirements, Geneva Switzerland: IEC, 2005. [13] International Electrotechnical Commission (IEC), IEC 60309-2 lugs, socket-outlets and couplers for industrial purposes - Part 2: Dimensional interchangeability requirements for pin and contact-tube accessories, Geneva Switzerland: IEC, 2005. [14] International Electrotechnical Commission (IEC), IEC 60320-1:2015 Appliance couplers for household and similar general purposes - Part 1: General requirements, International Electrotechnical Commission (IEC), 2015. [15] American Society of Heating Refrigerating and Air-Conditioning Engineers (ASHRAE), IT Equipment Design Impact on Data Center Solutions, Atlanta Georgia: ASHRAE, 2016. [16] International Electrotechnical Commission (IEC), IEC 60950-1 Information technology equipment – Safety, Part 1: General Requirements, 2nd Edition, Geneva, Switzerland: IEC, 2005. [17] IEC 61439 Low-voltage switchgear and controlgear assemblies, International Electrotechnical Commission (IEC), 2011. [18] ISO 12944: Paints and varnishes -- Corrosion protection of steel structures by protective paint systems, International Organization for Standardization, 2007. [19] N. Rasmussen, "The Different Types of UPS Systems, APC Schneider Electric White Paper #1, Revision 7," 2011. [Online]. Available: http://www.apcmedia.com/salestools/SADE-5TNM3Y/SADE- 5TNM3Y_R7_EN.pdf?sdirect=true. [Accessed 9 July 2015]. [20] K. McCarthy, "Comparing UPS System Design Configurations, APC Schneider Electric White Paper #75, Revision 3," [Online]. Available: http://www.apcmedia.com/salestools/SADE-5TPL8X/SADE- 5TPL8X_R3_EN.pdf?sdirect=true. [Accessed 9 July 2015]. [21] S. McCluer, "Comparing Data Center Batteries, Flywheels and Ultracapacitors, APC White Paper #65," 2009. [Online]. Available: http://www.apcdistributors.com/white-
- 53. 53 papers/Power/WP- 65%20Comparing%20Data%20Center%20Batteries,%20Flywheels,%20and%20Ultrac apacitors.pdf. [Accessed 10 June 2015]. [22] Underwriter's Laboratories, Standard for Lithium Batteries UL 1642, Underwriter's Laboratories, September, 2012. [23] M. Azarian, "Overview of Electrical Double Layer Capacitors," University of Maryland, Center for Advanced Life Cycle Engineering, College Park, Maryland, 2013. [24] Nichicon UM Series EDLC Capacitor Specification. [25] Nichicon, Application Guidelines For Electric Double Layer Capacitors "EVerCAP" CAT.8100A. [26] S. Mcluer, "Battery Technology for Data Centers and Network Rooms: Ventilation, APC Schneider White Paper #30," 2003. [Online]. Available: http://www.apcdistributors.com/white-papers/Power/WP- 34%20Battery%20Technology%20for%20Data%20Centers%20and%20Network%20r ooms%20Ventilation.pdf. [Accessed 10 June 2015]. [27] Eaton, "The Large UPS Battery Handbook," 2012. [Online]. Available: http://www.capitolpower.com/pdf/Batteryhandbk.pdf. [Accessed July 2015]. [28] IEEE, IEEE 450-2010, IEEE Recommended Practice for Maintenance, Testing, and Replacement of Vented Lead-Acid Batteries for Stationary Applications, New York: IEEE, 2011. [29] Battery University, "How to Prolong Lithium Based Batteries," 20 January 2016. [Online]. Available: http://batteryuniversity.com/learn/article/how_to_prolong_lithium_based_batteries. [Accessed 9 February 2016]. [30] F. Leng, "Effect of Temperature on the Aging rate of Li Ion Battery Operating above Room Temperature," Scientific Reports, vol. 5, no. Article number: 12967, 2015. [31] T. Reddy, Linden's Handbook of Batteries, New York: McGraw-Hill, 2011. [32] ASHRAE TC9.9, "Data Center Storage Equipment - Thermal Guidelines, Issues and Best Practices," 2014. [Online]. Available: http://tc99.ashraetcs.org/documents/ASHRAE_Storage_White_Paper_2015.pdf. [Accessed 6 July 2015]. [33] V. Avelar, "The Green Grid White Paper #49 PUE™: A Comprehensive Examination of the Metric," The Green Grid, 2012. [34] K.-D. Lange, Server Efficiency--Metrics For Computer Serverse And Storage, Atlanta Georgia: ASHRAE, 2015. [35] N. Rasmussen, "The Role of Isolation Transformers in Data Center UPS Systems, APC Schneider Electric White Paper #98, Revision 0," 2011. [Online]. Available: http://www.apcmedia.com/salestools/NRAN-7NB2FG/NRAN- 7NB2FG_R0_EN.pdf?sdirect=true. [Accessed 10 July 2015]. [36] R. Tajali, "LOW VOLTAGE CIRCUIT BREAKER GUIDELINES FOR DATA CENTERS, Rev 2," 13 May 2002. [Online]. Available: http://www.powerlogic.com/literature/DataCenterCBApplications_05132002.pdf. [Accessed 3 May 2016].
- 54. 54 [37] Eaton, "DATA CENTER – UNIQUE CHARACTERISTICS AND THE ROLE OF MEDIUM VOLTAGE VACUUM CIRCUIT BREAKERS," [Online]. Available: https://powerquality.eaton.com/About-Us/News-Events/whitepapers/Data-center- characteristics.asp?id=&key=&Quest_user_id=&leadg_Q_QRequired=&site=&menu= &cx=3&x=18&y=7. [Accessed 3 May 2016]. [38] Raritan, "Data Center Power Overload Protection:," 2009. [Online]. Available: http://www.gocsc.com/UserFiles/File/Raritan/TA2011/DCpoweroverloadprotection.pdf . [Accessed 3 May 2016]. [39] J. Anderson and M. Jansma, "Circuit Breakers in Data Centers: The Hidden Danger," [Online]. Available: http://www.on365.co.uk/files//The%20Ugly%20Truth%20About%20Circuit%20Break ers%20In%20Data%20Centers.pdf. [Accessed 3 May 2016]. [40] V. Avelar, "Powering Single Corded Equipment in a Dual Corded Environment, APC Schneider White Paper #62," [Online]. Available: http://www.apcmedia.com/salestools/SADE-5TNRLE/SADE- 5TNRLE_R1_EN.pdf?sdirect=true. [Accessed 14 July 2015]. [41] P. Lin, "Data Center Temperature Rise During a Cooling System Outage, APC White Paper 179," 2014. [Online]. Available: http://www.apcmedia.com/salestools/DBOY- 7CDJNW/DBOY-7CDJNW_R1_EN.pdf. [Accessed 27 August 2015]. [42] International Electrotechnical Commission, IEC 61439 Low-Voltage Switchgear and Control Gear Assemblies, Geneva Switzerland: IEC, 2011. [43] United States Department of Energy (DOE), Code of Federal Regulations at 10 CFR 431.196, Washington DC: United States Department of Energy (DOE), 2016. [44] National Electrical Manufacturers Association (NEMA), NEMA TP-1 Guide for Determining Energy Efficiency for Distribution Transformers, Rosslyn Virginia: NEMA, 2002. [45] P. Lin, "Implementing Hot and Cold Air Containment in Data Centers, APC White Paper #153," 2013. [Online]. Available: http://www.apcmedia.com/salestools/VAVR- 8K6P9G/VAVR-8K6P9G_R0_EN.pdf?sdirect=true. [Accessed 10 June 2015].
- 55. 55 APPENDIX A – Definitions Acronym/Term Definition Adiabatic Cooling See definition for Evaporative Cooling Air-Side Economization These are systems that may use direct outdoor air blown into the data center with hot air extracted and discharged back outdoors, or they may use an air-to-air heat exchanger. With the air to-air heat exchanger, cooler outdoor air is used to partially or fully cool the interior data center air. Air side systems may be enhanced with either direct or indirect evaporative cooling, extending their operating range. Ampacity Ampacity is defined as the maximum amount of electric current a conductor or device can carry before sustaining immediate or progressive deterioration. Arc Flash A release of thermal energy from an electric arc by the vaporization and ionization of materials reaching temperatures up to 19,400°C (35,000°F). Battery Backup Unit (BBU) A battery backup unit provides power to a system when the primary source of power is unavailable. Bus Bar In electrical power distribution, a bus bar is a metallic strip or bar (typically copper, brass or aluminum) that conducts electricity within a switchboard, distribution board, substation, battery bank, or other electrical apparatus. Its main purpose is to conduct a substantial current of electricity, and not to function as a structural member. Bus Way A prefabricated electrical distribution system consisting of bus bars in a protective enclosure, including straight lengths, fittings, devices, and accessories. The busway includes bus bars, an insulating and/or support material, and a housing. Cabinet Distribution Unit (CDU) See definition for Rack PDU Chiller A chiller is a machine that removes heat from a liquid via a vapor-compression or absorption refrigeration cycle. This liquid can then be circulated through a heat exchanger to cool air or equipment as required. Circuit Breaker “A device designed to open and close a circuit by non- automatic means and to open the circuit automatically on a predetermined overcurrent without damage to itself when properly applied within its rating.” Data Center Infrastructure Management (DCIM) DCIM is the management of all physical resources found in the facilities and IT domains of a data center. Many suppliers are developing software tools to automate DCIM functions. DIMM Dual in-line memory module Direct Expansion (DX) Direct expansion refers to traditional refrigerant based mechanical air conditioning systems.
- 56. 56 Economization The use of outside air for data center cooling whereby the compressor is bypassed from the cooling circuit. This can be accomplished by several different means: a) by bringing air directly into the data center (air-side economization), b) indirectly cooling data center air with an air-to-air heat exchanger (air-side economization), or c) by using outside air to cool water that is used to cool the air inside the data center (water-side economization). Electromagnetic Interference (EMI) Electromagnetic interference (EMI) is a disturbance generated by an external source that affects an electrical circuit by electromagnetic induction, electrostatic coupling, or conduction. The disturbance may degrade the performance of the circuit or even stop it from functioning. Evaporative Cooling Evaporative cooling is the addition of water vapor into air, which causes a lowering of the temperature of the air. The energy needed to evaporate the water is taken from the air in the form of sensible heat, which affects the temperature of the air, and converted into latent heat, the energy present in the water vapor component of the air, whilst the air remains at a constant enthalpy value. Evaporative cooling is also called adiabatic cooling. Flywheel A rotating mechanical device that is used to store rotational energy. In the context of a data center, the rotational energy is converted into electrical energy to provide short-term power backup for critical data center equipment. Free Cooling Free cooling is an economical method of using low external air temperatures to assist in chilling water, which can then be used for cooling of air in a data center. The chilled water can either be used immediately or be stored for the short or long- term. When outdoor temperatures are lower relative to indoor temperatures, this system utilizes the cool outdoor air as a free cooling source. In this manner, the system replaces the chiller in traditional air conditioning systems while achieving the same cooling result. In general, there are two types of cooling tower circuits: open and closed. The performance of open circuit towers is dependent on the wet bulb temperature, while the performance of closed-circuit towers (such as dry coolers) is dependent on the dry bulb temperature. Fuse A sacrificial device intended to provide overcurrent protection, of either the load or source circuit. See also OCP (overcurrent protective device). Gel Cell A gel cell is a type of VRLA battery (valve-regulated lead- acid battery) more commonly known as a sealed battery or maintenance free battery. Gel cells add silica dust to the electrolyte, forming a thick putty-like gel. These are sometimes referred to as "silicone batteries".
- 57. 57 Generator A device that converts mechanical energy to electrical energy for use in an external circuit. High Voltage (HV) High voltage is voltages above 100KV HVAC Heating, ventilating, and air conditioning Independent Distribution Facility (IDF) An independent distribution facility is typically used in large buildings and it is a place (usually a closet) where the network drops on each floor connect to the networking equipment that connects to the data center. IEC International Electrotechnical Commission Intelligent Power Distribution Unit (iPDU) See definition for rack PDU. Same as rack PDU except with intelligence built-in. ISO The International Organization for Standardization (ISO) is an international standard-setting body composed of representatives from various national standards organizations IT Information technology ITE Information technology equipment LAN A local area network (LAN) is a computer network that interconnects computers within a limited area such as a home, school, computer laboratory, or office building, using network media LFP Lithium iron phosphate battery Li-Ion Lithium ion battery Low Voltage (LV) Low voltage is voltages less than 1KV Main Distribution Facility (MDF) The MDF is where your data center is and the core of your network Main Distribution Unit (MDU) Another term for power distribution unit. See definition for Power Distribution Unit (PDU). Medium Voltage (MV) Medium voltage is voltage in the range of 600 to 1000V Modular Data Center A data center constructed from one or more modular steel containers. These are sometimes also called containerized data centers. MTBF Mean time between failures NAS Sodium sulfur chemistry battery NEMA National Electrical Manufacturers Associations Networking Equipment Networking equipment is equipment that facilitates the interconnection of devices and the sharing of data both within the data center and beyond. Network Operations Center (NOC) A network operations center (NOC), also known as a "network management center", is one or more locations from which network monitoring and control, or network management, is
- 58. 58 exercised over a computer, telecommunication or satellite network. NiCD Nickel cadmium battery NiMH Nickel metal hydride battery OCP Over Current Protection. A device capable of providing protection for service, feeder, and branch circuits and equipment over the full range of over-currents between its rated current and its interrupting rating. Such devices are provided with interrupting ratings appropriate for the intended use but no less than 5000 amperes.” Power Busway A raceway consisting of a grounded metal enclosure containing factory-mounted, bare or insulated conductors, which are usually copper or aluminum bars, rods, or tubes. Power Meter A power meter is an instrument for measuring the electric power (or the supply rate of electrical energy) in Watts of a given circuit. Power Supply Unit (PSU) A power supply is an electronic device that supplies electric energy to an electrical load. The primary function of a power supply is to convert one form of electrical energy to another. Power Distribution Unit (PDU) A Power Distribution Unit (PDU) is a free standing electrical distribution cabinet, whose main function is to provide a required point of power distribution. The PDU houses circuit breakers that are used to create multiple branch circuits from a single feeder circuit, and can also contain transformers, electrical panelboards, and surge protection devices. PUE Green Grid metric for energy efficiency. Stands for Power Usage Effectiveness. PUE is the ratio of total power supplied to the data center divided by the amount of power consumed by the IT equipment. A perfectly efficient PUE value is 1.0 Rack Power Distribution Unit (RPDU) An RPDU consists of multiple receptacles into which the power cords of the ITE can plug. A rack PDU is sometimes also known as a cabinet PDU or an enclosure PDU. It is packaged in a standard ICT equipment rack form factor so that it can be located either in the same rack/cabinet or in-row close to the IT equipment racks that it is powering. RPDU’s are typically cord-connected and available in a variety of form factors. Some are rack-mounted and take up 1-4U of shelf space. Others mount outside of the U area within a rack or cabinet (for example, mounted vertically in the rear of the rack or cabinet). RPDUs are also known by such names as “rack power module”, “rack PDU”, “zero-U PDU”, “enclosure PDU”, and “cabinet distribution unit”. Because it is the final connection point to the ITE, an RPDU can provide the finest level of power consumption granularity in the data center (except for what can be derived for the individual ITE itself).
- 59. 59 Remote Power Panel (RPP) A remote power panel is a distribution panel that is sub-fed from an upstream panel, usually from a floor-mount PDU. This PDU maximizes use of data center space by locating the point of distribution of moderately high current capacity close to compute loads. The device is basically a transformer-less PDU that meets a particular form factor requirement. Generally equipped with standard panelboards, this cabinet accommodates branch circuit breakers. Like its transformer- based relative, it can be configured with single or dual source feeds to accommodate an A-B bus and dual corded downstream equipment. Server A server is a running instance of an application (software) capable of accepting requests from the client and giving responses accordingly. Servers can run on any computer including dedicated computers, which individually are also often referred to as "the server". SNMP Simple network management protocol. Storage Array Typically a disk array provides increased availability, resiliency, and maintainability by using existing components (controllers, power supplies, fans, etc.), often up to the point where all single points of failure (SPOFs) are eliminated from the design. Structured Cabling Structured cabling is building or campus telecommunications cabling infrastructure that consists of a number of standardized smaller elements (hence structured) called subsystems. STS Static transfer switch - a transfer switch is an electrical switch that switches a load between two sources. Some switches are manual and some function automatically. Supercapacitor See definition of ultra-capacitor below Switch Disconnector A switch disconnector, disconnect switch or isolator switch is used to ensure that an electrical circuit is completely de- energized for service or maintenance. Such switches are often found in electrical distribution and industrial applications, where machinery must have its source of driving power removed for adjustment or repair. High-voltage isolation switches are used in electrical substations to allow isolation of apparatus such as circuit breakers, transformers, and transmission lines, for maintenance. The disconnector is usually not intended for normal control of the circuit, but only for safety isolation. Disconnectors can be operated either manually or automatically (motorized disconnector). Telnet Telnet is an application protocol used on the Internet or local area networks to provide a bidirectional interactive text- oriented communication facility using a virtual terminal connection
- 60. 60 TVSS Transient voltage surge suppressor Ultra-capacitor A super-capacitor (SC) (sometimes ultra-capacitor, formerly electric double-layer capacitor (EDLC), is a high-capacity electrochemical capacitor with capacitance values up to 10,000 farads at 1.2 volt that bridge the gap between electrolytic capacitors and rechargeable batteries. Uninterruptable Power Supplies (UPS) An uninterruptible power supply, also uninterruptible power source, UPS or battery/flywheel backup, is an electrical apparatus that provides emergency power to a load when the input power source, typically mains power, fails. A UPS differs from an auxiliary or emergency power system or standby generator in that it will provide near-instantaneous protection from input power interruptions, by supplying energy stored in batteries, super-capacitors, or flywheels. VRLA Valve regulated lead acid battery Water-Side Economization Water-side economization systems remove heat from the chilled water loop by a heat exchange process with outdoor air. Most water-side economization systems have an integrated economizer that allows outside air to contribute as much cooling benefit as conditions allow, while mechanical cooling operates simultaneously to satisfy any difference in meeting the total cooling load.