Data Center Networks

Sayed Chhattan Shah
Department of Information Communications Engineering
Hankuk University of Foreign Studies Korea
www.mgclab.com
Data Center Network

Data Center Network
 Data centers are developed to house a large-scale networked
computer system in a centralized and controlled environment
 Inside a data center, a large number of computing and
storage nodes are interconnected by a specially designed
network, called data center network

Data Center Network
 Challenges and requirements for the DCN design and operations
o Large Scale
 Modern DC to contain hundreds of thousands of servers
 Microsoft is hosting over 1 million servers in over 100 data centers
o Wide Variety of Applications
• Web search, Web mail, and interactive Games
• Infrastructure services such as distributed file systems and
distributed execution engines
 The diversified services and applications in DCs define a variety of
different traffic characteristics

Data Center Network
 Challenges and requirements for the DCN design and operations
o High Energy Consumption
 The annual data center energy consumption in the USA was
estimated to be more than 100 billion kWh in 2011
• 7.4 billion USD annual electricity cost
o Strict Service Requirement
 24 hours availability, which demands high system robustness
 Network failures from hardware, software, and human errors can be
inevitable
• Constant monitoring and agile failure recovery are required

Data Center Network Infrastructure
 The data center network infrastructure interconnects end devices in
a data center and across data centers
 DCN Infrastructure is categorized based on two dimensions
o Transmission technology
o Scale

 Intra Data Center Networks
o Highly complex since they interconnect a massive amount of
devices with critical performance requirements
o Ethernet is commonly used in data center networks

 The nodes can be configured to operate
o Ethernet-switched mode
 Ethernet MAC addressing is flat
 Require no address configuration. Server interfaces come ready for plug-n-play
deployment with manufacturer configured addresses
o IP-routed mode
 IP-routed networks more scalable
 IP addressing is hierarchical makes the size of forwarding tables smaller
 Disadvantage of hierarchical routing is that if a virtual machine is migrated to a
different host
• IP address needs to change to reflect its new topological position, which means
loss of live TCP connections
• A solution such as mobile IP is required
• Forwarding tables in all intermediate routers are updated

 Intra data center network topologies
o Electrical Element Based Topologies
o Electrical and Optical Element Based Topologies
o Electrical and Wireless Element Based Topologies

Electrical Element Based Topologies
 Switch-centric topologies
o Switches take the primary responsibility in network construction and
data transmission
o The switches are usually connected by hierarchy topologies and the
servers are generally connected to the low-level switches at network
edge

 Tree-based network architectures
o Unable to handle the growing demand of cloud computing
 The higher layers of the three-tier DCN are highly oversubscribed
o Tree-based network architectures are not scalable, fault tolerance, and
energy efficient

o Fat-Tree interconnects identical commodity Ethernet switches
 The advantage of Fat-Tree is that all switches are identical and cheap
commodity products can be used for all switches.
 There are multiple equal cost paths between any two hosts
 A drawback of Fat-Tree is its high cabling complexity
• A 48-ary Fat-Tree is with 27,648 servers, 2,880 switches, and 82,944 cables
The scalability is one of the
major issues and maximum
number of pods is equal to the
number of ports in each switch

o Core switches and aggregation switches forms a complete bipartite
graph, and each edge switch is connected to two aggregation switches
o VL2 reduces the number of cables leveraging higher speed
switch-to-switch links
 10 Gbps for switch-to-switch links and 1 Gbps for server-to-switch links

o Jellyfish constructs a degree-bounded random regular graph at
the edge layer
o An arbitrary server in Jellyfish can reach more servers in fewer
hops compared to Fat-Tree
A random graph is obtained by
starting with a set of n isolated
vertices and adding successive
edges between them at random.

 Server-centric topologies
o In switch-centric topologies, servers are merely endpoints
in the network
o In server-centric topologies, servers act as not only end
hosts, but also relay nodes for each other

o In a level-0 DCell, n servers are connected to a switch
o A level-1 DCell is constructed using n + 1 level-0 Dcells
 Specifically, one port of each server of each level-0 DCell connects to a
server in another level-0 Dcell
o The highlight of DCell is its excellent scalability
 A level-3 DCell can support
• 3,263,442 servers with 4-port servers
and 6-port switches

o A level-0 BCube consists of n servers connected to an n-port
switch, which is the same as a level-0 Dcell
o BCube makes use of more switches when constructing higher
level architecture

Electrical and Optical Element Based Topologies
 Combine
conventional
electrical
switching with
optical
switching
 Optical
network
connects ToR
electrical
switches
 High capacity optical links are offered to pairs of
racks transiently according to the traffic demand

 Helios is organized as a 2-level multi-rooted tree of pod
switches and core switches
o Core switches consist of both electrical switches and optical switches
 Helios estimates bandwidth demand and decides where to
forward traffic, the electrical network or the optical network
On each of the pod switches, the uplinks
are equipped with a optical transceiver.
Half of the uplinks are connected to the
electrical switches, while the other half are
connected to the optical switch through a
optical multiplexer.

 Explores the feasibility of a totally optical core network among
ToR switches
 Optical transceivers connected to a ToR switch use separated
send and receive fibers
o The multiplexers multiplex optical signals from many fibers to a single fiber
o The Wavelength Selective Switch forward optical signal to the 4 ports
according to the wavelength
 Switching time 14ms

Electrical and Wireless Element Based Topologies
 A hybrid network architecture is designed by adding 60 GHz
wireless links to the traditional electronic-based architecture
for extra capacity
 Each ToR switch is equipped with one or more 60 GHz
devices with directional antennas

Electrical and Wireless Element Based Topologies
 Wireless devices with rotatable directional antennas are
placed on top-of-rack
o Ceiling reflectors act as specular mirrors to reflect signals
o Electromagnetic absorbers are placed near each antenna to prevent
any local reflection and scattering
 3-D flyways
o reduce the interference footprint
o avoid blocking obstacles
o provide an indirect line-of-sight path
for reliable communication

 Inter Data Center Networks
o Geographically distributed data centers have been built
 Services from a local data center generally incur low latency
 Data backup and restore across geo-distributed data centers
can help avoid single point of failure

 Choice of the data center locations are influenced by multiple factors
o Geography
 Regions with minimum possibility of natural disasters
 Climate which support free cooling
o Electricity
 Cost, reliability, and cleanliness of the electricity are important
o Connectivity
 High quality of network connectivity
o Business
 Business friendly regulations and economic development incentives

Data Center Network Operations
 On the basis of the network hardware infrastructure, data
center network operation ensures data transport from sources
to destinations with various objectives
o Bandwidth guarantee
o Balanced load
o Energy efficiency

 Traffic Control in Data Center Networks
o To direct data traffic from sources and destinations
o Traditional approach
 Each switch learns the network topology based on exchanged
messages and constructs a forwarding table for packet forwarding

 Path Selection
o Packets are forwarded in DCNs are decided by various protocols
 Spanning tree
 Routing algorithm
 Multipath routing
 Encoding path information in the packets

 Path Selection
o DCell fault-tolerant routing
o BCube source routing protocol
o Traffic aware routing for FiConn
o Xpath
 Best path is selected according to various metrics
• hop distance
• path bandwidth
• link load
• MTU

 Rate Control
o Essential for congestion control, loading balancing, and
guaranteed bandwidth in a network
o It can be implemented at end hosts or in network
 Rate limiting at end hosts can be implemented explicitly using tool
provided by the OS
 Ethernet flow control use a PAUSE frame to pause the sender for a
time indicated in unit of quanta

 Priority Management
o Priority management delivers differentiated quality of service by
handling a packet based on its priority rather than the order of
arrival

 Network Utilization
 How to fully utilize the available bandwidth?
o Allocates paths for large flows based on the estimated demand
o Centralized traffic engineering, multipath routing, and rate limiting at
network edge
o Traffic limiting at end hosts, traffic path reconfiguration in network, and
priority differentiation

 Bandwidth Sharing
o Bandwidth is still shared in a best effort manner
o Malicious tenants can unfairly improve their network
performance
 establish multiple TCP connections
 Use UDP
o Efforts on bandwidth sharing often focus on two aspects
 Minimum bandwidth guarantee
 Bandwidth proportionality under different payment schemes

 Bandwidth Sharing
o Minimum bandwidth guarantee
 Ensures the amount of bandwidth that a tenant has paid for
 The most common method is bandwidth reservation
o Bandwidth proportionality under different payment schemes
 Without introducing extra SLAs on bandwidth, bandwidth
proportionality ensure that the allocated resource amount for a
tenant is proportional to what the tenant has paid for other
resources
• CPU
• memory

 Service Latency
o Shortest Remaining Time First is known to be the optimal
algorithm for minimizing average flow completion time over a
single link
 The flow with the least packets remaining is selected to be sent first
preemptively
o Deadline-Driven Delivery introduces deadline aware rating
allocation for flows
 Switches allocate bandwidth based on its capacity and the desired
rates when a flow starts or finishes

 Energy Consumption
o The most common approach for energy conservation in data
centers is to power off idle elements such as links, ports, and
switches

 Energy Consumption
o The most common approach for energy conservation in data
centers is to power off idle elements such as links, ports, and
switches
 ElasticTree is a typical power manager to dynamically choose a set
of active switches and links that can accommodate the traffic
demand and power down unneeded links and switches as many as
possible
 GreenTE optimizes the routing to maximize the number of links that
can be put into sleep while maintaining the performance

Resource Management in Cloud Data Centers

 In a traditional data center each physical machine can only
serve one application at a time
 In a Virtualized Cloud Data Center when a service request is
processed, a prebuilt image is used to create one or more VM
instances
o When the VM instances are deployed, they are provisioned with specific CPU,
memory, and disk capacity
o VMs are deployed on PMs, each of which may be shared by multiple VMs

 Objectives of resource management schemes
o Completion time
o Load balancing
o Throughput
o Utilization of resources
o Failure management
o Energy consumption
o Incentives
o Multiple objectives

 Clouds utilize hardware virtualization, which enables a
physical machine to run multiple virtual machines
 A cloud hosts multiple applications on the VMs
o Since the load of each VM on a PM varies over time, a PM may
become overloaded
 Overloaded PMs migrate their VMs to under-loaded PMs
• Process of selecting migration VMs and destination PMs is complex and
generates high delay and overhead
 PM predict VM resource demand
• PM does not know which VMs to migrate out

 The key challenges related to energy efficiency
o How to optimally solve the trade-off between energy savings and
delivered performance?
o How to determine when, which VMs, and where to migrate in order to
minimize energy consumption by the system, while minimizing migration
overhead and ensuring SLA?
o How to develop efficient decentralized and scalable algorithms for
resource allocation?
o How to develop comprehensive solution by combining several allocation
policies with different objectives?

 Most energy-efficient resource allocation solutions focus on
minimizing energy consumption or costs, and do not consider
dynamic service requirements of consumers that can be changed on
demand in Cloud computing environments
 Need for autonomic energy-aware resource management
mechanisms and policies

 Energy-Aware Data Centre Resource Allocation
o The problem of VM allocation can be divided in two parts
 Admission of new requests for VM provisioning and placing the VMs
on hosts
• A bin packing problem with variable bin sizes and prices
• Allocate each VM to a host that provides the least increase of power
consumption due to this allocation
 Optimization of current allocation of VMs
• Select VMs that need to be migrated
• Chosen VMs are placed on hosts

 Minimization of power consumption in a heterogeneous
cluster of computing nodes
o The main technique applied to minimize power consumption is
concentrating the workload to the minimum of physical nodes
and switching idle nodes off

 As the workload changes, resources allocated to applications
could automatically scale from
o the horizontal direction
 adding more VMs
• may take dozens of seconds
o the vertical direction
 allocating more resources to deployed VMs
• needs additional support from both the host operation system

 Design of resource management scheme that consider
o Characteristics of physical machines
 Processing power, memory, storage, energy consumption, queue
length, switch time, price
o Virtual Machines
o Application requirements
 Size, data, deadline, price
o Network Environment
 Bandwidth, Link quality, lifetime, traffic, energy consumption, price

Data Center Networks

More Related Content

What's hot (20)

Similar to Data Center Networks (20)

More from Sayed Chhattan Shah (15)

Recently uploaded (20)

Data Center Networks