White Paper: EMC Backup and Recovery for Microsoft Exchange and SharePoint 2010 Servers - EMC Avamar, EMC VNX, VMware vCenter

White Paper

EMC BACKUP AND RECOVERY FOR
MICROSOFT EXCHANGE AND
SHAREPOINT 2010 SERVERS
EMC Avamar, EMC VNX, VMware vCenter
• Improved multi-streaming backups with Avamar
• Backup space savings with deduplication

EMC Solutions Group

Abstract
This white paper explains the testing and validation of the backup and recovery
capabilities and benefits of EMC® Avamar™ in a virtualized Microsoft
SharePoint 2010 and Microsoft Exchange Server 2010 environment on EMC
VNX™.

March 2012

Copyright © 2012 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its
publication date. The information is subject to change without notice.

The information in this publication is provided “as is.” EMC Corporation makes
no representations or warranties of any kind with respect to the information in
this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this
publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation
Trademarks on EMC.com.

VMware, ESX, ESXi, VMware vCenter, VMware vCenter Server, and VMware
vSphere are registered trademarks or trademarks of VMware, Inc. in the United
States and/or other jurisdictions. All other trademarks used herein are the
property of their respective owners.

Part Number: H8936

EMC Backup and Recovery for Microsoft Exchange and SharePoint 2010 Servers 2

Table of contents
Executive summary ............................................................................................................................. 6
Business case .................................................................................................................................. 6
Microsoft Exchange 2010 ............................................................................................................ 6
Microsoft SharePoint 2010 .......................................................................................................... 6
Solution overview ............................................................................................................................ 6
Key results ....................................................................................................................................... 7

Introduction ....................................................................................................................................... 8
Purpose ........................................................................................................................................... 8
Scope .............................................................................................................................................. 8
Audience.......................................................................................................................................... 8

Technology overview .......................................................................................................................... 9
Introduction ..................................................................................................................................... 9
EMC Avamar ..................................................................................................................................... 9
Application plug-ins .................................................................................................................... 9
EMC VNX .......................................................................................................................................... 9
VMware vCenter ............................................................................................................................. 10
VMware vCenter Server .............................................................................................................. 10
VMware vSphere ............................................................................................................................ 10
Kroll Ontrack PowerControls ........................................................................................................... 10

Configuration ................................................................................................................................... 11
Introduction ................................................................................................................................... 11
Physical environment ..................................................................................................................... 11
Hardware resources ....................................................................................................................... 12
Software resources ........................................................................................................................ 13
Disk layout ..................................................................................................................................... 14
Networking .................................................................................................................................... 15
Environment profile........................................................................................................................ 16

Deployment ...................................................................................................................................... 17
Introduction ................................................................................................................................... 17
VNX5500 ....................................................................................................................................... 17
VMware vCenter Server .................................................................................................................. 17
VMware vSphere ............................................................................................................................ 17
Avamar .......................................................................................................................................... 17
Load balancer ................................................................................................................................ 18
Kroll Ontrack PowerControls ........................................................................................................... 18


Microsoft Exchange Server 2010 ...................................................................................................... 19
Introduction ................................................................................................................................... 19
Design ........................................................................................................................................... 19
Configuration ................................................................................................................................. 20
Jetstress validation ........................................................................................................................ 21

Microsoft SharePoint Server 2010 .................................................................................................... 23
Introduction ................................................................................................................................... 23
Design ........................................................................................................................................... 23
Test methodology .......................................................................................................................... 23
Configuration ................................................................................................................................. 23

Backup testing methodology ............................................................................................................ 25
Backup methodology ..................................................................................................................... 25
Data change rates explained .......................................................................................................... 25
Test types ...................................................................................................................................... 25
Initial baseline performance ...................................................................................................... 25
No data change and no load backup ......................................................................................... 25
Percentage data change with no load backup ............................................................................ 25
Same daily percentage data change with full user load backup ................................................. 26
Backup configuration ..................................................................................................................... 26
Exchange configuration ............................................................................................................. 26
SharePoint configuration ........................................................................................................... 28

Summary of test results .................................................................................................................... 31
Introduction ................................................................................................................................... 31
Summary charts and tables for backups and restores .................................................................... 31
Exchange backups .................................................................................................................... 31
SharePoint backups .................................................................................................................. 32
Exchange restores ..................................................................................................................... 32
SharePoint restores ................................................................................................................... 33
Deduplication measurements.................................................................................................... 34
Detailed observations .................................................................................................................... 34
Performance testing results ....................................................................................................... 34
Benefits ......................................................................................................................................... 35
Notes and recommendations ......................................................................................................... 36

Conclusion ....................................................................................................................................... 37
Summary ....................................................................................................................................... 37
Findings ......................................................................................................................................... 37


References ....................................................................................................................................... 39
White papers ................................................................................................................................. 39
Product documentation.................................................................................................................. 39

Appendix A: Detailed test results ...................................................................................................... 40
Baseline application performance testing ...................................................................................... 41
Introduction .............................................................................................................................. 41
Exchange 2010 Server ............................................................................................................... 41
SharePoint 2010 Server............................................................................................................. 46
Avamar: Backup testing with no data change and no load.............................................................. 49
Introduction .............................................................................................................................. 49
SharePoint 2010 Server: Test results ......................................................................................... 49
Combined Avamar backup testing: With data change and without load ......................................... 51
Introduction .............................................................................................................................. 51
Exchange 2010 Server: Test results ........................................................................................... 51
Combined Avamar backup testing: With data change and load ...................................................... 54
Introduction .............................................................................................................................. 54
Avamar test results with encryption ............................................................................................... 62
Exchange................................................................................................................................... 62
SharePoint ................................................................................................................................ 62
Avamar performance with encryption ........................................................................................ 62
Restore testing: Avamar Exchange native GLR restore .................................................................... 63
Introduction .............................................................................................................................. 63
Restore testing: Avamar/Exchange database recovery ................................................................... 64
Introduction .............................................................................................................................. 64
Restore testing: Avamar/ SharePoint VSS restore of site collection database ................................. 65
Introduction .............................................................................................................................. 65
Test results ................................................................................................................................ 65
Avamar restore testing: Kroll recovery of a site and single item in SharePoint ................................. 66
Introduction .............................................................................................................................. 66
List restore ................................................................................................................................ 66
Single-item restore .................................................................................................................... 68
Hardware load balancing ............................................................................................................... 69
Introduction .............................................................................................................................. 69
Test results ................................................................................................................................ 70


Executive summary
Business case Data is one of the single most valuable assets a company can own. With the
explosion in both the volume of data being retained and its speed of change, you are
challenged to ensure constant access and protection against corruption or loss,
either through hardware, environmental, or man-made disasters. The introduction of
virtualization and the move to the cloud also further complicates this challenge.

Downtime can represent failure to critical business processes, which can effectively
halt business operations and lead to lost revenue, regulatory fines, and, potentially,
lost customers. The loss of even some megabytes of data in recovery can amount to
hundreds and thousands of transactions lost, which affects businesses in terms of
financial revenue, operational effectiveness, and legal obligation.

Microsoft often forms the foundation for today's most demanding, enterprise-level,
transaction-based companies. Microsoft SharePoint Server and Exchange may be core
platforms within the mission-critical framework for many businesses. Availability and
recoverability become critical factors when designing the supporting IT infrastructure.

Microsoft Exchange 2010
Customers are striving to provide users with larger mailboxes while reducing their
businesses' Exchange backup data storage requirements and complexity. With
current trends, Exchange users' mailbox sizes are growing rapidly, thus making it
more challenging, if not impossible, to back up all of the Exchange data within the
nightly backup windows.

Microsoft SharePoint 2010
SharePoint collaboration is an integral part of many businesses. Designing and
building a database infrastructure that supports this collaboration and protects the
data is a major challenge.

SharePoint is always on and accessed by users. At the same time, users store more
and more data. In many cases, it is not acceptable to take the SharePoint application
and its databases offline for nightly backups. If you require a restore, it must happen
quickly with minimum impact to users.

Solution overview This solution offers an efficient approach to protect Microsoft Exchange 2010 and
Microsoft SharePoint 2010 on a physical or virtualized environment. This solution
uses a virtualized environment.

EMC® Avamar™ provides comprehensive backup and recovery for Microsoft
environments by using Avamar 6.0 Volume Shadow Copy Services (VSS) agent. This
solution tests and validates Avamar and provides recommendations and best
practices for using Avamar to back up Microsoft Exchange and SharePoint based on
the test results.

Avamar multi-streaming is highlighted, which enables higher throughput of backups.


Key results Benefits of the solution for Exchange and SharePoint are:
• Enables a rapid yet application-consistent backup and restore using Microsoft
VSS framework.
• Enables fast recovery to support strict recovery time objectives (RTOs) for
databases and single items without requiring a slow item-level based backup.
• Significantly reduces backup windows and backup-storage footprints by using
Avamar client-side data deduplication to only store unique daily changes, while
always maintaining daily full backups for immediate single-step restore.
• Provides easy management by using one Avamar GUI for Avamar backups.
• Provides hardware load balancing that introduces redundancy to SharePoint
and improves SharePoint farm performance.


Introduction
Purpose This white paper documents the testing and validation of backup, deduplication, and
restore processes for Microsoft Exchange Server 2010 and Microsoft SharePoint 2010
using Avamar and EMC VNX5500™ in a virtualized server environment, enabled by
VMware® vSphere™ 5.0.

This solution shows the new multi-streaming support of the Avamar agents to enable
higher throughput of backups. For Microsoft Exchange, VSS copies of passive
availability group (PAG) databases were used to offload backup from active
databases. For Microsoft SharePoint, VSS copies of the SharePoint farm databases
were used.

Scope This use case shows the performance of backups and restores to the Avamar grid.
Testing of this scenario provides best practices.

The solution was also tested for backup and recovery, and deduplication
performance aspects, with a focus on backup and recovery throughput and timing to
showcase the ability to meet RPO and RTO requirements.

Audience This white paper is intended for EMC employees, partners, and customers with an
interest in backup and recovery for Microsoft environments. Readers should already
be familiar with Microsoft Exchange and Microsoft SharePoint technologies and EMC
backup and recovery services.


Technology overview
Introduction This chapter provides an overview of the primary technologies used in this solution:
• EMC Avamar
• EMC VNX5500
• VMware vCenter Server™
• VMware vSphere
• Kroll Ontrack PowerControls

EMC Avamar EMC Avamar enables fast, efficient backup and recovery by reducing the size of
backup data at the client—before it is transferred across the network and stored.
Avamar’s data deduplication dramatically reduces network traffic by only sending
unique blocks over the LAN/WAN. Blocks that were previously stored are never
backed up again. This means huge savings in bandwidth for backup, a lot less disk
storage needed on the back end, and very fast backups—often as much as ten times
faster.

Avamar backups can be quickly recovered in just one step—eliminating the hassle of
restoring full and subsequent incremental backups to reach the desired recovery
point. The backup data is encrypted during transit across the network and at rest for
added security.

Application plug-ins
For tight integration with your Microsoft applications, EMC offers the following
Avamar application plug-ins:
• EMC Avamar plug-in for Exchange VSS—Delivers backup and recovery with data
deduplication for Microsoft Exchange Server. Supports granular recovery of
individual mailboxes, folders, or messages.
• EMC Avamar plug-in for SharePoint VSS—Provides backup and recovery with
data deduplication for Microsoft SharePoint. Achieves granular recovery for
SharePoint objects such as Web applications, sites, and calendars.

EMC VNX The EMC VNX family delivers industry-leading innovation and enterprise capabilities
for file, block, and object storage in a scalable, easy-to-use solution. This storage
platform combines powerful and flexible hardware with an advanced efficiency,
management, and protection software to meet the demanding needs of today's
businesses.

The VNX series is designed to meet the high-performance, high-scalability
requirements of midsize and large businesses.

The VNX array model used in this solution is VNX5500. While the VNX5500 is
selected, this solution also applies to EMC VNX5300™, EMC VNX5700™ and EMC
VNX7500™ models.


VMware vCenter You can simplify your IT management with VMware vCenter virtualization and cloud
management solutions. vCenter solutions help you accelerate IT service delivery,
transform operational efficiency, automatically assure compliance, and reduce
business risks.

VMware vCenter Server
By using VMware vCenter Server, key elements like hosts, clusters, resource pools,
datastores, networks, and virtual machines can be viewed, configured, and managed.
vCenter Server aggregates physical resources from multiple ESX/ESXi hosts and
presents a central collection of simple and flexible resources for you to provision to
virtual machines in the virtual environment.

vCenter Server has the following characteristics:
• Centralized management
• Operational automation
• Secure access control
• Availability and resource management
• High levels of security
• Automated energy efficiency

VMware vSphere VMware vSphere uses the power of virtualization to transform datacenters into
simplified cloud computing infrastructures, and enables IT organizations to deliver
flexible and reliable IT services. vSphere virtualizes and aggregates the underlying
physical hardware resources across multiple systems and provides pools of virtual
resources to the datacenter.

As a cloud operating system, vSphere manages large collections of infrastructure
(such as CPUs, storage, and networking) as a seamless and dynamic operating
environment, and also manages the complexity of a datacenter.

Kroll Ontrack Kroll Ontrack PowerControls software enables you to quickly and easily find, recover,
PowerControls and restore granular SharePoint content, such as documents, lists, libraries, and
folders, or entire SharePoint sites, from full Avamar backups.


Configuration
Introduction This solution shows how a customer can back up and restore applications, Microsoft
Exchange Server 2010, and Microsoft SharePoint 2010, using EMC Avamar
application plug-ins for Exchange and SharePoint.

Performance results are shown for the Avamar grid as backup targets.

The solution is configured as follows:
• All the virtual machines use VNX5500 for disk storage and management. The
virtual machines are split between the ESX servers.
• EMC used converged networking such as Fibre Channel over Ethernet (FCoE) to
significantly reduce the connectivity complexity and port count for this solution,
from server to VNX storage connectivity. The platform, optimized for virtual
environments, is designed within open industry-standard technologies and
aims to reduce the total cost of ownership (TCO) and increase business agility.
The system integrates a low-latency, lossless, 10-gigabit Ethernet (GbE) unified
network fabric with enterprise-class, x86-architecture servers.
• Avamar is used for backup and restore, and Kroll Ontrack is used to perform
granular restores for SharePoint. The Avamar plug-in for Exchange VSS includes
a granular level recovery (GLR) feature that enables recovery of mailboxes,
folders, and mail messages.
• A hardware load balancer is used between the client load and the web front-
ends (WFEs) for SharePoint load testing.

Physical Figure 1 shows the overall physical architecture of the solution.
environment


Figure 1. Physical architecture

Hardware The hardware used to validate the solution is listed in Table 1.
resources Table 1. Hardware

Equipment Used for Quantity Configuration
ESX Servers VMware, Exchange, 2 • 256 GB of RAM
SharePoint
• 32-core
• 146 GB RAID 5 internal hard
disk drives (HDD)

Management VMware vCenter server 1 • 16 GB of RAM
servers
• Dual-core

EMC VNX5500 VMWare virtual 1 • Exchange – NL-SAS
machines, Exchange
• SharePoint – SAS
databases and logs,
SharePoint content
databases and logs


Equipment Used for Quantity Configuration
EMC Avamar Backup of SharePoint 1 • Gen 4, 6-node
and Exchange
• OS version 6.0 SP1
databases

Software resources The software used to validate the solution is listed in Table 2.
Table 2. Software

Software Version
Microsoft Windows Server 2008 R2 Enterprise Edition

Microsoft Exchange Server 2010 SP1 Enterprise Edition

Microsoft SharePoint Server 2010 Enterprise Edition

Microsoft SQL Server 2008 R2 Enterprise Edition

EMC Avamar Windows Client 6.0 SP1

EMC Avamar Windows Exchange VSS Plug-in 6.0 SP1

EMC Avamar Windows Exchange GLR Plug-in 6.0 SP1

EMC Avamar Windows SharePoint VSS Plug-in 6.0 SP1

Kroll Ontrack PowerControls 6.1

VMware vSphere 4.1


Disk layout Figure 2 shows the disk layout for the solution.

Figure 2. VNX disk layout

As Figure 2 shows, the first enclosure 0_0 was not used. Two 8+8 RAID 1/0 storage
pools were used for the Exchange databases. The Exchange logs were on two 1+1
RAID 1/0 storage pools.

The boot LUNs for all the virtual machines were on a five-disk 4+1 RAID 5 storage
pool.

The SharePoint ContentDB data and log LUNs were on an nine-disk 8+1 storage pool,
while the TempDB and SearchDB LUNs were on a 2+2 RAID 1/0 storage pool.


Networking Figure 3 shows the network layout of the solution.

Figure 3. Network layout

At the time of this solution testing, Avamar supported 1 Gb network interfaces.
Figure 3 shows how EMC used a 1/10 Gb switch to connect Avamar to the 10 Gb FCoE
network. Since each Avamar node had a 1 Gb connection to the network, multi-
streaming allowed Avamar to use several nodes at once; therefore, multiple 1 Gb
network connections at once. The network was not a bottleneck in any of EMC’s tests.

EMC attached the VNX5500 and ESX servers to the 10 Gb FCoE network.

Client load was generated from the 1 Gb network in the same way it would in a normal
production environment. The client load used a public network to connect to the
virtual machines on the ESX servers.

The Avamar nodes used a separate backup network to do backups and restores.


Environment This solution was validated with the environment profile listed in Table 3.
profile
Table 3. Environment profiles

Application Requirements Quantity/Type/Size
Exchange Exchange mailbox count total 5,000
Server 2010 (2,500 per active server)

Exchange mailbox size 1.5 GB

Exchange User profile 150 messages sent/received
per day (0.15 IOPS per user)

Exchange—Availability DAG

Disk type and RAID configuration 2 TB NL-SAS–7.2k RAID 1/0

SharePoint SharePoint—Total user count 6,000
Server SharePoint—Total data 2.5 TB

SharePoint—Max content DB size 200 GB

SharePoint—Number of ContentDBs 14

SharePoint—Document size range 200 k to 2 MB

SharePoint—Total site collection count 14

SharePoint—Size per site About 10 GB

SharePoint—Sites per site collection 3–10

SharePoint—Total site count 200

SharePoint—Usage profile(s) (%browse/% 80%/10%/10%
search/%modify)

SharePoint—User concurrency 10%

Disk type 600 GB SAS–10k RAID 5


Deployment
Introduction This chapter describes the deployment of the hardware and software used in this
solution:
• VNX5500
• VMware vCenter Server
• Avamar
• Load balancer
• Kroll Ontrack PowerControls

VNX5500 VNX5500 was configured with eight 100 GB Flash drives, fifty-seven 600 GB SAS
drives, and forty-five 2 TB NL-SAS drives. The drives were split between two buses.
The Flash drives were not used for this use case. Two front-end ports per storage
processor (SP) were connected to the FCoE network.

VMware vCenter vCenter Server was installed on a management server and used to manage the ESX
Server servers, and provision and manage virtual machines for Exchange and SharePoint.

For high availability of the virtual machines across the ESX servers, VMware high
availability (HA) with vMotion was configured and enabled in vCenter Server.

VMware vSphere vSphere was used as a platform to which virtual machines for Exchange and
SharePoint were provisioned.

Avamar Avamar was deployed with one utility node and five storage nodes. Each Avamar
node had dual internal network connections and an external network connection. The
external network connections were split between two network switches. This layout
has an additional external switch, which is the only difference to the layout shown in
Figure 4.


Figure 4. Avamar deployment

Load balancer A hardware load balancer was used for the load balancing testing. This was
configured to accept requests for the WFEs and then to split the requests equally
between them.

Kroll Ontrack Kroll Ontrack PowerControls was installed on the Avamar front-end server (lightly
PowerControls utilized SharePoint application server) to do SharePoint granular restores. Kroll
Ontrack PowerControls is able to extract items from the raw database files. This saves
time as well as the extra space required by doing a full restore of the database to
another SQL Server and then extracting the items.


Microsoft Exchange Server 2010
Introduction In this solution, the Exchange organization consists of two mailbox servers and two
client access and hub transport servers. High availability is provided through the use
of the database availability group (DAG). Within a DAG, a set of mailbox servers
performs continuous database replication through the use of host-based log
shipping, to provide automatic database recovery in the event of failures. In this
solution, each database is deployed with two DAG copies on two Exchange Server
2010 mailbox servers.

Design Due to the many variables and diversities between different organizations, sizing and
configuring storage for use with Microsoft Exchange Server 2010 can be a
complicated process. One of the methods used to simplify the sizing and
configuration of storage for use with Exchange Server 2010 is to define a unit of
measure—a building block.

A building block represents the amount of disk and server resources required to
support a specific number of Exchange Server 2010 users. The amount of required
resources is derived from the:
• Specific user profile type
• Mailbox size
• Disk requirements

Using the building block approach helps to remove much of the guesswork required
to simplify the implementation of the Exchange Server 2010 mailbox server. After the
initial building block is designed, it can easily be reproduced and scaled to support
the required number of total users in the organization.

Your Exchange administrators can now create building blocks that are based on your
company’s specific requirements for the Exchange environment. This approach is very
helpful when you expect future growth, because it makes scalability of the Exchange
environment much easier and more straightforward.

EMC’s best practices involving the building-block approach for Exchange Server
design have proven to be very successful throughout many customer
implementations.

For more information about the building block calculation, refer to Microsoft
Exchange Server 2010 Performance Review Using the EMC VNX5300 Unified Storage
Platform—An Architectural Overview.
Table 4 and Table 5 show the user profile and building block used in this solution.
Table 4. User profile

User profile characteristic Value
Number of users supported 5,000 users

User profile supported 150 messages sent/received per day (0.15
IOPS per user)


User profile characteristic Value
Read/write ratio 3:2

Number of mailbox servers 2

Number of DAG copies 2

Mailbox size 1.5 GB

Table 5. Building block

Building block Value
User count per server 5,000 (2,500 active users, 2,500 passive
users)

Number of databases per server 10 (5 active, 5 passive)

User count per database 500

Database LUN size 1.4 TB

RAID type and disks required (database RAID 1/0 (16 x 2 TB NL-SAS drives) in a
LUN) storage pool

Log LUN size 72 GB

RAID type and disks required (log LUN) RAID 1/0 (2 x 2 TB NL-SAS drives)

Configuration In this solution, the first client access server also acts as the file share witness. To
remove a single point of failure, the virtual machines of the two mailbox servers and
the two client access and hub transport servers are hosted on different ESX hosts.

The virtual machine and LUN configurations are shown in Table 6 and Table 7.
Table 6. Virtual machine configurations

Virtual machine Quantity CPU RAM (GB)
Mailbox server 2 6 36

Hub/client access servers 2 4 16
combination

Table 7. LUN configurations

LUN size File system
Virtual machine Role Quantity Storage pools
(GB) type
Mailbox server Boot 1 100 SAS 600 GB RAID 5 4+1 VMFS
(shared pool)

Mount point 1 10 SAS 600 GB RAID 5 4+1 RDM
(shared pool)

Database 10 1,400 2 TB NL-SAS RAID 1/0 RDM

Log 10 72 2 TB NL-SAS RAID 1/0 RDM


LUN size File system
Virtual machine Role Quantity Storage pools
(GB) type
Hub/client Boot 1 100 SAS 600 GB RAID 5 4+1 VMFS
access servers
combination

Jetstress You should validate Exchange 2010 storage design for expected transactional I/O per
validation second (IOPS) before it is placed in a production environment. To ensure that the
environment functions appropriately, EMC recommends that the Microsoft Jetstress
tool is used to validate the Exchange storage design.

The Jetstress tool simulates Exchange I/O at the database level by interacting directly
with the Extensible Storage Engine (ESE) database technology (also known as Jet) on
which Exchange is built.

You can configure Jetstress to test the maximum I/O throughput available to the disk
subsystem within the required performance constraints of Exchange. Jetstress can
accept a simulated profile of specific user counts and IOPS per user to validate that
the disk subsystem is capable of maintaining an acceptable performance level by the
metrics defined in that profile.

The 64-bit version of Jetstress 2010 can be downloaded from the Microsoft
Download Center.

In this solution, EMC used Jetstress version 14.01.0225.017 to simulate an I/O
profile of 0.15 IOPS per user. EMC validated the building blocks using a two-hour
performance test.

Table 8 and Table 9 show the average I/O and the average latency on the mailbox
server. The performance meets the design target.

Table 8. I/O and latency for mailbox server (database)

Database I/O Target values Achieved values
Achieved transactional IOPS (I/O 750 (5,000 x 0.15) 755.6
database reads/sec + I/O
database writes/sec)

I/O database reads/sec N/A 459.8

I/O database writes/sec N/A 295.8

I/O database reads average Less than 20 ms 17.9
latency (ms)

I/O database writes average This counter is not a good 2.3
latency (ms) indicator for client latency
because database writes are
asynchronous


Table 9. I/O and latency for mailbox server (transaction log)

Transaction log I/O Target values Achieved values
I/O log writes/sec N/A 251.9

I/O log writes average latency (ms) Less than 10 ms 0.9

Total I/O Target values Achieved values

(DB+Logs+BDM+Replication)/sec N/A 1168.7


Microsoft SharePoint Server 2010
Introduction In this solution, the SharePoint farm consists of a Microsoft SQL Database Server, an
application server for central administration, two WFEs, and one crawl/query server.

Design The SharePoint farm is designed to support a specific number of users at a certain
concurrency. It is designed with SharePoint best practices in mind, the only change
being the use of SAS storage pools for all SharePoint components. EMC used storage
pools instead of RAID groups for all LUNs.

The design is based on the best practices provided in EMC Virtual Infrastructure for
Microsoft SharePoint Server 2010 Enabled by EMC CLARiiON and VMware vSphere 4—
A Detailed Review.

Test methodology EMC used Microsoft Visual Studio Team System (VSTS) to simulate the load on the
SharePoint farm. A client load emulation tool was used to ensure that the SharePoint
farm was operating at the optimal performance level.

All users adhered to a Microsoft heavy-user profile, which specifies 60 requests per
hour. A think time of zero percent was applied to all tests. “Zero percent think time”
is the elimination of typical user decision-making time when browsing, searching, or
modifying data using Microsoft Office SharePoint Server. Every user request is
completed from start to finish without a pause, which generates a continuous
workload on the system.

The maximum user capacity is derived from the following formula:
# = seconds per hour/RPH/Concurrency_percent*RPS

Example: 3,600/60/1 percent*12=72,000

Example: 3,600/60/10 percent*12=7,200 (supported user capacity for
10 percent concurrency)

The response times for each organization are detailed in Table 10.

Table 10. Response times

Test type Action Percent Mix Response time
Browse User browse 80 Less than 3 seconds

Search Unique value 10 Less than 3 seconds
search

Modify Browse and 10 Less than 3 seconds
metadata modify

Configuration The SharePoint farm consists of 14 site collections, with three to ten sites each, for a
total of 100 sites. Each site contains different numbers and sizes of documents for a
total of 2.5 TB of documents for all sites combined.


The virtual machine and LUN configurations are shown in Table 11, Table 12, and
Table 13.

Table 11. SharePoint virtual machine configuration

Virtual machine Quantity CPUs RAM (GB)
SQL Server 1 8 32

WFE 2 4 4

Crawl/Query 1 4 8

Application 1 4 4

Table 12. SharePoint LUN configuration

Virtual VMFS disk LUN
Role Quantity Storage pools
machine sizes sizes
SQL Server Boot 1 50 1 TB SAS 600 GB
(shared) RAID 5 4+1

ContentDBs 14 100-250 1.5 TB SAS 600 GB
CDB Logs 1 10-30 1.5 TB RAID 5 8+1
SP Config 1 100 500 GB
SQL Internal 1 10

TempDB Data 1 100 1 TB SAS 600 GB
TempDB Log 1 10 RAID 1/0 2+2
SearchDB 1 350
SearchDB Log 1 35

WFE Boot 6 50 1 TB SAS 600 GB
(shared) RAID 5 4+1

Crawl/Query Boot 2 50 1 TB SAS 600 GB
Index 2 100 (shared) RAID 5 4+1

Application Boot 1 50 1 TB SAS 600 GB
(shared) RAID 5 4+1

Table 13. Number of ContentDBs per LUN

LUN Number of ContentDBs
ContentDBs_1 6

ContentDBs_2 5

ContentDBs_3 3


Backup testing methodology
Backup EMC ran a full initial backup before backup testing to Avamar. This simulates the
methodology initial backup that customers would have to do when they first back up an
environment to Avamar. The backup times for these initial backups are not relevant to
EMC’s testing because they are not the day-to-day backup operations that would
happen in an organization.

When a backup to Avamar runs, the Avamar client separates the data into segments
and calculates a hash for each segment. These hashes are then checked against
existing hashes on the Avamar grid. If the hash already exists, then it is not backed
up. This is client-side deduplication.

Data change rates During testing, EMC used a specific rate of data change against the Exchange and
explained SharePoint databases, 3 percent for Exchange and 10 percent for SharePoint. These
parameters were accurately defined in the load tools used for the testing.

Application data change rate in Exchange or SharePoint is not that same as the data
change rate that is reported by Avamar. Avamar compares every segment of the
changed data file and only copies the changed segment of any data file and not the
whole data file. When a 2 MB file is modified in SharePoint, it does not mean the
whole file has changed; Avamar only backs up the changed segments.

In this solution, an Exchange data change rate of 3 percent equates to a 1.5 percent
data change rate for Avamar, and a SharePoint data change rate of 10 percent
equates to an 8 percent data change rate in Avamar.

Test types Initial baseline performance
EMC’s initial test was a baseline performance test to establish a baseline for
comparison with subsequent backup tests. EMC wanted to compare the impact of
backups on application performance.

For more details on the baseline performance test, see Baseline application
performance testing.

No data change and no load backup
EMC then did a backup test where there was no data change and no load. This gave a
baseline backup result that EMC could compare to other backup tests. Even though
there is no data change, there are still time, CPU, IOPS, and processing costs related
to the segmenting and hashing functions.

Percentage data change with no load backup
The next test was a percentage data change with no load. For SharePoint this was
approximately 10 percent, and for Exchange this was approximately 3 percent. This
test simulated a customer environment where backups would take place daily during
a backup window with no user load.


Same daily percentage data change with full user load backup
The final backup test was the same daily percentage data change but with full user
load. This test simulated backups run after a data change rate of 10 percent for
SharePoint and 3 percent for Exchange. This scenario might apply to customers who
have no downtime window for backups, for example, customers who need to operate
24 hours a day but still want to do daily backups.

Backup On-demand backups were used due to the test schedule, with a retention policy of No
configuration end date. Encryption was set to "none" apart from the specific encryption test that
was run.

Exchange configuration
In order to get optimized backup performance, Exchange backup was configured as
follows:
• Exchange 2010 was configured with a DAG, which contains two Exchange
servers with the mailbox role installed.
• Exchange backups were performed on passive database copies from both
mailbox servers.
• Each database was placed on two separate disks—one disk for the database
file and one disk for the transaction logs.
• Each database was roughly the same size.
• Six backup streams were specified in each backup job.

Figure 5 shows the Avamar backup GUI with only passive database copies selected.

Figure 5. Selecting an Exchange Information Store for backup


Figure 6 shows the Backup Options window with the default setting for backup set to
“Always” to backup all selected databases.

Figure 6. Setting backup options for Exchange

Alternatively, you can select all databases and change Set when backup occurs on
clustered or DAG systems to “Replica (passive) writer only”. Selecting this option
means you do not have to determine which databases are passive and you do not
have to individually select them for backup; the passive databases are automatically
selected and backed up by the program.

The maximum of six backup streams were chosen to back up the targets in parallel.
Consistency checking was disabled when simulating daily backups because this was
a DAG configuration and Exchange 2010 has internal database integrity checking. As
a best practice, EMC recommends that you enable consistency checks on a weekly
basis.

Note Improved internal database integrity checking in Exchange 2010 reduces the
likelihood that any database corruption will be included in backup images.
This reduces the need to take a database offline to perform consistency
checking by using the CHKSGFILES API or the ESEUTIL application. For more
details, refer to the Microsoft Library Exchange Server 2010 SP1 Backup and
Restore SDK.


SharePoint configuration
In order to get optimized backup performance, SharePoint backup was configured as
follows:
• EMC used the Multi-stream by database backup option with the maximum of
six streams selected.
• The Windows SharePoint VSS plug-in provided VSS backup and recovery of
SharePoint and discovered components.
• Database sizes ranged from 70 GB to 200 GB.
• Windows SharePoint GLR plug-in provided support for granular-level recovery of
SharePoint items when used in conjunction with a supported third-party
recovery tool.

Figure 7 shows the Avamar backup GUI where a SharePoint farm VSS backup is
selected. The tce-sp-app-01 server is the Avamar front-end server. When the farm is
selected, Avamar VSS automatically selects the SharePoint instance on the SQL
Server as well as the Search Service Application.

Figure 7. Selecting a SharePoint farm for backup


Figure 8 shows backup options where you can set the retention period as well as the
encryption method, if required.

Figure 8. Setting backup options for SharePoint


Figure 9 shows SharePoint VSS backup options where you can choose to select
multiple targets and set the maximum number of backup streams. You can also
choose how you want to group the backup components, either by database or by
volume. In this solution, we grouped by database.

Figure 9. Windows SharePoint VSS-specific backup options


Summary of test results
Introduction These results show the comparative backup and restore times and deduplication
rates based on EMC’s tests.

For more detailed information on the testing and validation done for this solution, see
Appendix A: Detailed test results

Summary charts Exchange backups
and tables for As Figure 10 shows, an Exchange backup to Avamar is running at a rate of 174 MB/s
backups and with data change and no load, and at a rate of 142 MB/s with data change and load.
restores This speed is because of Avamar’s client-side deduplication as well at its multi-
streaming capabilities. For Exchange, EMC did not test for No data change and no
load backup.

AV data change no load (174 MB/s)
AV data change and load (142 MB/s)

06:40

05:26

Figure 10. Exchange backups 3.4 TB per server (hours)

Note During all backup test scenarios, eight vCPUs and 48 GB RAM were allocated
on both Microsoft Exchange mailbox servers and the Microsoft SQL Server.
This is the minimum recommendation for Avamar multi-streaming.


SharePoint backups
As Figure 11 shows, SharePoint backups to Avamar are running at a rate of 176 MB/s
with no data change and no load. The rate decrease for data change and no load to
140 MB/s because of the new data that has to be compared and backed up.

The rate decreases to 113 MB/s for data change with load because less CPU is
available from the SQL Server under load.

AV No data change no load (176 MB/s)

AV data change no load (140 MB/s)

AV data change and load (113 MB/s)

06:40

05:22

04:15

Figure 11. SharePoint backups 2.5 TB (hours)

Exchange restores
Figure 12 shows the Exchange database restores from Avamar. For GLR restore, the
limiting factor is the Exchange application programming interface (API) retrieving
individual mail items, rather than the network.


06:00

04:48

03:36

Hours 02:24

01:12

00:00
AV single DB restore AV GLR single mailbox
(679GB) 35 MB/s restore (832MB)
Restore Times 05:22 01:05

Figure 12. Exchange restore times (hours)

SharePoint restores
Figure 13 shows the SharePoint restores from Avamar. For SharePoint granular
restores, most of the restore time is taken up by Kroll Ontrack PowerControls reading
the ContentDB and log files necessary for the granular restore.

02:09
01:55
01:40
01:26
01:12
Hours

00:57
00:43
00:28
00:14
00:00
AV DB AV list Kroll Kroll
restore (70 restore restore AV single restore
GB) 38 (1520 time from file restore time from
MB/s items) AvFS AvFS
Restore Times 00:31 02:02 02:00 00:10 00:08

Figure 13. SharePoint restore times (hours)


Deduplication measurements
Figure 14 shows the deduplicated data size for Avamar backups in GB. This example
is based on the SharePoint backups. For the first full backup, the data was
deduplicated by 49.56 percent. For the No data change backup, deduplication was
99.98 percent. For the Data change backup, deduplication was 92.07 percent. The
Data change and load backup deduplication was 92.67 percent.

Data size Data size with deduplication

2464.18 2500.3 2520.45 2520.65

1242.87

199.18 184.69
0.47

First Full backup No data change Data change backup Data Change and
backup load

Figure 14. Avamar deduplication for backups (GB)

Detailed Performance testing results
observations EMC made the following observations during and after testing.

Application workload
• The Exchange environment supported 5,000 users at 150 messages per day
• The SharePoint farm supported 6,000 users at 10 percent concurrency
• Hardware load balancing improved SharePoint performance by 3 percent

Backup
• No data change no load backup:
 2.6 TB of SharePoint data to Avamar took 4 hours and 15 minutes
• Data change no load backup (1.5 percent Exchange, 8 percent SharePoint data
change):
 6.9 TB of Exchange data to Avamar took 5 hours and 26 minutes
• Data change with load backup (2 percent Exchange, 10 percent SharePoint
data change):
 6.9 TB of Exchange data to Avamar took 6 hours and 40 minutes


Restores from Avamar
 Exchange single database restore of 679 GB took 5 hours and 22 minutes
 Exchange mailbox restore of 832 MB took 1 hour and 5 minutes
 SharePoint database restore of 70 GB took 31 minutes
 List restore of 1,520 SharePoint items took 2 hours and 2 minutes
 Single file SharePoint restore took 10 minutes

Deduplication rates
• Deduplication rate for first full backup to Avamar:
 Exchange was 1.67:1
 SharePoint was 1.98:1

• Deduplication rate for incremental backups to Avamar:
 Exchange was 81:1 for a 3 percent data change rate
 SharePoint was 12.61:1 for a 10 percent data change rate

Benefits Based on the testing results, these are the main benefits of this solution.
Reduce backup time through Avamar multi-streaming
Backing up terabytes of data took a long time with a traditional, tape-based, backup
solution. For example, when backing up 6.4 TB of data with LTO-5 (maximum 140
MB/s), the most advanced tape technology requires 13 hours for conventional tape
backup, not including backup application time and performance. However, it took
about only 5.5 hours to back up the same amount of Microsoft Exchange data
through 1 Gb Ethernet with Avamar multi-streaming, which means the backup time is
reduced by up to 58 percent. Reduction of backup times compared to traditional
backups is much faster because Avamar deduplicates and sends only unique
changes, thus less data, combined with faster ingests.

Reduce granular recovery time
The EMC Avamar plug-in for the Exchange GLR plug-in provides the capability to
mount an existing backup’s database from the Avamar grid, which makes that
database available on a mount point. This eliminates the time required to restore the
database to a recovery database. Once you mount the database, it can be browsed,
and individual items can be extracted from the database for recovery. The SharePoint
GLR plug-in also mounts the content database using a mount point and eliminates
the need to restore it to a different location before extracting items.

Reduced backup storage requirements
Compared with the traditional 100-percent-backup storage capacity requirement on
tape, only a relatively small amount of additional capacity is required with data
deduplication on Avamar grid.

In this solution, the first full backup to Avamar of 9.76 TB of data used 6.51 TB of disk
space after client-side deduplication. For a subsequent backup of a 3.15 percent data
change under load, 9.85 TB of data used 278.20 GB of disk space after client-side
deduplication.


Note Your results may vary depending on how many duplicates are in your specific
environment.

Notes and When EMC backed up to the Avamar grid, the CPU load on SharePoint SQL Server and
recommendations Exchange mailbox servers were at 90 percent with the recommended eight cores.
Avamar was configured to the maximum of six cores. Microsoft does not recommend
such a high CPU usage, even though it did not affect performance.

Restore times for Avamar were acceptable for both Exchange and SharePoint,
however, Avamar backup times can be improved by adding more cores to the SQL
and Exchange mailbox servers. Avamar backups used almost all of the spare CPU of
the SQL and Exchange mailbox servers but application performance was not affected
by this.

With the advent of Avamar 6.0 and its multi-streaming backup capability, Avamar
backs up and restores everything from small to enterprise-level environments, as
proven in this solution.


Conclusion
Summary EMC Avamar extends the optimization capabilities of EMC backup and recovery
solutions to accelerate performance, enhance replication control, and simplify
administration. Your IT backup administrator can now manage organization-wide
backup workloads associated with each dataset from a single management interface.

The EMC Avamar application plug-ins for Exchange and SharePoint enable
application-consistent backups. Avamar client-side deduplication saves backup
space and minimizes the amount of data that needs to be backed up.

Findings Table 14 to Table 16 summarize the objectives of this solution and the results
achieved.
Table 14. Baseline performance

Objective Results
Validate Exchange 2010 design and layout Exchange 2010 performed at acceptable
by showing acceptable performance levels levels, and storage layout was within
parameters for performance.

Validate SharePoint 2010 design and SharePoint 2010 performed at acceptable
layout by showing acceptable performance levels, and storage layout was within
levels parameters for performance.

Table 15. Hardware load balancing

Objective Results
Show SharePoint farm performance impact Using a load balancer gives a slightly
of a hardware load balancer better performance (3%).

Table 16. Avamar backups

Objective Results
Show backup times for SharePoint 2010 With the recommended eight cores, CPU
and Exchange 2010 to Avamar load on SharePoint SQL Server and
Exchange mailbox servers were at 90%.
Microsoft does not recommend such a
high CPU usage for SQL Server or
Exchange, although it did not affect
application performance in EMC’s tests.

Show deduplication rates for backups to • Initial deduplication rate of 1.67:1 for
Avamar Exchange data
• Initial deduplication rate of 1.98:1 for
SharePoint data
• Subsequent data change deduplication
rate of 85:1 for Exchange data
• Subsequent data change deduplication
rate of 12.61:1 for SharePoint data


Objective Results
Show restore times from Avamar for Restore times for Avamar were acceptable
SharePoint 2010 and Exchange 2010 for both Exchange and SharePoint.

Show how Avamar affects SharePoint and Avamar backups used almost all of the
Exchange spare CPU of the SQL Server and Exchange
mailbox servers, but application
performance was not affected by this.

Show in which environments Avamar is With the advent of Avamar 6.0 and its new
best suited multi-streaming backup capability,
Avamar backs up and restores everything
from small to enterprise-level
environments, as proven in this solution.

For more information on the testing and validation done for this solution, see


References
White papers For additional information, see the white papers listed below.
• EMC Virtual Infrastructure for Microsoft SharePoint Server 2010
• Microsoft Exchange Server 2010 Performance Review Using the EMC VNX5300
Unified Storage Platform—An Architectural Overview

Product For additional information, see the product documents listed below.
documentation
• Avamar 6 Portfolio
• Avamar Data Store Gen4 Multi-Node System Installation Guide
• Avamar 6.0 for Exchange VSS Guide


Testing scenarios for this solution include:
• Baseline application performance testing
• Avamar: Backup testing with no data change and no load
• Combined Avamar backup testing: With data change and without load
• Combined Avamar backup testing: With data change and load
• Avamar test results with encryption
• Restore testing: Avamar Exchange native GLR restore
• Restore testing: Avamar Exchange database recovery
• Restore testing: Avamar SharePoint VSS restore of site collection database
• Avamar restore testing: Kroll recovery of a site and single item in SharePoint
• Hardware load balancing


Baseline Introduction
application EMC ran this test to establish baseline performance and to validate the design and
performance configuration of Exchange and SharePoint.
testing
Baseline test
The objective of this test is to show the baseline performance when both Exchange
applications and SharePoint applications are running at the same time in the
environment. The load profiles are explained in each of the application sections that
follow:

• Microsoft Exchange 2010 Server
• Microsoft SharePoint 2010 Server

EMC ran all the tests for a minimum of eight hours concurrently to simulate a normal
working day.

Notes Benchmark results are highly dependent upon workload, specific application
requirements, and system design and implementation. Relative system
performance will vary as a result of these and other factors. Therefore, you
should not use this workload as a substitute for a specific customer
application benchmark when you are considering critical capacity planning
and/or product evaluation decisions.

All performance data contained in this report was obtained in a rigorously
controlled environment. Results obtained in other operating environments
may vary significantly.

EMC Corporation does not warrant or represent that a user can or will achieve
similar performance expressed in transactions per second.

Exchange 2010 Server
Performance
After EMC completed the storage validation with Jetstress and determined that the
storage was sized correctly and performed as expected, the next step in the
validation process was to use the Microsoft Exchange Server Load Generator
(LoadGen) tool to simulate a MAPI workload against the entire Exchange
infrastructure. LoadGen testing is necessary to determine how each Exchange
component performs under close-to-production user load.

LoadGen requires full deployment of the Exchange environment for validation testing.
You must perform all LoadGen validation testing in an isolated lab environment
where there is no connectivity to production data.

LoadGen:
• Generates users and workloads against the entire Exchange environment,
including network and storage components
• Simulates the entire email flow and locates any bottlenecks in the solution
• Assists in determining the CPU and memory resources that are required to
sustain the load for which the Exchange environment is designed


The 64-bit version of LoadGen 2010 can be downloaded from the Microsoft Download
Center.

Test results
In this solution, LoadGen 2010 was used to simulate Outlook 2007 online mode
mailboxes with the following characteristics:
• The action profile was 150 messages per mailbox per day
• Each mailbox was 1.16 GB in size

Note The building block design in this solution can support a mailbox size up to
1.5 GB. Due to the long time required for mailbox initialization, the mailbox
size configured in LoadGen was 1 GB. The actual mailbox size populated by
LoadGen was around 1.16 GB.

EMC determined the validity of each test run by comparing the results of selected
performance counters to Microsoft-specified criteria. Performance counter data was
collected at one-minute intervals for the duration of each test run. The results of the
first and last hours were discarded. Results were averaged over the remainder of the
test.

For additional information about monitoring Exchange 2010 performance and other
key performance counters, see Microsoft Exchange Server Library topic Performance
and Scalability Counters and Thresholds.
Table 17 shows the LoadGen tests used to measure the performance of the Exchange
infrastructure of this solution.

Table 17. LoadGen tests

Test Description
1 Normal operation: 8 hours, 100 percent concurrency test under normal
operating conditions with 150 messages MAPI profile. The objective was to
validate the entire Exchange environment under normal operating conditions.

2 Mailbox server failure: 8 hours, 100 percent concurrency test during the failure
of one mailbox server and one client access server. All databases become
active on the other mailbox server. All client RPC requests are processed by
one client access server. The objective was to validate the environment's
performance when one mailbox server and one client access server were lost.

As shown in Table 18, the performance achieved by the Exchange virtual machines
meets the target metrics in all test scenarios. In the second test scenario, the
processor utilization on the remaining servers is increased due to the loss of one
mailbox server and one client access server.


Table 18. Performance of virtual machines

Failover
Normal situation
situation
Server Performance counter Target (2,500 users per
(5,000 users on
server)
single server)

Mailbox server ProcessorPercent processor time Less than 80% 38% 59%

MSExchange databaseI/O Less than 20 ms 10 11
database reads (attached)
average latency

MSExchange databaseI/O Less than 20 ms 1 2
database writes (attached)
average latency Less than reads
avg.

MSExchange databaseI/O Less than 200 21 0
database reads (recovery) average ms
latency

MSExchange databaseI/O Less than 200 2 0
database writes (recovery) ms
average latency

MSExchange databaseI/O log Less than 20 ms 3 0
read average latency

MSExchange databaseI/O log Less than 20 ms 3 1
writes average latency

MSExchange Less than 2 1 0
Replication(*)ReplayQueueLengt
h

MSExchangeISRPC requests Less than 70 2 5

MSExchangeISRPC averaged Less than 10ms 3 4
latency

CAS/HUB ProcessorPercent processor time Less than 80% 30% 60%
servers combo
RpcClientAccessRPC Requests

RpcClientAccessRPC Averaged ms
Latency

MSExchangeTransport Less than 3,000 1 7
Queues(_total)Aggregate
Delivery Queue Length (All
Queues)

MSExchangeTransport Less than 250 0 0
Queues(_total)Active Remote
Delivery Queue Length


Failover
Normal situation
situation
Server Performance counter Target (2,500 users per
(5,000 users on
server)
single server)

MSExchangeTransport Less than 250 1 7
Queues(_total)Active Mailbox
Delivery Queue Length

Figure 15 shows the throughput on Exchange active database LUNs. LoadGen
simulated user actions were the main contributor to the activities on these LUNs. The
average IOPS was 58.75, while the maximum was 126.66.

140

120

100 Exchange MB1 DB_1
Exchange MB1 DB_3
Exchange MB1 DB_5
80
Exchange MB1 DB_7
IOPS

Exchange MB1 DB_9
60
Exchange MB2 DB_2
Exchange MB2 DB_4
40
Exchange MB2 DB_6
Exchange MB2 DB_8

0
03:01:16
03:11:16
03:21:16
03:31:16
03:41:16
03:51:16
04:01:16
04:11:16
04:21:16
04:31:16
04:41:16
04:51:16
05:01:16

Figure 15. Exchange active database LUN throughput


Figure 16 shows the throughput on Exchange passive database LUNs. Exchange DAG
replication was the main contributor to the activities on these LUNs. The average IOPS
was 45.45, while the maximum was 58.15.

100

90


Exchange MB1 DB_6
60
Exchange MB1 DB_8
IOPS

50
Exchange MB1 DB_10
40
Exchange MB2 DB_1

0 Exchange MB2 DB_9
03:01:16
03:11:16
03:21:16
03:31:16
03:41:16
03:51:16
04:01:16
04:11:16
04:21:16
04:31:16
04:41:16
04:51:16
05:01:16
Figure 16. Exchange passive database LUN throughput

Figure 17 shows the throughput on all Exchange log LUNs. The average IOPS was
36.24.

50
45
40
35
30
IOPS

25
20
15
10
5
0
03:01:16
03:06:16
03:11:16
03:16:16
03:21:16
03:26:16
03:31:16
03:36:16
03:41:16
03:46:16
03:51:16
03:56:16
04:01:16
04:06:16
04:11:16
04:16:16
04:21:16
04:26:16
04:31:16
04:36:16
04:41:16
04:46:16
04:51:16
04:56:16
05:01:16

Figure 17. Exchange log LUN throughput


In summary, all key counters were well within the recommended thresholds. The
results met the design target.

SharePoint 2010 Server
Introduction
EMC completed the SharePoint farm baseline test with the aim of a reasonable user
count to establish a baseline for backup testing.

Objective
The objective of this test was to have a baseline to compare the impact of backups
and load balancing on the environment.

Setup
The setup is explained in the configuration and deployment sections.

Test results
The SharePoint farm averaged 11.43 passed tests per second, as shown in Figure 18.
This is equivalent to 6,858 users at 10 percent concurrency. The load profile was 80
percent browse, 10 percent search, and 10 percent modify.

16
14
12
10
8
6 Passed Tests/Sec
4
2
0
02:00:00
02:05:45
02:11:30
02:17:15
02:23:00
02:28:45
02:34:30
02:40:15
02:46:00
02:51:45
02:57:30
03:03:15
03:09:00
03:14:45
03:20:30
03:26:15
03:32:00
03:37:45
03:43:30
03:49:15
03:55:00

Figure 18. SharePoint farm performance


Browse averaged 2.34 seconds, search averaged 1.01 seconds, and modify averaged
2.31 seconds, as shown in Figure 19. The recommended Microsoft limit for common
operations is three seconds; these times were well within that limit.

6

5

4
Seconds
3 Avg. Test Time, Browse

2 Avg. Test Time, Search
Avg. Test Time, Modify
1

0
02:00:00
02:07:15
02:14:30
02:21:45
02:29:00
02:36:15
02:43:30
02:50:45
02:58:00
03:05:15
03:12:30
03:19:45
03:27:00
03:34:15
03:41:30
03:48:45
03:56:00
Figure 19. SharePoint average test times

Figure 20 shows that the CPUs on the WFE servers were very busy averaging at 77.07
percent, and the SQL Server CPU averaging 50.40 percent.

Index crawls were not running during this test.

100
90
80
70
60 TCE-SP-SQL01
Percent

50 TCE-SP-INDEX01
40 TCE-SP-WFE01
30
TCE-SP-WFE02
20
TCE-SP-APP-01
10
0
02:00:00
02:06:30
02:13:00
02:19:30
02:26:00
02:32:30
02:39:00
02:45:30
02:52:00
02:58:30
03:05:00
03:11:30
03:18:00
03:24:30
03:31:00
03:37:30
03:44:00
03:50:30
03:57:00

Figure 20. SharePoint farm CPU usage

The ContentDB LUNs and log LUNs were spread over three VMFS file systems on three
LUNs. The Search and Temp DBs were on a single VMFS file system.


Figure 21 shows the IOPS for the ContentDB, log, Search, and Temp DB LUNs. The
ContentDBs_3 LUN has a lower IOPS because it hosted a smaller number of
ContentDB LUNs. IOPS averaged 155.16, while the maximum IOPS was 411.93.

450
400
350
300
250 ContentDBs_1(IO/s)
IOPS

100 Search and Temp(IO/s)

50
0
02:01:16
02:11:16
02:21:16
02:31:16
02:41:16
02:51:16
03:01:16
03:11:16
03:21:16
03:31:16
03:41:16
03:51:16
04:01:16
Figure 21. SharePoint ContentDB LUN throughput


Avamar: Backup Introduction
testing with no Backup SharePoint datasets to the grid using the relevant plug-ins without any user
data change and load and without data change on SharePoint.
no load
Objective
This test showed the effect of backup to Avamar with no data change and no load to
set a baseline for backup testing.

Setup
Setup is as per the configuration and deployment sections. EMC did a full initial
backup before backup testing to Avamar.

SharePoint 2010 Server: Test results
The backup time for this test was 4 hours and 15 minutes. This is an average backup
rate of 171 MB/s for 2.5 TB of SharePoint data. There was very little data to
deduplicate because there was no data change.

Figure 22 shows that the SQL server CPU averaged 80.47 percent. This is because the
Avamar client performs CPU-intensive source hashing and deduplication during the
backup.

100
90
80
70
60
Percent

50
40
TCE-SP-SQL01
30
20
10
0
10:49:35
11:03:20
11:17:05
11:30:50
11:44:35
11:58:20
12:12:05
12:25:50
12:39:35
12:53:20
13:07:05
13:20:50
13:34:35
13:47:20
13:59:35
14:13:20
14:27:05
14:40:50
14:54:35

Figure 22. SQL Server CPU usage


Figure 23 shows that the read IOPS for the ContentDBs averaged 200.06, while the
maximum was 743.08. As you can see from the figure different ContentDBs on the
three LUNs were being backed up at different times.

800
700
600

IOPS 500
400 ContentDBs_1
300 ContentDBs_2

200 ContentDBs_3

100
0
04:51:16
05:06:16
05:16:16
05:31:16
05:46:16
06:01:16
06:16:16
06:31:16
06:46:16
07:01:16
07:16:16
07:31:16
07:46:16
08:01:16
08:16:16
08:31:16
08:46:16
09:01:16
09:16:16
Figure 23. ContentDB LUNs read throughput

Figure 24 shows the disk utilization of two of the disks in the storage pool where the
ContentDBs and logs reside. The average utilization was 8.42 percent.

100
90
80
70
60
Percent

50 Bus 1 Enclosure 1 Disk 0
[Disk]
40
Bus 1 Enclosure 1 Disk 4
30
[Disk]
20
10
0
04:51:16
05:08:46
05:26:16
05:46:16
06:06:16
06:26:16
06:46:16
07:06:16
07:26:16
07:46:16
08:06:16
08:26:16
08:46:16
09:06:16
09:26:16

Figure 24. ContentDB and log LUN disk utilization


Combined Avamar Introduction
backup testing: Backup SharePoint and Exchange datasets to the grid using the relevant plug-ins
With data change without any user load on SharePoint and Exchange.
and without load
Objective
The objective of this test scenario was to validate daily backup performance in a
dedicated backup window when choosing Avamar grid as the backup target.

Setup
Eight-hour LoadGen and VSTS tools were running before each backup test to simulate
daily change in the environment. This change would be approximately 3 percent for
Exchange data and 10 percent for SharePoint data.

Exchange backups were performed on passive database copies from both mailbox
servers. Both Exchange backup jobs and SharePoint backup jobs were run with six
streams to the Avamar nodes. Exchange backup jobs and SharePoint backup jobs
were launched simultaneously.

Exchange 2010 Server: Test results
Figure 25 shows the CPU usage on Exchange mailbox server during the backup. The
average value was 82.3 percent.

100
90
80
70
60
50
40
30
20
10
0
11:20:18
11:25:18
11:30:18
11:35:18
11:40:18
11:45:18
11:50:18
11:55:18
12:00:18
12:05:18
12:10:18
12:15:18
12:20:18
12:25:18
12:30:18
12:35:18
12:39:18
12:44:18
12:49:18
12:54:18
12:59:18
13:04:18
13:09:18
13:14:18
13:19:18

Figure 25. Exchange CPU usage


Figure 26 shows the throughput on passive database LUNs from both Exchange
servers. Backup jobs were the main contributor to the activities on these LUNs. The
average IOPS was 148.32.

200
180
Exchange MB1 DB_2
160
Exchange MB1 DB_4
140
Exchange MB1 DB_6
120
Exchange MB1 DB_8
IOPS

100
Exchange MB1 DB_10
80
Exchange MB2 DB_1
60
Exchange MB2 DB_3
40
Exchange MB2 DB_5
20
Exchange MB2 DB_7
0 Exchange MB2 DB_9
07:01:16
07:11:16
07:21:16
07:31:16
07:41:16
07:51:16
08:01:16
08:11:16
08:21:16
08:31:16
08:41:16
08:51:16
09:01:16
Figure 26. Exchange passive DB LUN read throughput

rate of 171 MB/s for 3.2 TB of Exchange data. The deduplication ratio was 85:1.

Note The test results of deduplication ratio are based on the data generated by
LoadGen. Because LoadGen is not specialized to test deduplication ratio, the
results provided in this section may differ from the actual data in a customer
production environment.

These results show SharePoint farm performance with a data change rate of 8 percent
and without any user load.

Since there is no user load the SharePoint application has no performance results.

rate of 136 MB/s for 2.5 TB of Exchange data. The deduplication ratio was 12.61:1.

Figure 27 shows the SQL Server CPU usage averaging 80.28 percent. This is very
similar to SQL Server CPU usage for no data change, since the Avamar client has to do
the same hashing and deduplication each time before comparing the hashes to the
backup data.


90
80
70
60

Percent
50
40
TCE-SP-SQL01
30
20
10
0
13:20:14
13:26:29
13:32:44
13:38:59
13:45:14
13:51:29
13:57:44
14:03:59
14:10:14
14:16:29
14:22:44
14:28:59
14:35:14
14:41:29
14:47:44
14:53:59
15:00:14
15:06:29
15:12:44
15:18:59

As shown in Figure 28, read IOPS averaged 207.91. The drop at the end for the
ContentDBs_2 LUN shows the end of its backup while ContentDBs_3 is spiking as its
ContentDBs are backed up.

700

600

500

400
IOPS

ContentDBs_1
300 ContentDBs_2
ContentDBs_3
200

100

0
06:01:16
06:11:16
06:21:16
06:31:16
06:41:16
06:51:16
07:01:16
07:11:16
07:21:16
07:31:16
07:41:16
07:51:16
08:01:16


Read size averaged 227.74 KB and peaked at 256 KB. Avamar decides what the best
segment size is for the backup based on the content size.


Combined Avamar Introduction
backup testing: Backup SharePoint and Exchange datasets to the grid using the relevant plug-ins with
With data change predefined data change rates and user loads for SharePoint and Exchange.
and load
Objective
The objective of this test scenario was to validate daily backup performance with data
change and normal workload on Exchange and SharePoint servers when choosing
Avamar grid as the backup target.

Setup
Eight-hour LoadGen and VSTS tools were running before each backup test to simulate
daily change in the environment.

Exchange backups were performed on passive database copies from both mailbox
servers. Both Exchange backup jobs and SharePoint backup jobs were run with six
streams to the Avamar nodes. Exchange backup jobs and SharePoint backup jobs
were launched simultaneously.

LoadGen and VSTS were running throughout the entire backup window to evaluate
the impact to both backup jobs and end-user experience.

Figure 29 shows the CPU usage on Exchange mailbox servers during the backup. The
average value was 90.1 percent.

100
90
80
70
60
50
40
30
20
10
0
10:00:48
8:00:48
8:05:48
8:10:48
8:15:48
8:20:48
8:25:48
8:30:48
8:35:48
8:40:48
8:45:48
8:50:48
8:55:48
9:00:48
9:05:48
9:10:48
9:15:48
9:20:48
9:25:48
9:30:48
9:35:48
9:40:48
9:45:48
9:50:48
9:55:48

Figure 29. Exchange CPU usage


Table 19 shows the key Exchange performance counters that can reflect the end-user
experience during the backup window. Although the processor time exceeded the
recommended threshold, all other key counters were well below the target, which
means the end-user experience was not affected during backup.

Table 19. Key Exchange performance counters

AV backup
Performance counter Target
under load
ProcessorPercent processor time Less than 80% 90%

MSExchange databaseI/O database reads (attached) Less than 20 ms 17
average latency

MSExchange databaseI/O database writes (attached) Less than 20 ms 2
average latency
Less than reads avg.

MSExchange databaseI/O database reads (recovery) Less than 200 ms 40
average latency

MSExchange databaseI/O database writes (recovery) Less than 200 ms 15
average latency

MSExchange databaseIO log read average latency Less than 20 ms 3

MSExchange databaseIO log writes average latency Less than 20 ms 6

MSExchange Replication(*)ReplayQueueLength Less than 2 1

MSExchangeISRPC requests Less than 70 2

MSExchangeISRPC averaged latency Less than 10 ms 3


Figure 30 shows the disk utilization for the storage pool that the mailbox databases
used. It is an indication of how busy the backend disks are. Both LoadGen simulated
user actions and backup jobs contribute to the activities in the storage pool.
Utilization averaged 71.36 percent.

100

90

80

70
60 [Disk]
Axis Title

[Disk]
40
30 [Disk]

[Disk]
10

0
04:01:17
04:11:17
04:21:17
04:31:17
04:41:17
04:51:17
05:01:17
05:11:17
05:21:17
05:31:17
05:41:17
05:51:17
06:01:17
Figure 30. Exchange database disk utilization


Figure 31 shows the throughput on active database LUNs. LoadGen simulated user
actions were the main contributor to the activities on these LUNs. The average IOPS
was 42.45, while the maximum was 111.39.

120

Exchange MB1 DB_3
80
Exchange MB1 DB_5
Exchange MB1 DB_7
IOPS

60
Exchange MB1 DB_9
Exchange MB2 DB_2
40
Exchange MB2 DB_4

Exchange MB2 DB_8
04:01:17
04:11:17
04:21:17
04:31:17
04:41:17
04:51:17
05:01:17
05:11:17
05:21:17
05:31:17
05:41:17
05:51:17
06:01:17
Figure 31. Exchange active database LUN read throughput

Figure 32 shows the throughput on Exchange passive database LUNs. Backup jobs
were the main contributor to the activities on these LUNs. The average IOPS was
142.51, while the maximum was 172.33.

200
180
Exchange MB1 DB_2
160
Exchange MB1 DB_4
140
Exchange MB1 DB_6
120
Exchange MB1 DB_8
IOPS

100
Exchange MB1 DB_10
80
Exchange MB2 DB_1
60
Exchange MB2 DB_3
40
Exchange MB2 DB_5
20
Exchange MB2 DB_7
0
Exchange MB2 DB_9
04:01:17
04:11:17
04:21:17
04:31:17
04:41:17
04:51:17
05:01:17
05:11:17
05:21:17
05:31:17
05:41:17
05:51:17
06:01:17

Figure 32. Exchange passive database LUN read throughput


The average read size was 213.25 KB. The large I/O was used to improve backup
performance.

Wrote IOPS for the active databases averaged 17.20 because of load and for the
passive databases averaged 35.74 because of the backups running.

rate of 144 MB/s for 3.3 TB of Exchange data. The deduplication ratio was 78:1.

These results show SharePoint farm performance with a data change rate of 10
percent and with full user load.

SharePoint performance was acceptable even though the backup was running.

rate of 109 MB/s for 2.5 TB of Exchange data. The deduplication ratio was 13.65:1.

As shown in Figure 33, the SharePoint farm averaged 11.61passed test/sec, which is
equivalent to 6,966 users at 10 percent concurrency. This is lower than the baseline
because of the increased load of Avamar backups.

16
14
12
10
8
6 Passed Tests/Sec
4
2
0
02:00:00
02:06:15
02:12:00
02:17:45
02:23:30
02:29:15
02:35:00
02:40:45
02:46:30
02:52:15
02:58:00
03:03:45
03:09:30
03:15:15
03:21:00
03:26:45
03:32:30
03:38:15
03:44:00
03:49:45
03:55:30



As shown in Figure 34, average browse was 2.24 seconds, search was 0.89 seconds,
and modify was 2.88 seconds.

7

6

5

4
Seconds Avg. Test Time, Browse
3
Avg. Test Time, Search
2
1

0
02:00:00
02:07:45
02:15:00
02:22:15
02:29:30
02:36:45
02:44:00
02:51:15
02:58:30
03:05:45
03:13:00
03:20:15
03:27:30
03:34:45
03:42:00
03:49:15
03:56:30
Figure 34. Average test times

As shown in Figure 35, SQL Server CPU averaged 87.23 percent, which was 7 percent
higher than the no load tests. This extra load was caused by user load and Avamar
backup.

100
90
80
70
60
Percent

50
40 % Processor Time
30
20
10
0
08:59:16
09:06:16
09:13:16
09:20:16
09:27:16
09:34:16
09:41:16
09:48:16
09:55:16
10:02:16
10:09:16
10:16:16
10:23:16
10:30:16
10:37:16
10:44:16
10:51:16
10:58:16



WFE CPU usage averaged 75.57 percent, which is slightly lower than the baseline
load test, as shown in Figure 36. This is because the SQL Server CPU is much busier,
so the WFEs are no longer the bottleneck.

90
80
70
60
Percent
50
40 TCE-SP-WFE01
30 TCE-SP-WFE02
20
10
0
02:00:00
02:06:15
02:12:30
02:18:45
02:25:00
02:31:15
02:37:30
02:43:45
02:50:00
02:56:15
03:02:30
03:08:45
03:15:00
03:21:15
03:27:30
03:33:45
03:40:00
03:46:15
03:52:30
03:58:45
Figure 36. WFE CPU usage

As shown in Figure 37, utilization for disks in the ContentDB LUN RAID groups
averaged 44.46 percent, which shows that the back-end disks in the storage pool
could handle more load.

100
90
80
70
60
Percent

50
40
30
20
10
0
04:01:17
04:11:17
04:21:17
04:31:17
04:41:17
04:51:17
05:01:17
05:11:17
05:21:17
05:31:17
05:41:17
05:51:17
06:01:17

Figure 37. ContentDB and log disk utilization


Figure 38 shows that read IOPS averaged 431.93 and peaked at 1,012.84. The
busiest ContentDB LUN is the one which has the most ContentDBs being actively
backed up.

1200

1000

IOPS 800

600 ContentDBs_1
ContentDBs_2
400
ContentDBs_3

200

0
04:01:17
04:11:17
04:21:17
04:31:17
04:41:17
04:51:17
05:01:17
05:11:17
05:21:17
05:31:17
05:41:17
05:51:17
06:01:17

The read size varied because of Avamar and SharePoint reading at the same time.


Avamar test Exchange
results with As shown in Table 20, encryption setting has little impact on Exchange backup time.
encryption
Table 20. Exchange encryption settings

Encryption Backup time CPU usage
MB01 MB02 MB01 MB02

No encryption 5:46 5:38 83% 83%

High encryption 5:30 5:30 83% 83%

When EMC enabled high encryption, the backup time on Exchange servers were
slightly reduced (MB01 5 percent, MB02 2 percent). The CPU usages were almost the
same.

SharePoint
As shown in Table 21, encryption setting has little impact (2 percent) on SharePoint
backup time.

Table 21. SharePoint encryption settings

Encryption Backup time SQL Server CPU
No Encryption 4:15 87%

High Encryption 4:21 88%

Avamar performance with encryption
The above differences can be caused by variation between tests. Therefore, you can
consider the performance with encryption to be almost the same as with no
encryption.


Restore testing: Introduction
Avamar Exchange EMC used Avamar client for Individual exchange mailbox and message restores.
native GLR restore
Objective
The objective of this test scenario was to validate the granular-level recovery function
through the EMC Avamar plug-in for the Exchange GLR plug-in.

Setup
In this test, EMC used the EMC Avamar plug-in for the Exchange GLR plug-in to restore
one individual Exchange mailbox.

Note For the detailed configurations required to enable message level restores
(granular level recoveries), see Avamar 6.0 for Exchange VSS Guide.

The EMC Avamar plug-in for Exchange GLR plug-in provides the capability to mount an
existing backup’s content database through Avamar File System, which makes that
database available on a mount point. This eliminates the time required to restore the
database to a recovery database. Once the database is mounted, it can be browsed,
and individual items can be extracted from the database for recovery.

Figure 39 shows the Avamar GUI for GLR restore.

Figure 39. Avamar GUI for mailbox restore

The restore time for a single mailbox was 1 hour and 5 minutes. It restored 36,494
mail items with 832 MB in total size.


Avamar/Exchange EMC used Avamar client for individual Exchange mailbox database restores.
database recovery
Objective
The objective of this test scenario was to validate the Exchange mailbox database
recovery function.

Setup
In this test, EMC used the EMC Avamar Plug-in for Exchange VSS to restore one
Exchange database and replay all transaction logs.

Figure 40 shows the Avamar GUI for an Exchange database restore.

Figure 40. Avamar GUI for Exchange database restore

The restore time for this test was 5 hours and 22 minutes. This is an average restore
rate of 36 MB/s for 678.7 GB of Exchange data.


Avamar/ EMC used the EMC Avamar Plug-in for SharePoint VSS to restore a content database
SharePoint VSS (site collection).
restore of site
collection Objective
database The objective of this test scenario was to show the Avamar restore of a content
database in an existing SharePoint farm.

Setup
In this test, EMC used Avamar to restore directly back onto the SharePoint SQL server.
The original ContentDB was removed from the SQL Server and deleted before the
restore.

Test results
Figure 41 shows the Avamar GUI for restoring a single ContentDB.

Figure 41. Avamar GUI restoring a ContentDB

The restore time for this test was 31 minutes. This is an average restore rate of 38
MB/s for 70 GB of SharePoint data.

The ContentDB was restored onto the SQL Server in the original location and then was
automatically added back to SQL Server. If the site collection was not deleted in
SharePoint, then no user interaction was required besides running the restore.


Avamar restore Introduction
testing: Kroll EMC used Kroll Ontrack PowerControls against a mounted virtual saveset to recover a
recovery of a site site, list, and single item in SharePoint.
and single item in
SharePoint Objective
The objective of this test scenario was to restore a site and a single item back into a
SharePoint farm.

Setup
The site and item were deleted from SharePoint before the restores were run.

List restore
Figure 42 and Figure 43 show the Avamar GUI for doing a SharePoint GLR restore and
the Kroll Ontrack PowerControls GUI for restoring the list.

Figure 42. SharePoint GLR restore


Figure 43. Restoring a list

The restore time for this test was 2 hours and 2 minutes. The Kroll Ontrack
PowerControls restore time was 2 hours. The test restored a total of 1,520 items of
data.


Single-item restore
Figure 44 shows Kroll Ontrack PowerControls restoring a single item.

Figure 44. Restoring a single item

The restore time for this test was 10 minutes. The Kroll restore time was 8 minutes.
The test restored one item of data in total.


Hardware load Introduction
balancing Most customer deployments of SharePoint with multiple web front ends use a load
balancer to spread the client load equally and to add redundancy in case of WFE
failure.

For this solution, EMC tested the performance impact of using a hardware load
balancer on the environment. The load balancer only supported 1 Gb connections.

Objective
The objective of this test was to show the value and impact of using hardware load
balancers for SharePoint farms.

Setup
As shown in Figure 45, the load balancer is on the IP network and receives the
requests from the clients before distributing them equally between the WFE servers.

Load balancing logical design
Figure 45 shows the logical network design for the load balancer.

Figure 45. Logical load balancing design


Test results
As shown in Figure 46, passed test per second under load averaged 11.78, which is
equivalent to 7,068 users at 10 percent concurrency. This is a slight (3 percent)
improvement on the baseline test.

16
14
12
10
8
6 Passed Tests/Sec
4
2
0
01:00:00
01:06:45
01:13:30
01:20:15
01:27:00
01:33:45
01:40:30
01:47:15
01:54:00
02:00:45
02:07:30
02:14:15
02:21:00
02:27:45
02:34:30
02:41:15
02:48:00
02:54:45

As shown in Figure 47, browse averaged 2.26 seconds, search averaged 0.93
seconds, and modify averaged 2.35 seconds.

5
4.5
4
3.5
3
Seconds

2.5 Avg. Test Time, Browse
2 Avg. Test Time, Search
1.5
1
0.5
0
01:00:00
01:08:45
01:17:30
01:26:15
01:35:00
01:43:45
01:52:30
02:01:15
02:10:00
02:18:45
02:27:30
02:36:15
02:45:00
02:53:45

Figure 47. Average test times


White Paper: EMC Backup and Recovery for Microsoft Exchange and SharePoint 2010 Servers - EMC Avamar, EMC VNX, VMware vCenter

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to White Paper: EMC Backup and Recovery for Microsoft Exchange and SharePoint 2010 Servers - EMC Avamar, EMC VNX, VMware vCenter (20)

More from EMC (20)

Recently uploaded (20)

White Paper: EMC Backup and Recovery for Microsoft Exchange and SharePoint 2010 Servers - EMC Avamar, EMC VNX, VMware vCenter