Storage for Virtual Environments 2011 R2

Storage for Virtual EnvironmentsStephen FoskettFoskett Services and Gestalt ITLive Footnotes:@Sfoskett

#VirtualStorageThis is Not a Rah-Rah Session

Introducing the Virtual Data Center

This Hour’s Focus:What Virtualization DoesIntroducing storage and server virtualizationThe future of virtualizationThe virtual datacenterVirtualization confounds storageThree pillars of performanceOther issuesStorage features for virtualizationWhat’s new in VMware

Virtualization of Storage, Serverand NetworkStorage has been stuck in the Stone Age since the Stone Age!Fake disks, fake file systems, fixed allocationLittle integration and no communicationVirtualization is a bridge to the futureMaintains functionality for existing appsImproves flexibility and efficiency

Server Virtualization is On the RiseData: InformationWeek Analytics 2010 Virtualization Management Survey of 316 business technology professionals, August 2010

Server Virtualization is a Pile of Lies!What the OS thinks it’s running on…What the OS is actually running on…Physical HardwareVMkernelBinary Translation, Paravirtualization, Hardware AssistGuest OSVMGuest OSVMScheduler and Memory AllocatorvNICvSwitchNIC DrivervSCSI/PVVMDKVMFSI/O Driver

And It Gets Worse Outside the Server!

The Virtual Data Center of TomorrowManagementApplicationsThe Cloud™ApplicationsLegacyApplicationsApplicationsApplicationsCPUNetworkBackupStorage

The Real Future of IT InfrastructureOrchestration Software

Three Pillars of VM Performance

Confounding Storage PresentationStorage virtualization is nothing new…RAID and NAS virtualized disksCaching arrays and SANs masked volumesNew tricks: Thin provisioning, automated tiering, array virtualizationBut, we wrongly assume this is where it endsVolume managers and file systemsDatabasesNow we have hypervisors virtualizing storageVMFS/VMDK = storage array?Virtual storage appliances (VSAs)

Begging for Converged I/O4G FC Storage1 GbE Network1 GbE ClusterHow many I/O ports and cables does a server need?Typical server has 4 ports, 2 usedApplication servers have 4-8 ports used!Do FC and InfiniBand make sense with 10/40/100 GbE?When does commoditization hit I/O?Ethernet momentum is unbeatableBlades and hypervisors demand greater I/O integration and flexibilityOther side of the coin – need to virtualize I/O

Driving Storage VirtualizationServer virtualization demands storage featuresData protection with snapshots and replicationAllocation efficiency with thin provisioning+Performance and cost tweaking with automated sub-LUN tieringImproved locking and resource sharingFlexibility is the big oneMust be able to create, use, modify and destroy storage on demandMust move storage logically and physicallyMust allow OS to move too

“The I/O Blender” Demands New ArchitecturesShared storage is challenging to implementStorage arrays “guess” what’s coming next based on allocation (LUN) taking advantage of sequential performanceServer virtualization throws I/O into a blender – All I/O is now random I/O!

Server Virtualization Requires SAN and NASServer virtualization has transformed the data center and storage requirementsVMware is the #1 driver of SAN adoption today!60% of virtual server storage is on SAN or NAS86% have implemented some server virtualizationServer virtualization has enabled and demanded centralization and sharing of storage on arrays like never before!Source: ESG, 2008

Keys to the Future For Storage FolksYe Olde Seminar Content!

Primary Production Virtualization PlatformData: InformationWeek Analytics 2010 Virtualization Management Survey of 316 business technology professionals, August 2010

Storage Features for Virtualization

Which Features Are People Using?Source: VirtualGeek.typepad.com 2010 virtualization survey of 125 readers

What’s New in vSphere 4 and 4.1VMware vSphere 4 (AKA ESX/ESXi 4) is a major upgrade for storageLots of new features like thin provisioning, PSA, any-to-any Storage VMotion, PVSCSIMassive performance upgrade (400k IOPS!)vSphere 4.1 is equally huge for storageBoot from SANvStorage APIs for Array Integration (VAAI)Storage I/O control (SIOC)

What’s New in vSphere 5VMFS-5 – Scalability and efficiency improvementsStorage DRS – Datastore clusters and improved load balancingStorage I/O Control – Cluster-wide and NFS supportProfile-Driven Storage – Provisioning, compliance and monitoringFCoE Software InitiatoriSCSI Initiator GUIStorage APIs – Storage Awareness (VASA)Storage APIs – Array Integration (VAAI 2) – Thin Stun, NFS, T10Storage vMotion - Enhanced with mirror modevSphere Storage Appliance (VSA)vSphere Replication – New in SRM

And Then, There’s VDI…Virtual desktop infrastructure (VDI) takes everything we just worried about and amplifies it:Massive I/O crunchesHuge duplication of dataMore wasted capacityMore user visibilityMore backup trouble

What’s nextVendor Showcase and Networking Break

Technical Considerations - Configuring Storage for VMsThe mechanics of presenting and using storage in virtualized environments

This Hour’s Focus:Hypervisor Storage FeaturesStorage vMotionVMFSStorage presentation: Shared, raw, NFS, etc.Thin provisioningMultipathing (VMware Pluggable Storage Architecture)VAAI and VASAStorage I/O control and storage DRS

Storage vMotionIntroduced in ESX 3 as “Upgrade vMotion”ESX 3.5 used a snapshot while the datastore was in motionvSphere 4 used changed-block tracking (CBT) and recursive passesvSphere 5 Mirror Mode mirrors writes to in-progress vMotions and also supports migration of vSphere snapshots and Linked ClonesCan be offloaded for VAAI-Block (but not NFS)

vSphere 5: What’s New in VMFS 5Max VMDK size is still 2 TB – 512 bytesVirtual (non-passthru) RDM still limited to 2 TBMax LUNs per host is still 256

Hypervisor Storage Options:Shared StorageThe common/ workstation approachVMware: VMDK image in VMFS datastoreHyper-V: VHD image in CSV datastoreBlock storage (direct or FC/iSCSI SAN)Why?Traditional, familiar, common (~90%)Prime features (Storage VMotion, etc)Multipathing, load balancing, failover*But…Overhead of two storage stacks (5-8%)Harder to leverage storage featuresOften shares storage LUN and queueDifficult storage managementVMHostGuestOSVMFSVMDKDAS or SANStorage

Hypervisor Storage Options:Shared Storage on NASSkip VMFS and use NASNFS or SMB is the datastoreWow!Simple – no SANMultiple queuesFlexible (on-the-fly changes)Simple snap and replicate*Enables full VmotionLink aggregation (trunking) is possibleBut…Less familiar (ESX 3.0+)CPU load questionsLimited to 8 NFS datastores (ESX default)Snapshot consistency for multiple VMDKVMHostGuestOSNASStorageVMDK

Hypervisor Storage Options:Guest iSCSISkip VMFS and use iSCSI directlyAccess a LUN just like any physical serverVMware ESX can even boot from iSCSI!Ok…Storage folks love it!Can be faster than ESX iSCSIVery flexible (on-the-fly changes)Guest can move and still access storageBut…Less common to VM folksCPU load questionsNo Storage VMotion (but doesn’t need it)VMHostGuestOSiSCSIStorageLUN

Hypervisor Storage Options:Raw Device Mapping (RDM)Guest VM’s access storage directly over iSCSI or FCVM’s can even boot from raw devicesHyper-V pass-through LUN is similarGreat!Per-server queues for performanceEasier measurementThe only method for clusteringSupports LUNs larger than 2 TB (60 TB passthru in vSphere 5!)But…Tricky VMotion and dynamic resource scheduling (DRS)No storage VMotionMore management overheadLimited to 256 LUNs per data centerVMHostGuestOSI/OMapping FileSAN Storage

Hypervisor Storage Options:Direct I/OVMware ESX VMDirectPath - Guest VM’s access I/O hardware directlyLeverages AMD IOMMU or Intel VT-dGreat!Potential for native performanceJust like RDM but better!But…No VMotion or Storage VMotionNo ESX fault tolerance (FT)No ESX snapshots or VM suspendNo device hot-addNo performance benefit in the real world!VMHostGuestOSI/OMapping FileSAN Storage

Which VMware Storage Method Performs Best?Mixed random I/OCPU cost per I/OVMFS,RDM (p), or RDM (v)Source: “Performance Characterization of VMFS and RDM Using a SAN”, VMware Inc.,ESX 3.5, 2008

vSphere 5: Policy or Profile-Driven StorageAllows storage tiers to be defined in vCenter based on SLA, performance, etc.Used during provisioning, cloning, Storage vMotion, Storage DRSLeverages VASA for metrics and characterizationAll HCL arrays and types (NFS, iSCSI, FC)Custom descriptions and tagging for tiersCompliance status is a simple binary report

Native VMware Thin ProvisioningVMware ESX 4 allocates storage in 1 MB chunks as capacity is usedSimilar support enabled for virtual disks on NFS in VI 3Thin provisioning existed for block, could be enabled on the command line in VI 3Present in VMware desktop productsvSphere 4 fully supports and integrates thin provisioningEvery version/license includes thin provisioningAllows thick-to-thin conversion during Storage VMotionIn-array thin provisioning also supported (we’ll get to that…)

Four Types of VMware ESX VolumesNote: FT is not supportedWhat will your array do? VAAI helps…Friendly to on-array thin provisioning

Storage Allocation and Thin ProvisioningVMware tests show no performance impact from thin provisioning after zeroing

Pluggable Storage Architecture:Native MultipathingVMware ESX includes multipathing built inBasic native multipathing (NMP) is round-robin fail-over only – it will not load balance I/O across multiple paths or make more intelligent decisions about which paths to usePluggable Storage Architecture (PSA)VMware NMPThird-Party MPPVMware SATPThird-Party SATPVMware PSPThird-Party PSP

Pluggable Storage Architecture: PSP and SATPvSphere 4 Pluggable Storage Architecture allows third-party developers to replace ESX’s storage I/O stackESX Enterprise+ OnlyThere are two classes of third-party plug-ins:Path-selection plug-ins (PSPs) optimize the choice of which path to use, ideal for active/passive type arraysStorage array type plug-ins (SATPs) allow load balancing across multiple paths in addition to path selection for active/active arraysEMC PowerPath/VE for vSphere does everything

Storage Array Type Plug-ins (SATP)ESX native approachesActive/PassiveActive/ActivePseudo ActiveStorage Array Type Plug-InsVMW_SATP_LOCAL – Generic local direct-attached storageVMW_SATP_DEFAULT_AA – Generic for active/active arraysVMW_SATP_DEFAULT_AP – Generic for active/passive arraysVMW_SATP_LSI – LSI/NetApp arrays from Dell, HDS, IBM, Oracle, SGIVMW_SATP_SVC – IBM SVC-based systems (SVC, V7000, Actifio)VMW_SATP_ALUA – Asymmetric Logical Unit Access-compliant arraysVMW_SATP_CX – EMC/Dell CLARiiON and Celerra (also VMW_SATP_ALUA_CX)VMW_SATP_SYMM – EMC Symmetrix DMX-3/DMX-4/VMAX, InvistaVMW_SATP_INV – EMC Invista and VPLEXVMW_SATP_EQL – Dell EqualLogic systemsAlso, EMC PowerPath and HDS HDLM and vendor-unique plugins not detailed in the HCL

Path Selection Plug-ins (PSP)VMW_PSP_MRU – Most-Recently Used (MRU) – Supports hundreds of storage arraysVMW_PSP_FIXED – Fixed - Supports hundreds of storage arraysVMW_PSP_RR – Round-Robin - Supports dozens of storage arraysDELL_PSP_EQL_ROUTED – Dell EqualLogic iSCSI arraysAlso, EMC PowerPath and other vendor unique

vStorage APIs for Array Integration (VAAI)VAAI integrates advanced storage features with VMwareBasic requirements:A capable storage arrayESX 4.1+A software plug-in for ESXNot every implementation is equalBlock zeroing can be very demanding for some arraysZeroing might conflict with full copy

vSphere 5: VAAI 2Block(FC/iSCSI)T10 compliance is improved - No plug-in needed for many arraysFile(NFS)NAS plugins come from vendors, not VMware

vSphere 5: vSphereStorage APIs – Storage Awareness (VASA)VASA is communication mechanism for vCenter to detect array capabilitiesRAID level, thin provisioning state, replication state, etc.Two locations in vCenter Server:“System-Defined Capabilities” – per-datastore descriptorsStorage views and SMS API’s

Storage I/O Control (SIOC)Storage I/O Control (SIOC) is all about fairness:Prioritization and QoS for VMFSRe-distributes unused I/O resourcesMinimizes “noisy neighbor” issuesESX can provide quality of service for storage access to virtual machinesEnabled per-datastoreWhen a pre-defined latency level is exceeded on a VM it begins to throttle I/O (default 30 ms)Monitors queues on storage arrays and per-VM I/O latencyBut:vSphere 4.1 with Enterprise PlusDisabled by default but highly recommended!Block storage only (FC or ISCSI)Whole-LUN only (no extents)No RDM

Virtual Machine MobilityMoving virtual machines is the next big challengePhysical servers are difficult to move around and between data centersPent-up desire to move virtual machines from host to host and even to different physical locationsVMware DRS would move live VMs around the data centerThe “Holy Grail” for server managersRequires networked storage (SAN/NAS)

vSphere 5: Storage DRSDatastore clusters aggregate multiple datastoresVMs and VMDKs placement metrics:Space - Capacity utilization and availability (80% default)Performance – I/O latency (15 ms default)When thresholds are crossed, vSphere will rebalance all VMs and VMDKs according to Affinity RulesStorage DRS works with either VMFS/block or NFS datastoresMaintenance Mode evacuates a datastore

Expanding the ConversationConverged I/O, storage virtualization and new storage architectures

This Hour’s Focus:Non-Hypervisor Storage FeaturesConverged networkingStorage protocols (FC, iSCSI, NFS)Enhanced Ethernet (DCB, CAN, FCoE)I/O virtualizationStorage for virtual storageTiered storage and SSD/flashSpecialized arraysVirtual storage appliances (VSA)

Introduction: Converging on ConvergenceData centers rely more on standard ingredientsWhat will connect these systems together?IP and Ethernet are logical choices

Which Storage Protocol to Use?Server admins don’t know/care about storage protocols and will want whatever they are familiar withStorage admins have preconceived notions about the merits of various options:FC is fast, low-latency, low-CPU, expensiveNFS is slow, high-latency, high-CPU, cheapiSCSI is medium, medium, medium, medium

Which Storage Protocols Do People Use?Source: VirtualGeek.typepad.com 2010 virtualization survey of 125 readers

The Upshot: It Doesn’t MatterUse what you have and are familiar with!FC, iSCSI, NFS all work wellMost enterprise production VM data is on FC, many smaller shops using iSCSI or NFSEither/or? - 50% use a combinationFor IP storageNetwork hardware and config matter more than protocol (NFS, iSCSI, FC)Use a separate network or VLANUse a fast switch and consider jumbo framesFor FC storage8 Gb FC/FCoE is awesome for VMsLook into NPIVLook for VAAI

Serious Performance10 GbE is faster than most storage interconnectsiSCSI and FCoE both can perform at wire-rate

Latency is Critical TooLatency is even more critical in shared storageFCoE with 10 GbE can achieve well over 500,000 4K IOPS (if the array and client can handle it!)

Benefits Beyond Speed10 GbE takes performance off the table (for now…)But performance is only half the story:Simplified connectivityNew network architectureVirtual machine mobility1 GbE Cluster4G FC Storage1 GbE Network10 GbE(Plus 6 Gbps extra capacity)

Enhanced 10 Gb EthernetEthernet and SCSI were not made for each other

SCSI expects a lossless transport with guaranteed delivery

Ethernet expects higher-level protocols to take care of issues

“Data Center Bridging” is a project to create lossless Ethernet

AKA Data Center Ethernet (DCE), Converged Enhanced Ethernet (CEE)

iSCSI and NFS are happy with or without DCB

FCoE requires PFC (Qbb or PAUSE), DCBX (Qaz)

QCN (Qau) is still not readyPriority Flow Control (PFC)802.1QbbCongestion Management (QCN)802.1QauBandwidth Management (ETS)802.1QazPAUSE802.3xData Center Bridging Exchange Protocol (DCBX)Traffic Classes 802.1p/Q

FCoE CNAs for VMware ESXNo Intel (OpenFCoE) or Broadcom support in vSphere 4…

vSphere 5: FCoE Software InitiatorDramatically expands the FCoE footprint from just a few CNAsBased on Intel OpenFCoE? – Shows as “Intel Corporation FCoE Adapter”

I/O Virtualization: Virtual I/OExtends I/O capabilities beyond physical connections (PCIe slots, etc)Increases flexibility and mobility of VMs and bladesReduces hardware, cabling, and cost for high-I/O machinesIncreases density of blades and VMs

I/O Virtualization: IOMMU (Intel VT-d)IOMMU gives devices direct access to system memoryAMD IOMMU or Intel VT-dSimilar to AGP GARTVMware VMDirectPath leverages IOMMUAllows VMs to access devices directlyMay not improve real-world performanceSystem MemoryIOMMUMMUI/O DeviceCPU

Does SSD Change the Equation?RAM and flash promise high performance…But you have to use it right

Flash is Not A DiskFlash must be carefully engineered and integratedCache and intelligence to offset write penaltyAutomatic block-level data placement to maximize ROIIF a system can do this, everything else improvesOverall system performanceUtilization of disk capacitySpace and power efficiencyEven system cost can improve!

The Tiered Storage ClichéCost and PerformanceOptimized for Savings!

Storage for Virtual Environments 2011 R2

More Related Content

What's hot (20)

Similar to Storage for Virtual Environments 2011 R2 (20)

More from Stephen Foskett (18)

Recently uploaded (20)

Storage for Virtual Environments 2011 R2

Editor's Notes