TechTarget Event - Storage Architectures for the Modern Data Centre – Chris Evans
0 likes168 views
The document outlines the evolution and current trends in storage architectures for modern data centers, emphasizing the shift from monolithic designs of the 20th century to more standardized, software-defined solutions in the 21st century. Key topics include customer needs such as efficiency and cost, various storage solutions like hybrid and hyper-converged systems, and the importance of APIs and policy-driven management in addressing the complexities of modern application deployments. It also discusses new technologies such as flash storage, the significance of quality of service, and the diverse landscape of storage products available today.
TechTarget Event - Storage Architectures for the Modern Data Centre – Chris Evans
- 1. STORAGE ARCHITECTURES FOR THE MODERN DATA CENTRE Chris M Evans Langton Blue Ltd Copyright © 2016 Langton Blue Ltd 1
- 2. A WORD ABOUT ME Copyright © 2009-‐2016 Langton Blue Ltd 2 § 28 years experience in the IT industry across all platforms, including IBM mainframe, Windows & Open Systems. § Co-‐founder and independent consultant at Langton Blue Ltd, a specialist consulting company in the UK. § Blogger and part-‐time analyst at Architecting IT § Twitter: @chrismevans, @architectingIT, @langtonblue § Web: www.architecting.it, www.langtonblue.com
- 3. AGENDA § Evolution of The Data Centre § Understanding Customer Needs § Storage Architectures § Product Marketplace § Q&A Copyright © 2009-‐2016 Langton Blue Ltd 3
- 4. EVOLUTION OF THE DATA CENTRE A quick primer on storage heritage Copyright © 2009-‐2016 Langton Blue Ltd 4
- 5. 20TH CENTURY DESIGN § Monolithic § Bespoke hardware § Expensive to acquire, maintain & manage § Static designs and configuration § (Think time taken to run binfile change) § Limited or no data services § No RAID, replication, snapshots § All features that developed over time Copyright © 2009-‐2016 Langton Blue Ltd 5
- 6. 20TH CENTURY ARCHITECTURE § Custom hardware designs & components still reign § Hosts own LUNs mapped from physical RAID groups § Performance defined by RAID group size and disk speed § Wide striping for performance § Throughput & I/O latency influenced (and restricted) by controller capabilities § Lots of manual effort to rebalance systems, especially at scale § LUN migration, rebuilds, reliance on host-‐level LVMs for a lot of the work § Lots of risk fragmentation and orphan storage § Wasted space in RAID groups Copyright © 2009-‐2016 Langton Blue Ltd 6 CONTROLLER A CONTROLLER B BED BED BED BED BED BED BED BED
- 7. 21ST CENTURY TECHNOLOGY § Vendors have moved away from bespoke hardware designs § x86 Standardisation § Commodity, reliable components § High performance processors & memory – Moore’s Law § Interoperability § Components generally work together without problems § Storage protocols are robust and mature (still also developing – NVMe) § New Technology § Flash § NVDIMM § 3D Xpoint § Migration of “smarts” to software -‐ Software Defined Storage § Features implemented in code, not microcode Copyright © 2009-‐2016 Langton Blue Ltd 7
- 8. FLASH & NEW MEDIA § Designs § SLC (Single Level Cell), MLC (Multi-‐level Cell), TLC (Triple Level Cell) § 3D-‐NAND – 3-‐D rather than planar (2D) technology for higher density § 3D-‐Xpoint – faster and more scalable persistent storage than Flash, slower than DRAM and in between on price § Intel/Micron joint development § NVRAM/NVDIMM – DIMM form factor products delivering persistent flash memory on the motherboard § NVMe – new connectivity standard for writing to persistent storage § PCIe and SDD type form factors (not yet widely adopted) Copyright © 2009-‐2016 Langton Blue Ltd 8
- 9. PROCESSOR & MEMORY PERFORMANCE Copyright © 2009-‐2016 Langton Blue Ltd 9 § Processor, memory and bus speeds are increasing § Processor performance and bus speeds follow Moore’s Law § Bus speeds double with each release § PCIe v1.x = 250MB/s (per lane) -‐ 2003 § PCIe v2.x = 500MB/s (per lane) -‐ 2007 § PCIe v3.0 = 985MB/s (per lane) -‐ 2010 § PCIe v4.0 = 1969MB/s (per lane) – 2014 § Features much easier to deliver from software on x86 § New instruction sets like SSSE3 and AVX2 allow storage computations like erasure coding/Reed Solomon to be done in software § Fewer reasons to develop bespoke ASICs and FGPAs
- 10. SOFTWARE DEFINED STORAGE(KEY FEATURES) § Abstraction – I/O services should be delivered independent of the underlying hardware, through logical constructs like LUNs, volumes, file shares and repositories. § Automation – resources should be consumed using CLIs and APIs rather than manually allocated through a GUI. § Policy/Service Driven – the service received (IOPS, latency) should be established by policies that implement Quality of Service, availability and resiliency. § Scalable – solutions should enable performance & capacity scaling independent of I/O delivery. Copyright © 2009-‐2016 Langton Blue Ltd 10
- 11. MAJOR STORAGE THEMES § More choice in products (both hardware and software than ever before) § Divergence – storage platforms for specific purposes (no single platform for all requirements) § Object storage, cloud, backup, primary § Convergence – collapsing the storage/server hardware into a single unit integrating new technology (converged & hyper-‐converged) § Integration – VVOLs, APIs, platform drivers (e.g. Cinder), hypervisor extensions (VASA, VAAI) § New Media § Flash and its successors Copyright © 2009-‐2016 Langton Blue Ltd 11
- 12. UNDERSTANDING CUSTOMER NEEDS The changing face of application deployment Copyright © 2009-‐2016 Langton Blue Ltd 12
- 13. WHAT ARE CUSTOMERS’ PAIN POINTS § Cost § Performance § Operational Complexity § Reliability § Floor Space § Power Consumption Copyright © 2009-‐2016 Langton Blue Ltd 13 Source: Tintri State of Storage Report, March 2015 https://www.tintri.com/sites/default/files/fie ld/pdf/whitepapers/state_of_storage_ infographic_150513t10216.pdf
- 14. WHAT DO CUSTOMERS CARE ABOUT § Efficiency § Cost ($/GB, effective or raw) § Physical media § Performance § Latency, IOPS, throughput, Quality of Service § Management § Multi-‐tenancy, APIs (Cinder), Automation, Abstraction (VVOLs) § Reliability § Resiliency, Availability § Features § Data Services, Data Protection § Breaking the buying cycle § No forklift upgrades or 3-‐4 year refresh projects Copyright © 2009-‐2016 Langton Blue Ltd 14
- 15. PREDICTABLYUNPREDICTABLE § Workloads becoming more random in nature § I/O blender § Peaks and troughs in demand § VDI (Bootstorms, logoff storms) § Increase in I/O Density § Driven by virtualisation and soon containers § Agility § Desire to move data around, spin up copies, create test dev environments, destroy and recreate § Persistence § Applications expecting to be transient in nature, but more need for storage to be 100% available across the entire infrastructure Copyright © 2009-‐2016 Langton Blue Ltd 15
- 16. VIRTUAL FIRST STRATEGY § Provided the ability to consolidate over-‐provisioned server resources § Systems typically 25% allocated § Reduced delivery time for (virtual) servers from days/months to hours § Assuming sufficient capacity available § Enabled high availability through infrastructure § No need to deploy clustered systems § Centralised data protection (e.g. VADP) § Centralised storage and virtualisation don’t play well together § Storage doesn’t understand VMs § Storage sees LUNs & volumes § Virtualisation sees VMs § LUNs encapsulate many volumes – no visibility of VMs § Problems being worked around with VVOLs and other solutions that integrate directly into hyper-‐converged platforms. Copyright © 2009-‐2016 Langton Blue Ltd 16
- 17. TECHNICAL REQUIREMENTS– QUALITY OF SERVICE § Eliminate the “noisy neighbour” problem § Provide storage resources based on policy § Ensure storage isn’t under-‐delivering § Guarantee IOPS, performance minima § Ensure storage isn’t over-‐delivering § 1st on the array gets best performance (until there are more) § Set limits and prioritisation § Apply to VM/VMDK where possible Copyright © 2009-‐2016 Langton Blue Ltd 17
- 18. TECHNICAL REQUIREMENTS-‐ API DRIVEN § Admin-‐based functionality driven by API § Ability to create, manage, destroy LUNs from code § Integrate storage functions into workflow § Cloud, Automation, Self Service § Reduce dependence on manual process § Agility Copyright © 2009-‐2016 Langton Blue Ltd 18
- 19. TECHNICAL REQUIREMENT – SUPPORT VIRTUAL § Reduce waste from over-‐provisioning § Support – object (e.g. VM-‐based) management § Apply policies at the VM level § Data location, performance, latency, backup § VM friendly for data protection (snapshots/replication) § Offload heavy lifting tasks (e.g. VAAI) Copyright © 2009-‐2016 Langton Blue Ltd 19
- 20. STORAGE ARCHITECTURES A few words about design Copyright © 2009-‐2016 Langton Blue Ltd 20
- 21. DIVERSITY OF STORAGE PLATFORMS Copyright © 2009-‐2016 Langton Blue Ltd 21 HYBRID COMMODITY OPEN-‐ SOURCE APP/DATA AWARE SMART STORAGE OBJECT STORES SCALE-‐OUT SAN SCALE-‐OUT NAS ALL-‐FLASH SOFTWARE DEFINED SCALE-‐OUT BACKUP ULTRA-‐FAST FLASH HIGH-‐CAPACITY FLASH HPC
- 22. INFRASTRUCTURE CONSUMPTION CONTINUUM Copyright © 2009-‐2016 Langton Blue Ltd 22 As-‐a-‐Service Hyper-‐ converged Converged Proprietary Appliance Software on Commodity SMB/Start-‐ups Large Enterprise Hyperscale & SP Ease of Implementation Implementation Flexibility Degree of Vendor Lock-‐in Cost Efficiency Harder More Flexible Less Lock-‐in At Large Scale Easier Less Flexible More Lock-‐in At Small Scale
- 23. SCALE UP § Typically Dual controller implementation § Active/Active & Active/Passive § Throughput limited by controller performance § More throughput means better controller capability (CPU/memory) § I/O performance limited by media § Flash based implementations failed to use flash to fullest capability § Older designs don’t cater for write endurance issues § Automated tiering capabilities to place data on most appropriate technology Copyright © 2009-‐2016 Langton Blue Ltd 23 CONTROLLER A CONTROLLER B
- 24. SCALE OUT (CLOSELY COUPLED) § Scale out through adding nodes and/or capacity § Tight node integration either paired or with bespoke backplane (e.g. PCIe/Infiniband) § Symmetric design – scale with nodes of similar size/capacity § LUNs/Volumes delivered from specific nodes – not fully load balanced § Scale out limited by backplane traffic § Add/remove nodes dynamically Copyright © 2009-‐2016 Langton Blue Ltd 24 CTRL A CTRL B CTRL C CTRL D CTRL E CTRL F
- 25. SCALE OUT (LOOSELY COUPLED) § Fully distributed “Shared Nothing” architecture § Loose node coupling (usually IP-‐based) § All nodes contribute storage (no disk shelves) § At least one node of “spare” capacity required across the cluster § More nodes means more efficiency in redundancy § Data resiliency based on replicating data between nodes § Standard hardware (typically 1U server) design § Node power efficiency & size become important § High scalability § Smaller increments in adding capacity/performance § More complex design Copyright © 2009-‐2016 Langton Blue Ltd 25 NODE A NODE B NODE C NODE D NODE E NODE F
- 26. HYPER-‐CONVERGED § Single hardware device for storage and server § Node-‐based scale-‐out § Integrated hypervisor & storage services § Storage implemented in VM or hypervisor kernel § Distributed file system (scale-‐out shared nothing) § Efficient use of resources § Simplified management § At scale probably more expensive than separate components § Reduced VM density § Issues managing performance vs capacity scaling Copyright © 2009-‐2016 Langton Blue Ltd 26 NODE A NODE B NODE C NODE D NODE E NODE F
- 27. HYPER-‐CONVERGENCE -‐ TCO § Simplified § Less “plan” time -‐ no requirement to certify and test components § Less “build” time – deploy a new node, power up and add to the configuration § Less “operate” time – single management console, typically integrated with the hypervisor, tight hypervisor awareness § Efficient § Scale per node; less wastage, no up-‐front hardware acquisition § “Decommission” time almost eliminated; simply add new nodes and evacuate/remove old nodes over time § Provides overall TCO savings (not storage specific) § Not typically suited to high-‐scale environments Copyright © 2009-‐2016 Langton Blue Ltd 27
- 28. STORAGE VIRTUALISATION-‐ EVOLUTION § Monolithic – single central controller § Inline in the data path § Controller maps logical to “physical” storage § data management features built into the controller § Not highly scalable § Centralised Metadata – Distributed Data § Central metadata functions § Data distributed/replicated across many devices § System not in the data path § Separation of control and data planes § Totally Distributed § No central metadata or data § Data distributed across many devices § Fully scalable architecture Copyright © 2009-‐2016 Langton Blue Ltd 28
- 29. ADVANCED SERVICES § Space Efficiency § Compression § De-‐duplication § Thin Provisioning § Quality of Service § Object abstraction (VVOLS, per VM services) § Application consistent snapshots/backup Copyright © 2009-‐2016 Langton Blue Ltd 29
- 30. SPACE OPTIMISATION/EFFICIENCY Copyright © 2009-‐2016 Langton Blue Ltd 30 Feature Compression De-‐duplication Thin Provisioning Physical space reduction ✔ ✔ ❌ Architecture/metadata dependent ❌ ✔ ✔ CPU Friendly ❌ ✔ ✔ Traditional OLTP Database friendly ✔ ❌ ❌ VDI/VSI Friendly ❌ ✔ ✔ HDD Friendly ✔ ❌ ✔ Flash Friendly ✔ ✔ ✔
- 31. QUALITY OF SERVICE (AGAIN) § Assign service-‐based metrics to application workloads § Reduce/eliminate the “noisy neighbour” effect § Set application owner “expectations” on performance § Avoid first/last application syndrome from legacy arrays § Metrics § Latency § IOPS § Throughput § Establish maximum/minimum/burst values & priority Copyright © 2009-‐2016 Langton Blue Ltd 31
- 32. OBJECT ABSTRACTION § Implement per-‐VM services § Abstract away from LUNs/Volumes § Implement object-‐specific services § Snapshots § Replication § Cloning § Application-‐consistent data management § Examples § VVOLs § Object Stores Copyright © 2009-‐2016 Langton Blue Ltd 32
- 33. VENDOR LANDSCAPE A look at the product marketplace Copyright © 2009-‐2016 Langton Blue Ltd 33
- 34. STORAGE NASCAR SLIDE! Copyright © 2009-‐2016 Langton Blue Ltd 34
- 35. PUTTING IT ALL TOGETHER § Start with Requirements § Don’t buy what you don’t need § Focus on your sensitivities (cost, performance, scalability) § See the “bigger picture” § Don’t just storage in isolation § Don’t buy a technology § Look past the glamour of shiny boxes! § Evaluate your options § Use tools to create an equitable test environment § Use online resources and other views § Wisdom of the crowd applies Copyright © 2009-‐2016 Langton Blue Ltd 35