SlideShare a Scribd company logo

NetApp Se training storage grid webscale technical overview

S
solarisyougood

The document provides an overview of StorageGRID Webscale, an object storage platform from NetApp. It discusses key concepts such as object storage, metadata management, and StorageGRID's dynamic policy engine. The policy engine uses metadata and user-defined rules to intelligently place and manage objects across multiple sites, storage tiers, and protocols (e.g. S3) over their lifecycle. This allows building complex data management policies without impacting performance or capacity.

Technology
1 of 52
1
2
Most read
3
4
5
6
7
8
9
Most read
10
11
12
13
14
15
Most read
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
StorageGRID Webscale Technical Overview September 2014
Student Guide & Internal & Confidential Update Daly https://goo.gl/VVmVZ0 Download Full Docs © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only2 5673-StorageGRID for Standard Enterprise Solution http://ouo.io/SROnEV Building Private Clouds on E-Series The Swift Object Opportunity http://ouo.io/vWrya Deployment Guide - Acuo Universal Clinical Platform with NetApp StorageGRID Integration Architecture http://ouo.io/TowHxU Deployment Guide - StorageGRID E-Series http://ouo.io/jhf1M Getting Started with StorageGRID CDMI Integrations http://ouo.io/dAORQj NetApp and the Object-Based Storage and Archiving Landscape http://ouo.io/wvoEgQ NetApp Distributed Content Repositories What Are We Doing in Real Life http://ouo.io/3y65M NetApp Portfolio Overview http://ouo.io/O9aKQ POC - StorageGRID Webscale Proof of Concept Guide http://ouo.io/pzljIH QRG_StorageGRID Webscale http://ouo.io/3dqRr SE Training - StorageGRID Webscale Technical Overview http://ouo.io/BqtKUU Solution Brief - NetApp StorageGRID Manage Large Pools of Mission-Critical Patient Data Across Healthcare Facilities http://ouo.io/KTDFz StorageGRID Webscale 10.0 Installation Overview http://ouo.io/hIYhJ StorageGRID Webscale Nonstop Object Storage for Enterprise and Cloud http://ouo.io/dMRoOr StorageGRID Webscale Object-Enabled Data Management http://ouo.io/nhuj4f Technical Report - Integration Guide for NetApp StorageGRID with McKesson Horizon Medical Imaging PACS http://ouo.io/9ncafj
Agenda  What is Object Storage  StorageGRID Webscale Introduction  Technical Overview  Key Concepts  Data flow © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only3
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only4 Object storage growth driven by macro trends… Unstructured data continues to grow •New sources of unstructured data growth media, mobile and applications •Data continues to be retained for long periods (archival, compliance etc) Unstructured data profile is changing •Simultaneous access to the same data is rarely required •Most data accessed a few times initially and then rarely accessed Data access is changing •Geographically dispersed access •Applications accessing data – don’t care about POSIX semantics, file locking Highly cost sensitive petabyte scale repositories •Driving tradeoffs between $/GB, latency, throughput, and data protection Storage being managed in a cloud ecosystem •Unified management & orchestration •Growth in cloud hosted applications that leverage object storage
Introducing StorageGRID Webscale  A new variant of StorageGRID  Target object store (cloud, archive, media) use cases at massive scale  New Features  Protocols – native support for S3 API  Scalability – 100 billion objects, 70 PB  Simplicity – modular scalable resilient architecture, simplified deployment  Proven track record for reliability and innovation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only5
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only6 Block File Object What is Object Storage? Different ways to address data Specific location on disks / memory Tracks Sectors Specific folder in fixed logical order File path File name Date Flexible container size Data and Metadata Unique ID
Object Storage Example: File vs. Object © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only7 Object BasedFile Based ValetParking Garage Daily Garage 1 Floor 4 Row N Space 53 /users/jsmith/car/garage1/floor4/rown/space53.file C:UsersjsmithGarage1Floor4RowNSpace53.file Object UID 317
Why Object storage? © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only8  Massive Scale  Billions of objects  Petabytes of data  Global namespace  Explosive growth  Respond to compliance and retention requirements  Cost effectively meet SLAs with intelligent data placement  Access from anywhere  Control access, security, and data integrity
Object storage used across different workloads Emerging Object storage segments in the hybrid cloud © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only9 Service Providers (XaaS) Media Repository (Media Redistribution) • Large object sizes (+250MB) • Distributed repositories • High data rates for redistribution • Time to first byte latency < 50ms Secure multi-tenancy (billing, isolation, authentication, & self-service) Control plane APIs and workflow automation New apps requiring RESTful interfaces (S3, Swift) Web Data Repositories • Small object (~KB) performance • Extremely high transaction load • Searchable, scalable metadata • High object counts Data Archives • Long access latency tolerance • Integration to tape / Glacier • Long retention periods • Erasure coding Scale seamlessly Ease of install & management Global namespace Cost
StorageGRID Webscale Object-Enabled Data Management © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only10
NetApp StorageGRID Webscale Software Defined Object Storage  Billions of objects and multi-petabyte  Architected for massive scale Built for the Hybrid Cloud  Global, Always-On data availability and durability  Support for cloud applications (S3, CDMI)  10th Generation object store proven with product deployments Dynamic Policy Engine  Intelligent, policy driven data management for optimal availability, performance and cost over the life cycle of data © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only11
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only12 StorageGRID Platform for Distributed Content Repositories StorageGRID Object Storage Software + E-Series Storage Array MULTIPLE: APPLICATIONS + SITES + PROTOCOLS MULTIPLE: TARGETS + TIERS MULTIPLE: TENANTS + POLICIES + ADMINISTRATORS Site 1 Site 2 … Site NSite 3 APPLICATIONS APPLICATIONS APPLICATIONS APPLICATION NetApp E-Series Tape
StorageGRID Webscale features © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only13 Hardware Obsolescence Protection Non-Disruptive Operations Object Integrity and Security Multi-Tenancy Global Object Namespace Services Automation Manageability Reliability Scalability ILM, Metadata driven policies Seamless Scaling Audit & Reporting
Technical Overview StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only14
Solution Topology © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only15 • Clients can access global object namespace via Gateway Nodes or Storage Nodes E2760 DE6600 Administrators REST API clients 10GbE LAN Per Node Resource Requirement VMDK (GB) vCPU RAM (GB) 100 8 24 100 8 24 300 8 24 Storage Gateway Admin DE6600 E2760 DE6600 DE6600 • StorageGRID Webscale nodes running in VMware hosts S3, CDMIHTTPS 16Gb FC switch WAN Router • E2760 block-based storage • SSDs for read cache and VM datastores • Mixed disk types (SSD, SAS, NL-SAS) for tiered storage pools
StorageGRID Architecture © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only16 App1 App2  Admin Nodes  Management Services: Configuration, Monitoring, Audit and Logging  Storage Nodes  Manages object storage including replication  API Gateway Nodes  Load balancing interface through which applications connect to the system  Archive Nodes  Interface to archive media storage such as tape DATA CENTER 1 DC2 DC3 Design the grid to scale for performance, capacity & resiliency STORAGE STORAGE ADMIN ARCHIVE API LOAD BALANCER
 Use NetApp StorageGRID Webscale Designer to architect the grid to your requirements  Quickly deploy the grid via NetApp StorageGRID Webscale Installer  Adapt your grid to changing requirements  Add sites  Add nodes and capacity  Support rolling upgrades Centralized Deployment & Rolling Upgrades Design, deploy, and maintain configuration control © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only17 DC1 DC2 DC4 DC3
Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only18 Client Connections Storage Pools Metadata Object Identifier ILM Policy
Client Connections, Object Identifiers & Metadata © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only19 Client CDMI S3 Read/Write StorageGRID Webscale System HTTPs Client Connections Object Identifier Object type: JPG Date modified: 07/21/ 2014 GPS Coordinates: Lat, Long Location: DC @ Seattle Metadata
 Scale beyond traditional application metadata to enable new capabilities  Flexibility:  Metadata is application defined  Up to 4096 fields can be created as requirements change (CDMI)  No lock-in to predefined schema  ILM engine evaluates the metadata and applies policies  S3 metadata available to the policy engine Extensive metadata management Metadata – Why it matters ?  Metadata is distributed throughout the grid  Increased scalability and resiliency  Faster retrieval and efficient ILM policy evaluation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only20 Object Identifier: 00006FFD00192A1200555FFEE12039468EBF622 D9402C4F962 Locations: Location 1: Data Center 1/DC1-S1/LDR Location 2: Data Center 2/DC2-S3/LDR Metadata CDMI/CVTE: 0 CDMI/META: {"application":"finance","doctype":"contract", "project":"45667}
Extensive metadata management Metadata Management  Metadata Includes – Object size, User Metadata, Bucket Name, Account Id  CDMI supports a large number of user metadata fields (up to 4096)  Policy support  Metadata can be used in polices (CDMI & S3)  No pre-configuration required for metadata, immediately available for policy use © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only21
Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only22 Client Connections Storage Pools Metadata Object Identifier ILM Policy
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only23 Sites (Link Cost Groups) Storage Pools StorageGRID Data Management Storage Pools Storage Grade TAPE Storage Grade SAS Storage Grade FLASH Munich, Germany San Francisco, USA Storage Node T1 Storage Node T2 Tape Storage Pool EU Storage Pool Storage Node S1Storage Node F1 Storage Node F2 Fast Storage Pool Vancouver, CAStorage Node T3Storage Node F3  Simple configuration of SLO based storage across multiple sites and storage grades
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only24 Customer Defined Storage grades StorageGRID Data Management Associating Storage Nodes with Storage Grades Creating Storage Grades  Customers can configure their own Storage Grades, and associate them with specific nodes
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only25 Site configuration – Not just a site StorageGRID Data Management  Customers can model network costs, creating powerful configurations  Possible Examples  Configure two sites to be treated as one,  Specify which sites are connected by high b/w pipes & low b/w pipes,  Prefer traffic to go in a particular direction
Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only26 Client Connections Storage Pools Metadata Object Identifier ILM Policy
Dynamic Policy Engine – An Overview Data management key concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only27  Manage policies not objects  Evaluate objects based on metadata such as:  Custom user application metadata  Method of ingest (S3 or CDMI)  Size of object  Last access time  Apply ILM rules to set:  Geography — Placement of an object  Storage grade — Type of storage used to store an object  Replication — Number of copies stored  Retention — Set time during which an object cannot be purged App1 Site1 Site2 Site3
© 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only28 Policy Management Data Management Key Concepts ILM Policy When If.. then.. • At Ingest • Objects at rest (already ingested) • After a read (enabling caching) • Metadata matches specific criteria • Move/Copy to one or more Storage Pools
ILM Rule – Specify the Conditions Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only29 When If..  Customers designate when a rule will be applied and what conditions must be met to trigger the action for placement and retention.
ILM Rule – Defining data placement and retention © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only30 then..  Simple configuration of complex placement rules, that are graphically displayed for easy understanding Data Management Key Concepts
Combine ILM rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only31 Data Management Key Concepts  Highly complex business logic can be implemented by combining multiple rules into a policy
ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only32 Application writes to grid via S3 Metadata is evaluated DC1 DC2 DC3  Store objects with S3 metadata “Bucket Name = ClientX” on ingest at DC1 on SSD and DC2 on SATA for 90 days  After 90 days store on DC1 on SATA and DC3 on Tape 1 x copy DC1SSD 1 x copy DC2SATA 90 Days later…. 1 x copy DC1SATA 1 x copy DC3Tape Bucket Name = ClientX
ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only33 DC1 DC2 DC3  What if requirements change?  A new Data Center is brought on line and now we must store a copy at DC4  We can edit the policy – and apply not just to new data, but re-evaluate existing data and create new replicas as needed  Without impact to the performance of the grid DC4
 Create ILM Rules to set customized service levels  GOLD = 2 x copies on SSD, 1 x copy on SATA  SILVER = 1 x copy on SAS, 2 x SATA  BRONZE = 1 x copy on SATA, 1 x copy on tape Set custom service levels via metadata SLA Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only34 {"sla":"gold"} DC1 DC2 DC3
Built-in Object Integrity Verification and Self Healing Durability at the object level  Digital fingerprint is calculated per object upon ingest  Interlocking layers of object-wide and sub-object level integrity protection  Object hash value  Content hash value  CRC checksum  HMAC message authentication digest © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only35 ID: 41E85A1D Data Metadata Data Metadata Fingerprint
Built-in Object Integrity Verification and Self Healing Durability at the object level  Continuous verification: on ingest, retrieval, replication, migration and at rest  Object failing integrity test is automatically replaced with another copy © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only36
 Real time audit feed can be used to create custom reports  Audit logs created in open format allows the use of partner products such as Splunk  Comprehensive audit feed for:  Chargeback and billing  Search integration  Custom reporting  Security diagnostics  Compliance events  Validate performance for SLAs Continuous and active monitoring Audit and Reporting © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only37
System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only38  Analyze system performance and activity at every level  Grid Wide
System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only39  Analyze system performance and activity at every level  Grid Wide  Site Specific
System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only40  Analyze system performance and activity at every level  Grid Wide  Site Specific  Service Level
Data Flow Understanding how data flows through a StorageGRID Webscale system for different operations © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only41
Data flow overview Basic topology and business rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 42  Three departments  Sales  Finance  Marketing  Two pools per department  Capacity pool  Performance pool  Three sites  Two datacenter sites  One remote office  Policy - Finance  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Receives object write request from client along with custom metadata  Returns object ID to client and forms the file payload into an object  Packetization  Digital fingerprint  Compression (optional)  Encryption (optional) 43 0x05DFF4338ADCE6F5
Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 44  By default, creates semi-sync local copy for immediate redundancy  Metadata is stored and replicated  Replicate object as per ILM policy  Finance Policy Example  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Replication Metadata-driven ILM with optimal resource utilization © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Optimal resources in target storage pools are selected for the replication destination  While honoring ILM the grid considers  Network costs  Server utilization  Storage utilization 45
Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Retrieval Requesting objects by the client from StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Receives object read request from client Determines optimal object location relative to request location Streams a copy of the object to the client and verifies integrity of the object on-the- fly 46
Thank You © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only47
Management Services Admin Node  NMS – Network Management System: Provides the administrative interface for configuration and monitoring of the grid.  CMN – Configuration Management Node: Manages system-wide configurations such as connection profiles, grid tasks, and system configuration options.  AMS - Audit Management System: Keeps logs of system activity and events. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only48
Object, Metadata Storage and Replication Storage Nodes  LDR - Local Distribution Router: Stores, moves, verifies, and retrieves object data stored on disks  DDS - Distributed Data Store: Stores, replicates, and protects metadata in the key value store. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only49
Object, Metadata Storage and Replication Storage Nodes - Continued  CMS - Content Management System: Manages object placement and replication based on ILM rules.  ADC - Administrative Domain Controller: Maintains topology information and provides authentication services. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only50
Load Balancing and Client Connectivity API Gateway Nodes  CLB - Connection Load Balancer: Acts as switchboard for connecting clients to the most efficient LDR service for ingest and retrieval.  Ports  8081 CDMI  8082 S3 © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only51
Data storage on archive media Archive Nodes  ARC – Archive: Communicates with archiving middleware to store and retrieve data to and from archive media such as tape. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only52

More Related Content

PPTX
NetApp & Storage fundamentals
Shashidhar Basavaraju
 
PDF
Netapp Storage
Prime Infoserv
 
PPTX
Snowflake Datawarehouse Architecturing
Ishan Bhawantha Hewanayake
 
PPTX
Building a modern data warehouse
James Serra
 
PDF
Lustre Best Practices
George Markomanolis
 
PPTX
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
PDF
Introduction to Hadoop Administration
Ramesh Pabba - seeking new projects
 
PDF
Let’s get to know Snowflake
Knoldus Inc.
 
NetApp & Storage fundamentals
Shashidhar Basavaraju
 
Netapp Storage
Prime Infoserv
 
Snowflake Datawarehouse Architecturing
Ishan Bhawantha Hewanayake
 
Building a modern data warehouse
James Serra
 
Lustre Best Practices
George Markomanolis
 
Real-time Analytics with Trino and Apache Pinot
Xiang Fu
 
Introduction to Hadoop Administration
Ramesh Pabba - seeking new projects
 
Let’s get to know Snowflake
Knoldus Inc.
 

What's hot (20)

PDF
Snowflake for Data Engineering
Harald Erb
 
PDF
Gartner 2021 Magic Quadrant for Cloud Database Management Systems.pdf
momirlan
 
PPTX
Snowflake Data Loading.pptx
Parag860410
 
PDF
Seastore: Next Generation Backing Store for Ceph
ScyllaDB
 
PPTX
Introducing the Snowflake Computing Cloud Data Warehouse
Snowflake Computing
 
PDF
KSnow: Getting started with Snowflake
Knoldus Inc.
 
PDF
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
PDF
Ceph and RocksDB
Sage Weil
 
PDF
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Paula Koziol
 
PDF
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
PPTX
Azure DataBricks for Data Engineering by Eugene Polonichko
Dimko Zhluktenko
 
PPTX
Snowflake essentials
qureshihamid
 
PDF
Storage tiering and erasure coding in Ceph (SCaLE13x)
Sage Weil
 
KEY
Everything I Ever Learned About JVM Performance Tuning @Twitter
Attila Szegedi
 
PDF
Nifi
Julio Castro
 
PDF
Azure DRaaS v0.7
Luca Mauri
 
PDF
RBD: What will the future bring? - Jason Dillaman
Ceph Community
 
PDF
VMware Tanzu Introduction
VMware Tanzu
 
PPTX
Zero to Snowflake Presentation
Brett VanderPlaats
 
PPTX
AWS Snowball
zekeLabs Technologies
 
Snowflake for Data Engineering
Harald Erb
 
Gartner 2021 Magic Quadrant for Cloud Database Management Systems.pdf
momirlan
 
Snowflake Data Loading.pptx
Parag860410
 
Seastore: Next Generation Backing Store for Ceph
ScyllaDB
 
Introducing the Snowflake Computing Cloud Data Warehouse
Snowflake Computing
 
KSnow: Getting started with Snowflake
Knoldus Inc.
 
Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...
GetInData
 
Ceph and RocksDB
Sage Weil
 
Sizing Splunk SmartStore - Spend Less and Get More Out of Splunk
Paula Koziol
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 
Azure DataBricks for Data Engineering by Eugene Polonichko
Dimko Zhluktenko
 
Snowflake essentials
qureshihamid
 
Storage tiering and erasure coding in Ceph (SCaLE13x)
Sage Weil
 
Everything I Ever Learned About JVM Performance Tuning @Twitter
Attila Szegedi
 
Nifi
Julio Castro
 
Azure DRaaS v0.7
Luca Mauri
 
RBD: What will the future bring? - Jason Dillaman
Ceph Community
 
VMware Tanzu Introduction
VMware Tanzu
 
Zero to Snowflake Presentation
Brett VanderPlaats
 
AWS Snowball
zekeLabs Technologies
 
Ad

Viewers also liked (19)

PPTX
Student guide power systems for aix - virtualization i implementing virtual...
solarisyougood
 
PPTX
How to shutdown and power up of the netapp cluster mode storage system
Saroj Sahu
 
PDF
Ensuring Data Integrity white paper
Future Perfect 2012
 
PDF
NetApp - 10martie2011
Agora Group
 
PPTX
Snap protect se_presentation_v3.0
Mikis Eminov
 
PDF
NetApp
Best Workplaces Conference
 
PDF
NetApp Industry Keynote - Flash Memory Summit - Aug2015
Val Bercovici
 
PPTX
NetApp SAPPHIRE 2016 in SUSE booth: "Safeguarding HANA"
Mike Nelson
 
PPTX
Multi-Site Perforce at NetApp
Perforce
 
PDF
Geekiest Conference Quotes by NetApp Insight Attendees
NetApp Insight
 
PPT
NetApp-ClusteredONTAP-Fall2012
Michael Harding
 
PDF
SOFTBANK TELECOM Corp.
NetApp
 
PDF
Top Mandalay Bay Attractions
NetApp Insight
 
PPTX
How to make sure the right quality is delivered by my translation vendor? (Ed...
TAUS - The Language Data Network
 
PPTX
VMware PEX Boot Camp - VMware View on NetApp: Technical Integration to Drive ...
NetApp
 
PDF
NetApp Insight 2015 Las Vegas Sponsors Guide
NetApp Insight
 
PDF
How NetApp IT Integrates ServiceNow with OnCommand Insight (OCI)
NetApp
 
PDF
How to solve misalignment lun netapp on linux servers by Ivan
Ivan Silva
 
PDF
NetApp Flash Storage Facts
NetApp Insight
 
Student guide power systems for aix - virtualization i implementing virtual...
solarisyougood
 
How to shutdown and power up of the netapp cluster mode storage system
Saroj Sahu
 
Ensuring Data Integrity white paper
Future Perfect 2012
 
NetApp - 10martie2011
Agora Group
 
Snap protect se_presentation_v3.0
Mikis Eminov
 
NetApp
Best Workplaces Conference
 
NetApp Industry Keynote - Flash Memory Summit - Aug2015
Val Bercovici
 
NetApp SAPPHIRE 2016 in SUSE booth: "Safeguarding HANA"
Mike Nelson
 
Multi-Site Perforce at NetApp
Perforce
 
Geekiest Conference Quotes by NetApp Insight Attendees
NetApp Insight
 
NetApp-ClusteredONTAP-Fall2012
Michael Harding
 
SOFTBANK TELECOM Corp.
NetApp
 
Top Mandalay Bay Attractions
NetApp Insight
 
How to make sure the right quality is delivered by my translation vendor? (Ed...
TAUS - The Language Data Network
 
VMware PEX Boot Camp - VMware View on NetApp: Technical Integration to Drive ...
NetApp
 
NetApp Insight 2015 Las Vegas Sponsors Guide
NetApp Insight
 
How NetApp IT Integrates ServiceNow with OnCommand Insight (OCI)
NetApp
 
How to solve misalignment lun netapp on linux servers by Ivan
Ivan Silva
 
NetApp Flash Storage Facts
NetApp Insight
 
Ad

Similar to NetApp Se training storage grid webscale technical overview (20)

PPTX
Se training storage grid webscale technical overview
solarisyougood
 
PPTX
ECS/Cloud Object Storage - DevOps Day
Bob Sokol
 
PPTX
Webinar: Data Protection for Kubernetes
MayaData Inc
 
PDF
Presentation atmos architecture overview
xKinAnx
 
PPTX
Five Tips for Running Cloudera on AWS
Cloudera, Inc.
 
PPTX
New Ceph capabilities and Reference Architectures
Kamesh Pemmaraju
 
PPTX
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
Red_Hat_Storage
 
PPTX
Webinar: Cloud Storage: The 5 Reasons IT Can Do it Better
Storage Switzerland
 
PDF
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
PDF
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
PDF
Cloud Bursting: Leveraging the Cloud to Maintain App Performance during Peak ...
Veritas Technologies LLC
 
PPTX
Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...
Ceph Community
 
PPTX
Introducing IBM Spectrum Scale 4.2 and Elastic Storage Server 3.5
Doug O'Flaherty
 
PDF
Big data and cloud computing 9 sep-2017
Dr. Anita Goel
 
PDF
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
PDF
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
PPTX
EMC EC Overview
solarisyougood
 
PDF
Accelerate Analytics and ML in the Hybrid Cloud Era
Alluxio, Inc.
 
PDF
MySQL day Dublin - OCI & Application Development
Henry J. Kröger
 
PPTX
Geode Meetup Apachecon
upthewaterspout
 
Se training storage grid webscale technical overview
solarisyougood
 
ECS/Cloud Object Storage - DevOps Day
Bob Sokol
 
Webinar: Data Protection for Kubernetes
MayaData Inc
 
Presentation atmos architecture overview
xKinAnx
 
Five Tips for Running Cloudera on AWS
Cloudera, Inc.
 
New Ceph capabilities and Reference Architectures
Kamesh Pemmaraju
 
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?
Red_Hat_Storage
 
Webinar: Cloud Storage: The 5 Reasons IT Can Do it Better
Storage Switzerland
 
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
Cloud Bursting: Leveraging the Cloud to Maintain App Performance during Peak ...
Veritas Technologies LLC
 
Ceph Day New York 2014: Best Practices for Ceph-Powered Implementations of St...
Ceph Community
 
Introducing IBM Spectrum Scale 4.2 and Elastic Storage Server 3.5
Doug O'Flaherty
 
Big data and cloud computing 9 sep-2017
Dr. Anita Goel
 
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
Persistent Storage with Kubernetes in Production
Cheryl Hung
 
EMC EC Overview
solarisyougood
 
Accelerate Analytics and ML in the Hybrid Cloud Era
Alluxio, Inc.
 
MySQL day Dublin - OCI & Application Development
Henry J. Kröger
 
Geode Meetup Apachecon
upthewaterspout
 

More from solarisyougood (20)

PPTX
Emc vipr srm workshop
solarisyougood
 
PPTX
Emc recoverpoint technical
solarisyougood
 
PPTX
Emc vmax3 technical deep workshop
solarisyougood
 
PPTX
EMC Atmos for service providers
solarisyougood
 
PPTX
Cisco prime network 4.1 technical overview
solarisyougood
 
PPTX
Designing your xen desktop 7.5 environment with training guide
solarisyougood
 
PPT
Ibm aix technical deep dive workshop advanced administration and problem dete...
solarisyougood
 
PPT
Ibm power ha v7 technical deep dive workshop
solarisyougood
 
PPT
Power8 hardware technical deep dive workshop
solarisyougood
 
PPT
Power systems virtualization with power kvm
solarisyougood
 
PPTX
Power vc for powervm deep dive tips &amp; tricks
solarisyougood
 
PPTX
Emc data domain technical deep dive workshop
solarisyougood
 
PPT
Ibm flash system v9000 technical deep dive workshop
solarisyougood
 
PPTX
Emc vnx2 technical deep dive workshop
solarisyougood
 
PPTX
Emc isilon technical deep dive workshop
solarisyougood
 
PPTX
Emc ecs 2 technical deep dive workshop
solarisyougood
 
PPTX
Emc vplex deep dive
solarisyougood
 
PPTX
Cisco mds 9148 s training workshop
solarisyougood
 
PPTX
Cisco cloud computing deploying openstack
solarisyougood
 
PPTX
Vmware 2015 with vsphereHigh performance application platforms
solarisyougood
 
Emc vipr srm workshop
solarisyougood
 
Emc recoverpoint technical
solarisyougood
 
Emc vmax3 technical deep workshop
solarisyougood
 
EMC Atmos for service providers
solarisyougood
 
Cisco prime network 4.1 technical overview
solarisyougood
 
Designing your xen desktop 7.5 environment with training guide
solarisyougood
 
Ibm aix technical deep dive workshop advanced administration and problem dete...
solarisyougood
 
Ibm power ha v7 technical deep dive workshop
solarisyougood
 
Power8 hardware technical deep dive workshop
solarisyougood
 
Power systems virtualization with power kvm
solarisyougood
 
Power vc for powervm deep dive tips &amp; tricks
solarisyougood
 
Emc data domain technical deep dive workshop
solarisyougood
 
Ibm flash system v9000 technical deep dive workshop
solarisyougood
 
Emc vnx2 technical deep dive workshop
solarisyougood
 
Emc isilon technical deep dive workshop
solarisyougood
 
Emc ecs 2 technical deep dive workshop
solarisyougood
 
Emc vplex deep dive
solarisyougood
 
Cisco mds 9148 s training workshop
solarisyougood
 
Cisco cloud computing deploying openstack
solarisyougood
 
Vmware 2015 with vsphereHigh performance application platforms
solarisyougood
 

Recently uploaded (20)

PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
MYSQL Presentation for SQL database connectivity
Swati270511
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PPTX
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
PPTX
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
MYSQL Presentation for SQL database connectivity
Swati270511
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
Teaching material agriculture food technology
LiaRayya
 
CIFDAQ's Market Insight: SEC Turns Pro Crypto
CIFDAQ
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Understanding_Digital_Forensics_Presentation.pptx
ImranKhan423233
 
Big Data Technologies - Introduction.pptx
kuthubussaman1
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Shreyas Phanse Resume: Experienced Backend Engineer | Java • Spring Boot • Ka...
SHREYAS PHANSE
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
IAESIJAI
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
NewMind AI Monthly Chronicles - July 2025
NewMind AI
 

NetApp Se training storage grid webscale technical overview

  • 2. Student Guide & Internal & Confidential Update Daly https://goo.gl/VVmVZ0 Download Full Docs © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only2 5673-StorageGRID for Standard Enterprise Solution http://ouo.io/SROnEV Building Private Clouds on E-Series The Swift Object Opportunity http://ouo.io/vWrya Deployment Guide - Acuo Universal Clinical Platform with NetApp StorageGRID Integration Architecture http://ouo.io/TowHxU Deployment Guide - StorageGRID E-Series http://ouo.io/jhf1M Getting Started with StorageGRID CDMI Integrations http://ouo.io/dAORQj NetApp and the Object-Based Storage and Archiving Landscape http://ouo.io/wvoEgQ NetApp Distributed Content Repositories What Are We Doing in Real Life http://ouo.io/3y65M NetApp Portfolio Overview http://ouo.io/O9aKQ POC - StorageGRID Webscale Proof of Concept Guide http://ouo.io/pzljIH QRG_StorageGRID Webscale http://ouo.io/3dqRr SE Training - StorageGRID Webscale Technical Overview http://ouo.io/BqtKUU Solution Brief - NetApp StorageGRID Manage Large Pools of Mission-Critical Patient Data Across Healthcare Facilities http://ouo.io/KTDFz StorageGRID Webscale 10.0 Installation Overview http://ouo.io/hIYhJ StorageGRID Webscale Nonstop Object Storage for Enterprise and Cloud http://ouo.io/dMRoOr StorageGRID Webscale Object-Enabled Data Management http://ouo.io/nhuj4f Technical Report - Integration Guide for NetApp StorageGRID with McKesson Horizon Medical Imaging PACS http://ouo.io/9ncafj
  • 3. Agenda  What is Object Storage  StorageGRID Webscale Introduction  Technical Overview  Key Concepts  Data flow © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only3
  • 4. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only4 Object storage growth driven by macro trends… Unstructured data continues to grow •New sources of unstructured data growth media, mobile and applications •Data continues to be retained for long periods (archival, compliance etc) Unstructured data profile is changing •Simultaneous access to the same data is rarely required •Most data accessed a few times initially and then rarely accessed Data access is changing •Geographically dispersed access •Applications accessing data – don’t care about POSIX semantics, file locking Highly cost sensitive petabyte scale repositories •Driving tradeoffs between $/GB, latency, throughput, and data protection Storage being managed in a cloud ecosystem •Unified management & orchestration •Growth in cloud hosted applications that leverage object storage
  • 5. Introducing StorageGRID Webscale  A new variant of StorageGRID  Target object store (cloud, archive, media) use cases at massive scale  New Features  Protocols – native support for S3 API  Scalability – 100 billion objects, 70 PB  Simplicity – modular scalable resilient architecture, simplified deployment  Proven track record for reliability and innovation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only5
  • 6. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only6 Block File Object What is Object Storage? Different ways to address data Specific location on disks / memory Tracks Sectors Specific folder in fixed logical order File path File name Date Flexible container size Data and Metadata Unique ID
  • 7. Object Storage Example: File vs. Object © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only7 Object BasedFile Based ValetParking Garage Daily Garage 1 Floor 4 Row N Space 53 /users/jsmith/car/garage1/floor4/rown/space53.file C:UsersjsmithGarage1Floor4RowNSpace53.file Object UID 317
  • 8. Why Object storage? © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only8  Massive Scale  Billions of objects  Petabytes of data  Global namespace  Explosive growth  Respond to compliance and retention requirements  Cost effectively meet SLAs with intelligent data placement  Access from anywhere  Control access, security, and data integrity
  • 9. Object storage used across different workloads Emerging Object storage segments in the hybrid cloud © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only9 Service Providers (XaaS) Media Repository (Media Redistribution) • Large object sizes (+250MB) • Distributed repositories • High data rates for redistribution • Time to first byte latency < 50ms Secure multi-tenancy (billing, isolation, authentication, & self-service) Control plane APIs and workflow automation New apps requiring RESTful interfaces (S3, Swift) Web Data Repositories • Small object (~KB) performance • Extremely high transaction load • Searchable, scalable metadata • High object counts Data Archives • Long access latency tolerance • Integration to tape / Glacier • Long retention periods • Erasure coding Scale seamlessly Ease of install & management Global namespace Cost
  • 10. StorageGRID Webscale Object-Enabled Data Management © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only10
  • 11. NetApp StorageGRID Webscale Software Defined Object Storage  Billions of objects and multi-petabyte  Architected for massive scale Built for the Hybrid Cloud  Global, Always-On data availability and durability  Support for cloud applications (S3, CDMI)  10th Generation object store proven with product deployments Dynamic Policy Engine  Intelligent, policy driven data management for optimal availability, performance and cost over the life cycle of data © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only11
  • 12. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only12 StorageGRID Platform for Distributed Content Repositories StorageGRID Object Storage Software + E-Series Storage Array MULTIPLE: APPLICATIONS + SITES + PROTOCOLS MULTIPLE: TARGETS + TIERS MULTIPLE: TENANTS + POLICIES + ADMINISTRATORS Site 1 Site 2 … Site NSite 3 APPLICATIONS APPLICATIONS APPLICATIONS APPLICATION NetApp E-Series Tape
  • 13. StorageGRID Webscale features © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only13 Hardware Obsolescence Protection Non-Disruptive Operations Object Integrity and Security Multi-Tenancy Global Object Namespace Services Automation Manageability Reliability Scalability ILM, Metadata driven policies Seamless Scaling Audit & Reporting
  • 14. Technical Overview StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only14
  • 15. Solution Topology © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only15 • Clients can access global object namespace via Gateway Nodes or Storage Nodes E2760 DE6600 Administrators REST API clients 10GbE LAN Per Node Resource Requirement VMDK (GB) vCPU RAM (GB) 100 8 24 100 8 24 300 8 24 Storage Gateway Admin DE6600 E2760 DE6600 DE6600 • StorageGRID Webscale nodes running in VMware hosts S3, CDMIHTTPS 16Gb FC switch WAN Router • E2760 block-based storage • SSDs for read cache and VM datastores • Mixed disk types (SSD, SAS, NL-SAS) for tiered storage pools
  • 16. StorageGRID Architecture © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only16 App1 App2  Admin Nodes  Management Services: Configuration, Monitoring, Audit and Logging  Storage Nodes  Manages object storage including replication  API Gateway Nodes  Load balancing interface through which applications connect to the system  Archive Nodes  Interface to archive media storage such as tape DATA CENTER 1 DC2 DC3 Design the grid to scale for performance, capacity & resiliency STORAGE STORAGE ADMIN ARCHIVE API LOAD BALANCER
  • 17.  Use NetApp StorageGRID Webscale Designer to architect the grid to your requirements  Quickly deploy the grid via NetApp StorageGRID Webscale Installer  Adapt your grid to changing requirements  Add sites  Add nodes and capacity  Support rolling upgrades Centralized Deployment & Rolling Upgrades Design, deploy, and maintain configuration control © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only17 DC1 DC2 DC4 DC3
  • 18. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only18 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  • 19. Client Connections, Object Identifiers & Metadata © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only19 Client CDMI S3 Read/Write StorageGRID Webscale System HTTPs Client Connections Object Identifier Object type: JPG Date modified: 07/21/ 2014 GPS Coordinates: Lat, Long Location: DC @ Seattle Metadata
  • 20.  Scale beyond traditional application metadata to enable new capabilities  Flexibility:  Metadata is application defined  Up to 4096 fields can be created as requirements change (CDMI)  No lock-in to predefined schema  ILM engine evaluates the metadata and applies policies  S3 metadata available to the policy engine Extensive metadata management Metadata – Why it matters ?  Metadata is distributed throughout the grid  Increased scalability and resiliency  Faster retrieval and efficient ILM policy evaluation © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only20 Object Identifier: 00006FFD00192A1200555FFEE12039468EBF622 D9402C4F962 Locations: Location 1: Data Center 1/DC1-S1/LDR Location 2: Data Center 2/DC2-S3/LDR Metadata CDMI/CVTE: 0 CDMI/META: {"application":"finance","doctype":"contract", "project":"45667}
  • 21. Extensive metadata management Metadata Management  Metadata Includes – Object size, User Metadata, Bucket Name, Account Id  CDMI supports a large number of user metadata fields (up to 4096)  Policy support  Metadata can be used in polices (CDMI & S3)  No pre-configuration required for metadata, immediately available for policy use © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only21
  • 22. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only22 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  • 23. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only23 Sites (Link Cost Groups) Storage Pools StorageGRID Data Management Storage Pools Storage Grade TAPE Storage Grade SAS Storage Grade FLASH Munich, Germany San Francisco, USA Storage Node T1 Storage Node T2 Tape Storage Pool EU Storage Pool Storage Node S1Storage Node F1 Storage Node F2 Fast Storage Pool Vancouver, CAStorage Node T3Storage Node F3  Simple configuration of SLO based storage across multiple sites and storage grades
  • 24. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only24 Customer Defined Storage grades StorageGRID Data Management Associating Storage Nodes with Storage Grades Creating Storage Grades  Customers can configure their own Storage Grades, and associate them with specific nodes
  • 25. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only25 Site configuration – Not just a site StorageGRID Data Management  Customers can model network costs, creating powerful configurations  Possible Examples  Configure two sites to be treated as one,  Specify which sites are connected by high b/w pipes & low b/w pipes,  Prefer traffic to go in a particular direction
  • 26. Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only26 Client Connections Storage Pools Metadata Object Identifier ILM Policy
  • 27. Dynamic Policy Engine – An Overview Data management key concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only27  Manage policies not objects  Evaluate objects based on metadata such as:  Custom user application metadata  Method of ingest (S3 or CDMI)  Size of object  Last access time  Apply ILM rules to set:  Geography — Placement of an object  Storage grade — Type of storage used to store an object  Replication — Number of copies stored  Retention — Set time during which an object cannot be purged App1 Site1 Site2 Site3
  • 28. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only28 Policy Management Data Management Key Concepts ILM Policy When If.. then.. • At Ingest • Objects at rest (already ingested) • After a read (enabling caching) • Metadata matches specific criteria • Move/Copy to one or more Storage Pools
  • 29. ILM Rule – Specify the Conditions Data Management Key Concepts © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only29 When If..  Customers designate when a rule will be applied and what conditions must be met to trigger the action for placement and retention.
  • 30. ILM Rule – Defining data placement and retention © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only30 then..  Simple configuration of complex placement rules, that are graphically displayed for easy understanding Data Management Key Concepts
  • 31. Combine ILM rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only31 Data Management Key Concepts  Highly complex business logic can be implemented by combining multiple rules into a policy
  • 32. ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only32 Application writes to grid via S3 Metadata is evaluated DC1 DC2 DC3  Store objects with S3 metadata “Bucket Name = ClientX” on ingest at DC1 on SSD and DC2 on SATA for 90 days  After 90 days store on DC1 on SATA and DC3 on Tape 1 x copy DC1SSD 1 x copy DC2SATA 90 Days later…. 1 x copy DC1SATA 1 x copy DC3Tape Bucket Name = ClientX
  • 33. ILM Policy Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only33 DC1 DC2 DC3  What if requirements change?  A new Data Center is brought on line and now we must store a copy at DC4  We can edit the policy – and apply not just to new data, but re-evaluate existing data and create new replicas as needed  Without impact to the performance of the grid DC4
  • 34.  Create ILM Rules to set customized service levels  GOLD = 2 x copies on SSD, 1 x copy on SATA  SILVER = 1 x copy on SAS, 2 x SATA  BRONZE = 1 x copy on SATA, 1 x copy on tape Set custom service levels via metadata SLA Example © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only34 {"sla":"gold"} DC1 DC2 DC3
  • 35. Built-in Object Integrity Verification and Self Healing Durability at the object level  Digital fingerprint is calculated per object upon ingest  Interlocking layers of object-wide and sub-object level integrity protection  Object hash value  Content hash value  CRC checksum  HMAC message authentication digest © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only35 ID: 41E85A1D Data Metadata Data Metadata Fingerprint
  • 36. Built-in Object Integrity Verification and Self Healing Durability at the object level  Continuous verification: on ingest, retrieval, replication, migration and at rest  Object failing integrity test is automatically replaced with another copy © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only36
  • 37.  Real time audit feed can be used to create custom reports  Audit logs created in open format allows the use of partner products such as Splunk  Comprehensive audit feed for:  Chargeback and billing  Search integration  Custom reporting  Security diagnostics  Compliance events  Validate performance for SLAs Continuous and active monitoring Audit and Reporting © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only37
  • 38. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only38  Analyze system performance and activity at every level  Grid Wide
  • 39. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only39  Analyze system performance and activity at every level  Grid Wide  Site Specific
  • 40. System Performance Analysis Perform deep performance analysis to fine tune your infrastructure © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only40  Analyze system performance and activity at every level  Grid Wide  Site Specific  Service Level
  • 41. Data Flow Understanding how data flows through a StorageGRID Webscale system for different operations © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only41
  • 42. Data flow overview Basic topology and business rules © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 42  Three departments  Sales  Finance  Marketing  Two pools per department  Capacity pool  Performance pool  Three sites  Two datacenter sites  One remote office  Policy - Finance  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
  • 43. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Receives object write request from client along with custom metadata  Returns object ID to client and forms the file payload into an object  Packetization  Digital fingerprint  Compression (optional)  Encryption (optional) 43 0x05DFF4338ADCE6F5
  • 44. Object Ingest and Replication Transmitting objects from client to StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office 44  By default, creates semi-sync local copy for immediate redundancy  Metadata is stored and replicated  Replicate object as per ILM policy  Finance Policy Example  Ingest into local pool  After 1 day create copy in performance pool  After 30 days move copy to capacity poolData Center 1 Data Center 2
  • 45. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Replication Metadata-driven ILM with optimal resource utilization © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only  Optimal resources in target storage pools are selected for the replication destination  While honoring ILM the grid considers  Network costs  Server utilization  Storage utilization 45
  • 46. Sales Capacity Storage Pool Finance Performance Storage Pool Marketing Performance Storage Pool Finance Capacity Storage Pool Sales Performance Storage Pool Marketing Capacity Storage Pool Local Storage Pool WAN Satellite Office Data Center 1 Data Center 2 Object Retrieval Requesting objects by the client from StorageGRID Webscale © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only Receives object read request from client Determines optimal object location relative to request location Streams a copy of the object to the client and verifies integrity of the object on-the- fly 46
  • 47. Thank You © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only47
  • 48. Management Services Admin Node  NMS – Network Management System: Provides the administrative interface for configuration and monitoring of the grid.  CMN – Configuration Management Node: Manages system-wide configurations such as connection profiles, grid tasks, and system configuration options.  AMS - Audit Management System: Keeps logs of system activity and events. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only48
  • 49. Object, Metadata Storage and Replication Storage Nodes  LDR - Local Distribution Router: Stores, moves, verifies, and retrieves object data stored on disks  DDS - Distributed Data Store: Stores, replicates, and protects metadata in the key value store. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only49
  • 50. Object, Metadata Storage and Replication Storage Nodes - Continued  CMS - Content Management System: Manages object placement and replication based on ILM rules.  ADC - Administrative Domain Controller: Maintains topology information and provides authentication services. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only50
  • 51. Load Balancing and Client Connectivity API Gateway Nodes  CLB - Connection Load Balancer: Acts as switchboard for connecting clients to the most efficient LDR service for ingest and retrieval.  Ports  8081 CDMI  8082 S3 © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only51
  • 52. Data storage on archive media Archive Nodes  ARC – Archive: Communicates with archiving middleware to store and retrieve data to and from archive media such as tape. © 2014 NetApp, Inc. All rights reserved. NetApp Proprietary – Limited Use Only52

Editor's Notes

  • #12: Introducing NetApp’s Content Repository solution for large, unstructured distributed data sets – StorageGRID NetApp StorageGRID was designed from the ground up to solve Big Content challenges. StorageGRID offers a large, global content repository for unstructured data, capable of storing billions of data sets and managing dozens of petabytes of capacity. StorageGRID enables customers to create, manage and consume content globally, while providing complete location independence. With location independence you get all the benefits of local storage (meaning that you get the performance that you expect and access to all content you require) while providing a truly global namespace across hundreds of sites. Every site can be set up differently, from remote sites that simple have a virtual machine or smaller system that only stores small amounts of data locally, to large datacenters that retain a full set of all data sets. Remember though – this is all managed automatically with policies, metadata information and intelligent caching, no administrator intervention required. StorageGRID uses metadata-based management for data classification and access, meaning that StorageGRID manages where data is physically stored, how many copies exist (and where) for disaster recovery purposes, how long those copies are retained and when they are destroyed. Further, metadata-based access to your data means that instead of looking for a file name, you simple look for “Mortgage documents”, customer “John Doe”, account number “123456” – greatly simplifying how your applications interact with your storage.
  • #16: The CPU and RAM requirements are for production deployments. Lab \ Test configurations can be smaller: Admin 2CPU, 4GB RAM Gateway 2CPU 4GB RAM Storage 2CPU, 6GB RAM Storage recommendations: The best practice and recommended storage for StorageGRID Webscale is as follows: NetApp E-Series Storage System with capacity optimized NL-SAS in DDP SSDs for Storage Node system disks (100GB per node) SSDs for all Admin Node disks (about 300GB per node)