IBM Cloud Object Storage: How it works and typical use cases

IBM Cloud Object Storage:
How it works and typical use
cases
Tony Pearson
IBM Master Inventor,
Senior IT Management Consultant,
TechU Content Manager
2019 IBM Systems Technical University
10-12 Sep 2019 | Johannesburg, SA

Abstract
Can't afford to save your world's collection of data on your
IT budget?
This session explains why object storage is becoming so
popular and how it differs from block or file-level storage.
The speaker will provide an overview of IBM Cloud Object
Storage System and its clever, highly scalable, highly
reliable use of Erasure Coding to reduce storage costs by
up to 70 percent from traditional disk arrays
IBM Systems Technical University © Copyright IBM Corporation 2019 2

Agenda
Why Object Store?
What is IBM Cloud Object
Storage System?
How to use it: Applications
and Use Cases

How is Object Storage Different?
Block and File Storage
– Decide where to put it
• For block, which array/volume/LUN
• For file, which filer/subdirectory
– Remember where it is to get it back
– Don’t let anyone else move it
– Read/Write records, append data
– Limits on LUN size, number of files
Object Storage
– Provide data over to the Object storage
• Get “claim stub” reference locator
– Use or share “claim stub” to access data HTTP, Openstack
Swift, S3
– Get/Put/Delete object in its entirety
– Effectively “unlimited” scalability

Object Storage for Static and Stable data
Hot Data
High-IOPS and Low-Latency
All-Flash and Hybrid Flash/Disk
Information Lifecycle Management (ILM)
Structured data / Random-Access
Active logs and traces
Virtual Machines and VDI
Single-Tenancy
Static and Stable data
Backups, Files, Archives
Seismic, Research, Telemetry, HPC
Video, Animation, Body Cams
Photos, Images, CAD/CAM, GIS
Music, Audio
Genomic, Medical Images
Multi-tenancy
Object Store provides a
• Secure
• Reliable
• Scalable
• Cost Effective
Platform For Unstructured data
Object Store
is not designed for
• High IOPS workflows
• Transaction
Processing
• Inherent ILM

Object Storage is Simpler for Application Development
POSIX – over 60 commands NFS – over 30 commands
HEAD
• Read metadata
GET
• Read content and
metadata
PUT
• Write content and
metadata
DELETE
• Remove object or empty
container
POST
• Update metadata
Object – 5 commands
SMB – over 80 commands

Why Object Storage?
It enables my business
and reduces costs!
• Handles 80% of my data
• Scales easily, growth does
not impact my business
plan
• Less expensive than
traditional flash and disk
arrays
It is easy to deploy and
manage!
• Easily provision to users
and teams
• Location-independent,
Flat namespace
• Software-Defined or
Pre-built systems
• Ethernet-based and Cloud
friendly
• Online backup/archive
alternative to tape and
virtual tape libraries
It is great for
developing software!
• GET/PUT interface
reduces coding effort
• Robust metadata for
search of documents,
images, and videos
• Can be used for
Analytics, IoT, Mobile,
Web and Cloud
Applications
IT
Director
Software
Developer
Storage
Administrator

Agenda
Why Object Store?
Storage System?
and Use Cases

Moderate Performance, Moderate Cost
Unified file and
object storage.
Optimized for high
performance, across
flash and disk
Flash 15K
Unified file and object
storage on tape
Information Lifecycle
Management (ILM) across tiers
HighestPerformance
Lowest cost
Tape
IBM was looking to offer easy to
manage, scalable disk-based object
storage for unstructured data
• Moderate performance
• Moderate cost
10K 7200 rpm

RAID versus Erasure Coding
Tolerate 1 drive failure Tolerate 2 drive failures Tolerate “M” failures
RAID-1 / RAID-10
K pieces : 2 x K slices
RAID-5
K pieces : K + 1 slices
2.0X
1.2X
3.0X
1.5X
1.7X
Triplication
K pieces : 3 x K slices
RAID-6
K pieces : K + 2 slices
Erasure Coding
K pieces : K+M = N slices

Data Growth at Petabyte (PB) Scale
PB of data
3 to 5x
Data Protection
RAID, Mirrors,
Replication, Tape
Data Protection
High Availability & Disaster Recovery
Geo-Distribution & Erasure Coding
Infrastructure
Proprietary, specialized
hardware, multiple systems
Operations
More than 1 FTE per PB
Maintenance outages
Infrastructure
Software Defined,
Commodity Hardware,
Single System
Operations
Less than 1 FTE per 6 PB
Single system, Secure
Self-healing
1.7 x 60% Less
Hardware &
Rack space
Traditional
Approach
IBM Cloud
Object Storage
Approach
70%
Lower
TCO

IBM Cloud Object Storage System economics beat
legacy NAS storage and Amazon Web Services (AWS)
$8,400
$4,210
$1,613
$1,053
Legacy NAS DR protected
Legacy NAS single copy
IBM Cloud Object + NAS
gateway
dsNet object protected
IBM Cloud Object
IBM Cloud Object vs NAS
Cost: 80% lower
$0
$2,000,000
$4,000,000
$6,000,000
$8,000,000
$10,000,000
480 TB 960 TB 1920 TB 3840 TB
Cost: 10 to 60% lower
IBM Cloud Object vs AWS S3
IBM Cloud Object
Amazon S3

IBM Cloud Object Storage System – Terminology
HTTP
Application
Server
IBM COS
Manager
Accesser
Slicestor
Device
Set
Accesser Pool
Vault
Load
Balancer
Site A Site B Site C
End user
Global Namespace
IBM Cloud Object Storage
System organizes objects into
“Vaults” and “Containers”
Storage Pools can hold multiple
vaults
Vaults can have millions of
containers
OpenStack Swift
S3 Protocol

Global Manager UI Changes – Top Navigation
New changes in the IBM Cloud Object Storage control portal (Manager) make the visual experience
consistent with other IBM Storage products
Top navigation bar is now textual based main navigation with icons for secondary information

The Math Behind Reed-Solomon Erasure Coding
“K” variables of input data: a, b, c, d, e
Example: K=5 M=3 K+M= 8
Here we create “K+M”
equations, adding and
subtracting by different
co-factors
Results in “K+M” slices that
can be geographically
dispersed
We can tolerate losing up to
“M” slices of data, and still
solve for the original
“K” pieces of data.

Information Dispersal Algorithm (IDA)
01 02 03 04 05 06 07 08 09 10 11 12
A B C D
E F G
IDA is represented as Width/Read/Write
This example is 12/7/9
Original data is received by
IBM COS Accesser and
chopped up into 7 pieces
(Read threshold)
Erasure Coding:
7 variables in,
12 equation results out
(Width)
12 slices written in parallel
to IBM COS Slicestor devices
A B C D
E F G
IBM SmartWrite™
At least 9 slices (Write Threshold) must be written for
the I/O to be considered complete, rest will be written
later. IBM COS periodically scans and rebuilds slices as
needed
IBM SmartRead™
To read the data back, only 7
slices are used to solve 7
equations (Read threshold) Original data is
re-assembled

Dedicated
(Private)
IBM Cloud Object Storage System –
Single-site, Two-site and Multi-Site
Single Site
Better performance,
when site-tolerance
not a factor, better
than traditional
RAID-5 / RAID-6
Two Site Vault Mirroring
Allows customers to leverage existing two-site
infrastructure. Provides concurrent reads and
writes despite communication disruption
between locations. Local data better than
traditional RAID-5 / RAID-6
Geographically Dispersed
Three to Nine Sites
Consider adding IBM Cloud or leverage existing
datacenter locations to provide a broader
distribution of data for higher availability, site-
tolerance and scalability

Two Dispersal Modes
— Standard Dispersal
— One or more Accessers
— Each slice on its own
Slicestor
— Concentrated Dispersal (CD Mode)
— Entry Level System (1-3 sites)
— Accessers can be embedded into Slicestor
(Optional)
— Each slice stored on separate HDD within
Slicestor for availability
01 02 03
04 05 06
07 08 09
10 11 12
01 02
03 04
05 06
07 08
09 10
11 12

Scalability
Site 1 Site 2 Site 3
S3 Compatible API Openstack Swift
Compatible API
Simple Object API
Slicestor
Software
Accesser
Software
Scalability – Scale performance and/or capacity at any time
with no downtime to operations
Need more Performance?
Add more Accesser nodes
Need more Capacity?
Add more disks to existing Slicestor nodes, add more device
sets to existing storage pools, or add more storage pools
–All nodes in single device set must have same amount of
storage
• New device sets can have different size drives than existing drives
sets
–All device sets in a storage pool must have the same
width
• Standard and Concentrated IDA can be intermixed in same storage
pool
–Different storage pools can have different widths
Device
Set 1
Device
Set 2
Device
Set 3

Encryption Options for IBM Cloud Object Storage
A
A B C D
E F G
Data
A
A#
AONT
Data
SecureSlice™
System-generated key
Used to encrypt data
Data
A
#
Hash taken of
encrypted data
Key wrapped with hash makes
All-or-Nothing Transform (AONT)
Information
Dispersal
Algorithm (IDA)
User sends data
Data
B
Data
B
Server-Side Encryption with Customer key
System-generated key used to encrypt data
Key wrapped with customer-supplied key
User sends data and key
SSE-C
Data
CData
C AMK
Application-managed key
User sends encrypted data
Encryption disabled in IBM COS

Compliance-Enabled Vault
Standard
Vault
Compliance
Enabled
Vault
Government Regulations specify that “electronic storage
must preserve the records exclusively in a non-erasable,
non-rewriteable, format” (SEC 17a-4)
Various Terms have been used:
• Write Once Read Many (WORM)
for tape and optical media
• Fixed-Content or Content-Addressable for disk-
based storage
• Immutable for file systems
Non-erasable, Non-rewriteable (NENR) adopted to
cover all of the above
Compliance-Enabled Vaults prevent objects from being
deleted or modified for a specified retention period
Vaults can
co-exist in same
storage pool

Compliance-Enabled Objects in Container Mode - Features
IBM Cloud Object Storage now supports Compliance-Enabled Objects (NENR/WORM) in the
Container Mode configuration
 Retention Policy (min, max, default) - per bucket
 Retention Period - per object level
 Extending Retention - per object
 Temporary Legal Hold - per object
 Event based Legal Hold (indefinite) – per object
 Application owned & initiated Event trigger -per object
 Permanent Retention - per bucket/object
 UI support
 API support
 SDK support

The Global Data Privacy Regulation (GDPR) Readiness
A comprehensive readiness program took place to ensure the enablement of IBM Cloud Object
Storage to be used by customers processing European Union citizen personal data
— New features added to handle retention and redaction of log data and enhance security access
controls
— Risk assessments of data processing activities completed
— Potential IBM hardware parts with EU customer data identified for additional destruction and
secure handling

IBM Cloud Object Storage System –
Software, Pre-Built Systems or Cloud Services
Software
• ClevOS – IBM Cloud Object
Storage System software packaged
with Debian Linux OS
• Software-defined, hardware-
aware model for flexibility of x86
platform choice
• Can be deployed on qualified
vendor hardware
Pre-built Systems
• Fully integrated appliance
models for easy
deployment and support
• On-premises object storage
solution
Cloud Services
• IBM Cloud Object Storage
System as a service for
customers that want security
and controlled performance

IBM Spectrum Storage Suite
Bundle license for flash,
disk, tape and storage-
rich servers
Per TiB
IBM Spectrum
Storage Suite
Control Protect
Archive
Virtualize Accelerate
Scale
I’ve deployed Spectrum Storage Suite. Now
what?
1. Well, since you already own entitlement
to all capabilities in the Spectrum Storage
suite, you are free to download any of the
software you like, including IBM Cloud
Object Storage!
2. To help facilitate quick adoption of
additional use cases, Spectrum Storage
Suite licensing offers the ability to
perform extended tests in an evaluation
sandbox proving ground without
additional charge. Tebibytes (TiB) in this
sandbox are not counted.
So go ahead, experiment with your next use
case. Prove it, become familiar with it, pay for
it only when it’s deployed for productive use.

Spectrum Discover Integration - Architecture
IBM Cloud Object Storage now supports integration with IBM’s Spectrum Discover through
metadata scanning and event notifications
October 11, 2018 IBM & IBM Business Partner Internal Use Only
Scanning and
Event Notifications
I BM Spectrum
Discover
File and Object Storage Data Activation/OptimizationData Insight
Analyze
Governance
Optimize
•Data discovery
•Dataset identification
•Data pipeline progression
•Data inspection
•Data classification
•Data clean-up
•Archive / tiering
•Duplicate data removal
•Trivial data removal
Use
Cases
•Simple to deploy
(VMware virtual appliance)
•Metadata curation
•Custom metadata tagging
•Automatic indexing
•Policy-Engine
•Action Agent API
Reporting DashboardSearch
Planned for 2019 and other 3rd parties

IBM Cloud Object Storage public cloud services
IBM Cloud Object Storage Flex
DataAccessPattern
Standard
IBM Cloud Object
Storage
Vault
Variable data access
pattern / Mixed
workloads
Predictable
data access
pattern /
workload
Frequent
data access
Less frequent
data access
Minimal
data access
IBM Cloud Object
Storage
Cold Vault
• High Availability – Cross Regional USA
• Low Latency – Regional (Dallas)

Agenda
Why Object Store?
Storage System?
and Use Cases

A broad spectrum of strategies to build, buy, or assemble an end to end solution
Integrations, Connectors, Gateways Ready to RunCustom Applications
Off-PremisesHybridOn-Premises
and/or and/or
How to Consume IBM Cloud Object Storage

Large
multimedia
files
Video, Audio,
Images
Store
telemetric,
instrumentatio
n and sensor
data
Web generated
and social
media data
Backup, Active
archive, DR,
inactive and
compliance
data
Security
audits, Log
files and
Analytics
Collaboration
and content
distribution
Global file
services,
mobile access,
and StaaS
Genomics,
HPC, Geo-
spatiial
8
7
6
5
4
1
2
3
Object Storage Typical Use Cases Storage Services

Market Industry Verticals
Secure
customer
trust and
business
compliance
.
Financial
Services &
Insurance
Incorporate
into IoT
processes,
analytics
Industrial,
Distribution
Manage the
data essential
to serving the
good of the
public.
Public Sector,
Government
Scale your
market
offering
without
worry.
Telco &
Service
Providers
Create and
Deliver
Content on
demand.
Media &
Entertainment
Heath Care &
Life Sciences
Put
medical
progress
before
everything
else.
Every industry does backup, all have some
amount of static-and-stable unstructured data,
most need file or content archive…

Client Experience – IBM Cloud Object Storage
Users upload photo
and video content via
web based application
Photo and video objects are sent to
IBM COS via REST based protocols
Metadata is
captured and
stored
• Scale – 150 petabytes and growing: more than 50 Billion images stored
• Manageability – 3 Administrators manage entire environment
• Security – 50,000+ uploads per minute with zero touch security
• Always-on availability – SLA of 100% download on demand – even during California to
Nevada datacenter move
• Economics – Operating costs reduced by more than 70%
• Key decision makers – Technical team backed by financial cost cutting mandates

IBM Spectrum Protect –
Cloud and Object Store Capabilities
Client nodes
Off-premises:
• IBM Cloud
• Amazon Web Services
• Microsoft Azure
On-premises
IBM Spectrum
Protect Server
• IBM Cloud Object Storage System
(using S3-compatible API)

IBM Spectrum Storage and IBM Cloud Object
storage. Optimized for
high performance, across
flash, disk and object
store
Flash
Object
Store
15K
Object storage on disk
( File, backup and archive interfaces
available through variety of options )
IBM Cloud
Amazon Web Services
Microsoft Azure
Swift S3 emulation
OpenStack Swift
storage on tape
Transparent Cloud Tiering
Information Lifecycle
Management (ILM) across tiers
HighestPerformance
Lowest cost 
Tape10K 7200

Summary
Object Storage scales easily to
handle your unstructured data, easy
to manage, ideal for software
development
IBM is the #1 leading vendor of
Object Storage, with over 400
patents, and several clients >100PB
IBM Cloud Object Storage can be
used on-premises, hybrid, or in the
Cloud
Integrates with the rest of the IBM
Spectrum Storage suite

Thank you!
Tony Pearson
tpearson@us.ibm.com
+1-520-799-4309
Please complete the Session
Evaluation!

Special Thanks
I would like to thank the following colleagues
who contributed charts, insights, and review
comments for these presentation materials
— John Shubeck
— Chris Maestas

COS Knowledge Center – Overview
— This feature creates a public COS Knowledge Center (KC) at
https://www.ibm.com/support/knowledgecenter/STXNRM for on-premises documentation.
— This feature embeds a standalone KC application in the Manager UI for use as a help system. The
embedded KC replaces the current contextual help and Help Index.
P R E - K N O W L E D G E C E N T E R P O S T - K N O W L E D G E C E N T E R
— Access issues to documentation
— No search capability
— Help content and product documentation
can be out of sync
— Non-Compliance with IBM Documentation
Standards
— Slow documentation update time
— Publicly accessible
— Enterprise search capability
— All document versions housed in one place
— Consistent with other IBM product
documentation
— Shorter times for documentation updates
— Mobile support

About the Speaker
39
Tony Pearson is a Master Inventor, Senior IT Management Consultant, and Content Manager for the
IBM Systems Technical University events. Tony joined IBM Corporation in 1986 in Tucson, Arizona,
USA, and has lived there ever since. Tony presents briefings on storage topics covering the entire IBM
Storage product line, IBM Spectrum Storage software products, and topics related to Cloud Computing,
Analytics and Cognitive Solutions. He interacts with clients, speaks at conferences and events, and
leads client workshops to help clients with strategic planning for IBM’s integrated set of storage
management software, hardware, and virtualization solutions.
Tony writes the “Inside System Storage” blog, which is read by thousands of clients, IBM sales reps and
IBM Business Partners every week. This blog was rated one of the top 10 blogs for the IT storage
industry by “Networking World” magazine, and #1 most read IBM blog on IBM’s developerWorks. The
blog has been published in series of books, Inside System Storage: Volume I through V.
Over the past years, Tony has worked in development, marketing and consulting for various IBM
Systems hardware and software products. Tony has a Bachelor of Science degree in Software
Engineering, and a Master of Science degree in Electrical Engineering, both from the University of
Arizona. Tony is an inventor or co-inventor of 19 patents in the field of IBM Systems and electronic data
storage.
9000 S. Rita Road
Bldg 9032 Floor 1
Tucson, AZ 85744
+1 520-799-4309 (Office)
tpearson@us.ibm.com
Tony Pearson
Master Inventor
Senior Management
Consultant, IBM Systems
La Services
IBM Storage
IBM Systems Technical University © Copyright IBM Corporation 2019

My Social Media Presence
Blog*:
ibm.co/Pearson
LinkedIn:
https://www.linkedin.com/in/az990tony
Books:
www.lulu.com/spotlight/990_tony
IBM Expert Network on Slideshare:
www.slideshare.net/az990tony
Twitter:
twitter.com/az990tony
Facebook:
www.facebook.com/tony.pearson.16121
Instagram:
www.instagram.com/az990tony/
Email:
tpearson@us.ibm.com
* Not a typo. This is short URL for https://www.ibm.com/developerworks/mydeveloperworks/blogs/InsideSystemStorage/

Notices and disclaimers
— © 2019 International Business Machines Corporation. No part of
this document may be reproduced or transmitted in any form
without written permission from IBM.
— U.S. Government Users Restricted Rights — use, duplication or
disclosure restricted by GSA ADP Schedule Contract with IBM.
— Information in these presentations (including information
relating to products that have not yet been announced by IBM)
has been reviewed for accuracy as of the date of
initial publication and could include unintentional technical or
typographical errors. IBM shall have no responsibility to update
this information. This document is distributed “as is” without
any warranty, either express or implied. In no event, shall IBM
be liable for any damage arising from the use of this
information, including but not limited to, loss of data, business
interruption, loss of profit or loss of opportunity.
IBM products and services are warranted per the terms and
conditions of the agreements under which they are provided.
— IBM products are manufactured from new parts or new and used
parts.
In some cases, a product may not be new and may have been
previously installed. Regardless, our warranty terms apply.”
— Any statements regarding IBM's future direction, intent or
product plans are subject to change or withdrawal without
notice.
— Performance data contained herein was generally obtained in a
controlled, isolated environments. Customer examples are
presented as illustrations of how those
— customers have used IBM products and the results they may have
achieved. Actual performance, cost, savings or other results in
other operating environments may vary.
— References in this document to IBM products, programs, or
services does not imply that IBM intends to make such products,
programs or services available in all countries in which
IBM operates or does business.
— Workshops, sessions and associated materials may have been
prepared by independent session speakers, and do not necessarily
reflect the views of IBM. All materials and discussions are provided
for informational purposes only, and are neither intended to, nor
shall constitute legal or other guidance or advice to any individual
participant or their specific situation.
— It is the customer’s responsibility to insure its own compliance
with legal requirements and to obtain advice of competent legal
counsel as to the identification and interpretation of any
relevant laws and regulatory requirements that may affect the
customer’s business and any actions the customer may need to
take to comply with such laws. IBM does not provide legal advice
or represent or warrant that its services or products will ensure that
the customer follows any law.

Notices and disclaimers continued
— Information concerning non-IBM products was obtained from the suppliers
of those products, their published announcements or other publicly
available sources. IBM has not tested those products about this publication
and cannot confirm the accuracy of performance, compatibility or any other
claims related to non-IBM products. Questions on the capabilities of non-
IBM products should be addressed to the suppliers of those products.
IBM does not warrant the quality of any third-party products, or the ability of
any such third-party products to interoperate with IBM’s products. IBM
expressly disclaims all warranties, expressed or implied, including but
not limited to, the implied warranties of merchantability and fitness for a
purpose.
— The provision of the information contained herein is not intended to, and
does not, grant any right or license under any IBM patents, copyrights,
trademarks or other intellectual property right.
— IBM, the IBM logo, ibm.com and [names of other referenced
IBM products and services used in the presentation] are
trademarks of International Business Machines Corporation,
registered in many jurisdictions worldwide. Other product and
service names might be trademarks of IBM or other
companies. A current list of IBM trademarks is available on
the Web at "Copyright and trademark information" at:
www.ibm.com/legal/copytrade.shtml

This presentation uses the IBM Plex™ font
IBM Plex™ is our new typeface. It’s global, it’s versatile and it’s
distinctly IBM.
IBM Plex
Sans
The IBM company is freeing itself from the cold, modernist cliché
and replacing Helvetica with a new corporate typeface. Also
replaces Arial, Calibri, Lucida Grande, Trebuchet, etc.
IBM Plex
Mono
A little something for developers. Replaces
Courier New, Letter Gothic, Lucida Console, etc.
IBM Plex
Serif
A hybrid of the third kind (combining the best of Plex, Bodoni,
and Janson into a contemporary serif). Replaces Cambria,
Garamond, Lucida Bright, Times New Roman, etc.
IBM Plex is freely available as TrueType and OpenType at: https://github.com/IBM/plex/releases
and looks consistently good across Windows, Linux and Mac

IBM Cloud Object Storage: How it works and typical use cases

More Related Content

What's hot (20)

Similar to IBM Cloud Object Storage: How it works and typical use cases (20)

More from Tony Pearson (20)

Recently uploaded (20)

IBM Cloud Object Storage: How it works and typical use cases