IBM Cloud Object Storage
- 2. Agenda • ICOS Overview • Storage Classes • Resiliency Options • End points • Access Policies • Service credentials and HMAC credentials • Firewalls & Encryption • Aspera High Speed Transfer • Lifecycle Rules: Expiration and Archival • Immutable Object Storage • IBM Cloud SQL Query
- 3. ICOS Overview • Formerly known as Cleversafe. • IBM COS supports objects up to 10 TB, and maximum of 100 buckets. • S3 API support is available in order to provide compatibility to standalone clients for AWS S3 storage. • IBM COS is IAM enabled. • We can enable Activity tracker based API logging for Each bucket level management and data events
- 4. Storage Classes Four storage Classes: • Standard :Used for active workloads , no retrieval fee • Vault: Used for Cold data and retrieval fee applicable • Cold Vault: Used for cold data , not accessed for more than 90 days . More retrieval fee applicable • Flex: Used for dynamic workloads with no predictable usage patterns
- 5. Resiliency Options Three types of resiliency/replication provided: Cross-Region ( Data replicated across three regions in a geography) Regional ( Data is replicated across three AZs in a region) Single Datacenter ( Data is replicated across multiple servers in the same location)
- 6. End Points • ICOS supports private and public end points. • VPC endpoints can connect to ICOS using a separate direct end points privately . • There are different end points for Regional , Cross-regional and datacenter locations. • Regional End Points for US-South Region example: Public: s3.us-south.cloud-object-storage.appdomain.cloud Private: s3.private.us-south.cloud-object-storage.appdomain.cloud Direct: s3.direct.us-south.cloud-object-storage.appdomain.cloud
- 7. Access Policy • Every user that accesses the IBM® Cloud Object Storage service in your account must be assigned an access policy with an IAM user role pre-defined ( Platform management and service access) • There is no bucket resource level permission option other than through IAM method. • Using IAM access policies , permissions can be granted at individual bucket level. • Public access can be granted by clicking on "access policy" inside bucket configuration
- 8. Service and HMAC credentials • A service credential provides the necessary information to connect an application to Object Storage packaged in a JSON document. • "Service credentials" option under object storage tab allows to create service id and associate privileges for all the buckets in the storage service along with end point details in a json document. • When a service credential is created, the underlying Service ID is granted a role on the entire instance of Object Storage. • If the intention that the credential be used to grant, access to a subset of buckets and not the entire instance, this policy needs to be edited. • HMAC credentials contains an access key and secret access key which is compatible to AWS S3 API. • HMAC credentials can be generated as part of "service credentials" option
- 9. Firewalls and Encryption • We can set up firewall by allowing certain limited number of IPs to access the bucket. • Once the firewall is setup , other IBM coud services can't access the bucket privately. • The objects are encrypted by default at rest with automatic provider side Advanced Encryption Standard (AES) 256-bit encryption and Secure Hash Algorithm (SHA)- 256 hash. • IBM Cloud Object storage provides option to encrypt through customer provided keys which is called server side encryption with customer provided keys (SSE-C) and also through SSE-KP (Server side encryption with IBM Key protect)
- 10. Aspera High-Speed Transfer • Aspera High Speed transfer allows transfers larger than 200 MB through console using proprietary FASP ( Fast and secure Protocol) • Aspera High Speed transfer requires either a browser plug-in or a desktop agent • Aspera High Speed transfer supports Java and Python SDKs • Aspera High Speed transfer supports windows, Ubuntu Linux and Mac OS agents
- 11. Lifecycle Rules: Expiration , Archival • Expiration rule makes the objects deleted automatically after given number of days from object creation. • IBM Cloud object storage archive is a low cost option for data that is rarely accessed. • You can transition data from any storage class ( Standard , Vault, Cold Vault ,Flex) to Archive. • For immediate archival , the archival time should be set to 0 days. • To access the data that is archived , it should be restored by specifying the period of which the object should be kept in the original class. • The restoration duration can be up to 12 hours • Together Expiration and Archive policies , we can set up to 1000 life cycle policies
- 12. Immutable Object Storage • Immutable Object Storage preserves electronic records and maintains data integrity. • Retention policies ensure that data is stored in a WORM (Write-Once-Read-Many), non- erasable and non-rewritable manner. • Retention Policies allows prevention of deletion of object within specified time. • Retention policies once enabled, can't be disabled • Retention policy can be set while uploading an object as well but the specified value should be within minimum and maximum value set at the bucket level. • The default retention period can be set at the bucket configuration. • Enabling "Permanent retention" at bucket level ,never allows objects deletion
- 13. IBM Cloud SQL • IBM Cloud SQL is a fully managed service which allows to run "SELECT" statements on object storage files of ORC, CSV, JSON format. • The query results are stored in a CSV file in the object storage. • Actions with Cloud SQL such as CREATE, DELETE, INSERT, and UPDATE are not possible.