Data storage in clouds
Download as PPTX, PDF3 likes1,682 views
This document details the 12th meetup of the Athens OpenStack User Group, discussing various OpenStack storage services like Glance, Cinder, and Swift, along with their functionalities and architectures. It also highlights additional projects like Ceph and GlusterFS, emphasizing the importance of scalable, software-defined storage systems. The document includes information on community involvement, upcoming events, and resources for contributing to OpenStack development.
Data storage in clouds
- 1. Data storage in clouds Athens OpenStack User Group #OSATH 12th Meetup, 22th April 2015 Thanassis Parathyras, aparathyras@stackmasters.eu, @parathyras
- 2. Announcements #OSATH has 200 members ! Greek Mailing List • openstack-el@lists.openstack.org • Join at http://lists.openstack.org OpenStack Summit (https://www.openstack.org) • 18-22 May, Vancouver OpenStack CEE Day (http://openstackceeday.com) • 8 June, Budapest
- 3. Outline • Storage types • OpenStack Storage Services – Glance – Cinder – Swift • Higher abstractions – Data Processing aaS (Sahara) – DBaaS (Trove) • Other OSS projects – GlusterFS – Ceph
- 4. It’s all about DATA “often called storage or memory, is a technology consisting of computer components and recording media used to retain digital data. It is a core function and fundamental component of computers.” ent. Computer data storage, Wikipedia
- 5. File Disk Drive CD-ROM Object Volume Image Swift Cinder Glance Amazon S3 Amazon EBS AWS Marketplace – Software Infrastructure Storage types
- 6. Glance OpenStack Image service • REST API • CRUD and Search features • Caching and prefetching • Supports several formats: – raw, qcow2, vmdk, vhd, ami (aki, ari), iso, vdi • Containers: – bare, ovf, ami (aki, ari) • Backend to storage (default: Swift) • Able to aggregate multiple back-ends – Can also increase availability
- 7. Cinder OpenStack Block Storage service • Volumes • Snapshots • Backups • Modular architecture to support 50+ back-ends • LVM is the default (iSCSI) • Diverse storage types – iSCSI, Fibre, RBD (Ceph), NFS, GPFS, … • Manages storage resources separately Ephemeral vs Block Storage • Nova manages ephemeral storage coupled with VM state (non-persistent) • Cinder manages block storage decoupled from VM lifecylce (persistent)
- 8. Swift OpenStack Object Storage service • REST API • Data redundancy (3x or more) • Drive auditing built-in • RAID not required • Commodity hardware (not low-end) • High availability • Distributed • Eventual consistency CAP theorem • Choose 2 out of 3 (Consistency, Availability, Partition Tolerance) • Swift implements an eventual consistency model
- 9. Swift in action Access tier • Proxy server Storage tier • Account server • Container server • Object server • Consistency servers (Auditors, Updaters, Replicators) Zones • Selected per deployment • Determine replica isolation (disk, server, rack, room)
- 10. Swift in detail Upload • PUT http://<swift_url>/<acc>/<cont>/fileA Internals • Consistent hashing – DHT • Hash function: md5sum Ring • Static mapping to direct data location • Zones, disks, partitions and replicas • Across every node in the cluster 1 2 3 45 6 7 8 partition
- 11. Swift in detail Partition power • Estimate 100 partitions per disk at max capacity • Calculate closest power of 2 rounded up • 2partition_power=partition number Disks and partitions • Swift partition <> filesystem partitions • Presume 4 disks named A, B, C, D A A B BC C D D partition
- 12. Putting it all together A. Spawn a new VM from an Image B. Attach volume(s) to VM C. Store application objects D. Retrieve stored objects
- 13. Lifting more … data OpenStack services moving up to stack • Infrastructure (image, volume, snapshot, backup) • Platform (database, analytics) Trove – DBaaS • MySQL, Percona, MariaDB, MongoDB, Couchbase, Cassandra, Redis, PostgreSQL, Oracle • Equivelant to Amazon {RDS, DynamoDB} • Backup/Restore, Resize, Replication, User/DB management, etc. Sahara – Big data analytics • Hadoop from Hortonworks or Cloudera, Spark • Amazon Elastic MapReduce (EMR) • Manage and configure cluster, HDFS, MapReduce
- 14. OSS storage projects Lots of them, common goals to support scalable, large, software- defined storage systems Ceph • Based on RADOS and CRUSH provides object, block and file-system storage • 10+ years development effort • http://www.meetup.com/Ceph-Athens/ GlusterFS • Simple to use scale-out storage provides unified access to files and objects • Data stored in native format, no metadata completely algorithmic
- 15. Get involved • Documentation – http://docs.openstack.org • Join the community – http://www.openstack.org/community • Greek mailing list – http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-el • Contribute – code (development, blueprints, reviews, bugs) – docs writing, translations, infrastructure support
- 16. Thank you! Athens OpenStack User Group #OSATH http://www.meetup.com/Athens-OpenStack-User-Group Thanassis Parathyras aparathyras@stackmasters.eu, @parathyras