Storing VMs with Cinder and Ceph RBD.pdf
6 likes4,010 views
The document discusses different ways of storing virtual machine images and volumes using Ceph storage. It describes growing from using proprietary storage hardware to using Ceph with its open source, scalable, and fault-tolerant design. It outlines how virtual machine images can be stored in Ceph using the Cinder volume driver and RADOS Block Device (RBD) for efficient cloning and live migration. Client access to volumes is provided through a Linux kernel module while container/VM access is through device drivers.
![C D
C D
C D
C D
C D
HUMAN !! C D
[DEVELOPER] C D
C D
C D
C D
C D
C D
54](https://image.slidesharecdn.com/storing-vms-with-cinder-and-ceph-rbdpdf2743-121018165253-phpapp02/85/Storing-VMs-with-Cinder-and-Ceph-RBD-pdf-5-320.jpg)
Storing VMs with Cinder and Ceph RBD.pdf
- 1. Storing VMs with Cinder and Ceph RBD
- 2. Growing With Hardware Appliances C D First PB C D Second PB C D • Proprietary C D • Proprietary storage C D storage hardware C D hardware C D • Well-known C D • Same storage C D storage vendor C D vendor C D C D C D C D $14 b’zillion Another C D C D C D C D $14 b’zillion C D C D C D C D C D C D 47
- 3. C D C D C C D C D D C D C D C++ C D C D C D C D C D 52
- 4. X C D C D C C D C D D C D C D C++ C D C D C D C D C D 53
- 5. C D C D C D C D C D HUMAN !! C D [DEVELOPER] C D C D C D C D C D C D 54
- 6. Hard Drives Are Tiny Record Players and They Fail Often jon_a_ross, Flickr / CC BY 2.0 71
- 7. D D D D D D = D D x 1 MILLION 55 times / day 72
- 8. 73
- 9. philosophy design OPEN SOURCE SCALABLE COMMUNITY-FOCUSED NO SINGLE POINT OF FAILURE SOFTWARE BASED SELF-MANAGING
- 10. APP APP HOST/VM CLIENT RADOSGW RBD CEPH FS LIBRADOS A bucket-based REST A reliable and fully- A POSIX-compliant A library allowing gateway, compatible distributed block distributed file apps to directly with S3 and Swift device, with a Linux system, with a Linux access RADOS, kernel client and a kernel client and with support for QEMU/KVM driver support for FUSE C, C++, Java, Python, Ruby, and PHP RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 79
- 11. OSD OSD OSD OSD OSD btrfs FS FS FS FS FS xfs ext4 DISK DISK DISK DISK DISK M M M 81
- 12. HUMAN M M M 82
- 13. Monitors: M • Maintain cluster map • Provide consensus for distributed decision-making • Must have an odd number • These do not serve stored objects to clients OSDs: • One per disk (recommended) • At least three in a cluster • Serve stored objects to clients • Intelligently peer to perform replication tasks • Supports object classes 83
- 14. C D C D C D C D C D ?? APP C D C D C D C D C D C D C D
- 15. C D C D C D C D C D APP C D C D C D C D C D C D C D
- 16. C D C D A-G C D C D C D H-N F APP * C D C D C D O-T C D C D C D U-Z C D
- 17. 10 10 01 01 10 10 01 11 01 10 hash(object name) % num pg 10 10 01 01 10 10 01 11 01 10 CRUSH(pg, cluster state, rule set) 107
- 18. 10 10 01 01 10 10 01 11 01 10 10 10 01 01 10 10 01 11 01 10 108
- 19. CRUSH • Pseudo-random placement algorithm • Ensures even distribution • Repeatable, deterministic • Rule-based configuration • Replica count • Infrastructure topology • Weighting 109
- 20. CLIENT ?? 110
- 21. 112
- 22. CLIENT ?? 113
- 23. 111
- 24. APP APP HOST/VM CLIENT RADOSGW RBD CEPH FS LIBRADOS A bucket-based REST A reliable and fully- A POSIX-compliant A library allowing gateway, compatible distributed block distributed file apps to directly with S3 and Swift device, with a Linux system, with a Linux access RADOS, kernel client and a kernel client and with support for QEMU/KVM driver support for FUSE C, C++, Java, Python, Ruby, and PHP RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 84
- 25. APP LIBRADOS native M M M 85
- 26. LIBRADOS L • Provides direct access to RADOS for applications • C, C++, Python, PHP, Java • No HTTP overhead
- 27. APP APP HOST/VM CLIENT RADOSGW RBD CEPH FS LIBRADOS A bucket-based REST A reliable and fully- A POSIX-compliant A library allowing gateway, compatible distributed block distributed file apps to directly with S3 and Swift device, with a Linux system, with a Linux access RADOS, kernel client and a kernel client and with support for QEMU/KVM driver support for FUSE C, C++, Java, Python, Ruby, and PHP RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 87
- 28. APP APP REST RADOSGW RADOSGW LIBRADOS LIBRADOS native M M M 88
- 29. RADOS Gateway: • REST-based interface to RADOS • Supports buckets, accounting • Compatible with S3 and Swift applications 89
- 30. APP APP HOST/VM CLIENT RADOSGW RBD CEPH FS LIBRADOS A bucket-based REST A reliable and fully- A POSIX-compliant A library allowing gateway, compatible distributed block distributed file apps to directly with S3 and Swift device, with a Linux system, with a Linux access RADOS, kernel client and a kernel client and with support for QEMU/KVM driver support for FUSE C, C++, Java, Python, Ruby, and PHP RADOS A reliable, autonomous, distributed object store comprised of self-healing, self-managing, intelligent storage nodes 90
- 31. VM VIRTUALIZATION CONTAINER LIBRBD LIBRADOS M M M 91
- 32. CONTAINER VM CONTAINER LIBRBD LIBRBD LIBRADOS LIBRADOS M M M 92
- 33. HOST KRBD (KERNEL MODULE) LIBRADOS M M M 93
- 34. RADOS Block Device: • Storage of virtual disks in RADOS • Allows decoupling of VMs and containers • Live migration! • Images are striped across the cluster • Thin-provisioning • Snapshots and cloning
- 35. VM VIRTUALIZATION CONTAINER LIBRBD LIBRADOS M M M 115
- 36. HOW DO YOU SPIN UP THOUSANDS OF VMs INSTANTLY AND EFFICIENTLY? 116
- 37. instant copy 144 0 0 0 0 = 144 117
- 38. write CLIENT write write write 144 4 = 148 118
- 39. read read CLIENT read 144 4 = 148 119
- 40. old-style VM image creation local disk Nova Glance (VM images) compute (templates) read X ● ephemeral ● expensive to create X X' 29
- 41. Why use block storage? • Persistent • More familiar to users • Not tied to a single host • Decouples compute and storage • Enables Live migration • Extra capabilities of storage system • Efficient snapshots • Different types of storage available • Cloning for fast restore or scaling
- 42. Cinder volume creation Cinder Cinder volume Glance API volume driver (templates) create image from X locate X location of X read X X flexibility in where VM images are stored X' reference to X' 31
- 43. Efficient volume creation Cinder Cinder volume Glance API volume driver (templates) create image from X locate X location of X clone X to X' X fast CoW clone X' X' complete reference to X' 32
- 44. Questions? Josh Durgin josh.durgin@inktank.com jdurgin on freenode inktank.com | ceph.com