Linux High Availability Overview - openSUSE.Asia Summit 2015
1 like700 views
The document discusses high availability (HA) in Linux, covering architectural components, use case examples, and future outlooks. It details the evolution of HA, the significance of clustering, key software components like Pacemaker and Corosync, and various HA implementations and problems such as split-brain scenarios. Additionally, it emphasizes the necessity of proper fencing methods and explores future developments in HA technology.
Linux High Availability Overview - openSUSE.Asia Summit 2015
- 1. 1 High Availability on Linux Roger Zhou zzhou@suse.com openSUSE.Asia Summit 2015 SUSE way
- 2. © SUSE, All rights reserved.2 CURIOSITY in the land of Linux High Availability
- 3. 3 Agenda • HA architectural components • Use case examples • Future outlook & Demo
- 4. 4 What is Cluster? • HPC (super computing) • Load Balancer (Very high capacity) • High Availability ‒ 99.999% = 5 m/year MTTR ‒ SPOF(single point of failure) ‒ Murphy's Law "Everything that can go wrong will go wrong"
- 5. 5 "HA", widely used, often confusing • VMWare vSphere HA ‒ hypervisor and hardware level. Close-source. ‒ Agnostic on Guest OS inside the VM. • SUSE HA ‒ Inside Linux OS. ‒ That said, Windows need Windows HA solution. • Different Industries ‒ We are Enterprise. ‒ HADOOP HA (dot com) ‒ OpenStack HA ( paas ) ‒ OpenSAF (telecom)
- 6. 6 History of HA in Linux OS • 1990s, Heartbeat project. Simply two nodes. • Early 2000s, Heartbeat 2.0 too complex. ‒ Industry demands to split. 1) one for cluster membership 2) one for resource management • Today, ClusterLabs.org ‒ A completely different solution in early days, pacemaker + corosnc ‒ While merged Heartbeat project. ‒ 2015 HA Summit
- 7. 7 Typical HA Problem - Split Brain • Clustering ‒ multiple nodes share the same resources. • Split partitions run the same service ‒ It just breaks data integrity !!! • Two key concepts as the solution: ‒ Fencing Cluster doesn't accept any confusing state. STONITH - "shoot the other node in the head". ‒ Quorum It stands for "majority". No quorum, then no actions, no resource management, no fencing.
- 8. 8 HA Hardware Components • Multiple networks ‒ A user network for end user access. ‒ A dedicated network for cluster communication/heartbeat. ‒ A dedicated storage network infrastructure. • Network Bonding ‒ aka. Link Aggregation • Fencing/STONITH devices ‒ remote “powerswitch” • Shared storage ‒ NAS(nfs/cifs), SAN(fc/iscsi)
- 9. 9 Architectural Software Components "clusterlabs.org" ‒ Corosync ‒ Pacemaker ‒ Resource Agents ‒ Fencing/STONITH Devices ‒ UI(crmsh and Hawk2) ‒ Booth for GEO Cluster Outside of "clusterlabs.org" ‒ LVS: Layer 4, ip+port, kernel space. ‒ HAproxy: Layer 7/ HTTP, user space. ‒ Shared filesystem: OCFS2 / GFS2 ‒ Block device replication: DRBD, cLVM mirroring, cluster-md ‒ Shared storage: SAN (FC / FCoE / iSCSI) ‒ Multipathing
- 10. Software Components in details
- 11. 11 Corosync: messaging and membership • Consensus algorithm ‒ "Totem Single Ring Ordering and Membership protocol" • Closed Process Group ‒ Analogue “TCP/IP 3-way hand shaking” ‒ Membership handling. ‒ Message ordering. • A quorum system ‒ notifies apps when quorum is achieved or lost. • In-memory object database ‒ for Configuration engines and Service engines. ‒ Shared-nothing cluster.
- 12. 12 Pacemaker: the resources manager • The brain of the cluster. • Policy engine for decision making. ‒ To start/stop resources on a node according to the score. ‒ To monitor resources according to interval. ‒ To restart resources if monitor fails. ‒ To fence/STONITH a node if stop operation fails.
- 13. 13 Shoot The Other Node In The Head • Data integrity does not tolerate any confusing state. Before migrating resources to another node in the cluster, the cluster must confirm the suspicious node really is down. • STONITH is mandatory for *enterprise* Linux HA clusters.
- 14. 14 Popular STONITH devices • APC PDU ‒ network based powerswitch • Standard Protocols Integrated with Servers ‒ Intel AMT, HP iLO, Dell DRAC, IBM IMM, IPMI Alliance • Software libraries ‒ to deal with KVM, Xen and VMware Vms. • Software based ‒ SBD (STONITH Block Device) to do self termination. The last implicit option in the fencing topology. • NOTE: Fencing devices can be chained.
- 15. 15 Resources Agents (RAs) • Write RA for your applications • LSB shell scripts: start / stop / monitor • More than hundred contributors in upstream github.
- 16. 16 Cluster Filesystem • OCFS2 / GFS2 ‒ On the shared storage. ‒ Multiple nodes concurrently access the same filesystem. http://clusterlabs.org/doc/fr/Pacemaker/1.1/html/Clusters_from_Scratch/_pacemaker_architecture.html
- 17. 17 Cluster Block Device • DRBD ‒ network based raid1. ‒ high performance data replication over network. • cLVM2 + cmirrord ‒ Clustered lvm2 mirroring. ‒ Multiple nodes can manipulate volumes on the shared disk. ‒ clvmd distributes LVM metadata updates in the cluster. ‒ Data replication speed is way too slow. • Cluster md raid1 ‒ multiple nodes use the shared disks as md-raid1. ‒ High performance raid1 solution in cluster.
- 18. Cluster Examples
- 19. 19 NFS Server ( High Available NAS ) Cluster Example in Diagram VM1 DRBD MasterDRBD Master LVMLVM Filesystem(ext4)Filesystem(ext4) NFS exportsNFS exports Virtual IPVirtual IP VM2 DRBD SlaveDRBD Slave LVMLVM Filesystem(ext4)Filesystem(ext4) NFS exportsNFS exports Virtual IPVirtual IP Failover Pacemaker + CorosyncPacemaker + Corosync KernelKernel KernelKernel network raid1
- 20. 20 HA iSCSI Server ( Active/Passive ) Cluster Example in Diagram VM1 DRBD MasterDRBD Master LVMLVM iSCSITargetiSCSITarget iSCSILogicalUnitiSCSILogicalUnit Virtual IPVirtual IP VM2 DRBD SlaveDRBD Slave LVMLVM iSCSITargetiSCSITarget iSCSILogicalUnitiSCSILogicalUnit Virtual IPVirtual IP Failover Pacemaker + CorosyncPacemaker + Corosync network raid1 KernelKernel KernelKernel
- 21. 21 Cluster FS - OCFS2 on shared disk Cluster Example in Diagram Host3Host1 OCFS2OCFS2 Host2 OCFS2OCFS2 KernelKernel KernelKernel Virtual Machine Images and Configuration ( /mnt/images/ )Virtual Machine Images and Configuration ( /mnt/images/ ) Cluster MD-RAID1Cluster MD-RAID1 Cluster MD-RAID1Cluster MD-RAID1 Replication / Backup /dev/md127 /dev/md127 VMVM VMmigration Pacemaker + CorosyncPacemaker + Corosync
- 22. 22 Future outlook • Upstream activities ‒ OpenStack: from the control plane into the compute domain. ‒ Scalability of corosync/pacemaker ‒ Docker adoption ‒ “Zero” Downtime HA VM ‒ ...
- 23. 23 Join us ( all *open-source* ) • Play with Leap 42.1: http://www.opensuse.org Doc: https://www.suse.com/documentation/sle-ha-12/ • Report and Fix Bugs: http://bugzilla.opensuse.org • Discussion: opensuse-ha@opensuse.org • HA ClusterLabs: http://clusterlabs.org/ • General HA Users: users@clusterlabs.org
- 24. Demo + Q&A + Have fun
- 25. 25