Autoscaling with magnum and senlin
2 likes1,185 views
1. The document discusses integrating Magnum and Senlin for autoscaling containers across multiple levels. Magnum provisions and manages container clusters while Senlin provides clustering and autoscaling capabilities. 2. A design is proposed where Senlin policies and triggers drive autoscaling at both the container and cluster level. Scaling would be coordinated between applications and clusters through control flows in both directions. 3. Integration with existing autoscaling in container orchestration engines like Kubernetes is also considered, where Senlin could leverage autoscalers but also handle scaling requirements not met by the native solutions.
Autoscaling with magnum and senlin
- 1. 1 Exploring Magnum and Senlin integration for autoscaling containers Hongbin Lu (Huawei) Jay Lau, Ton Ngo (IBM) Qiming Teng, Julio Ruano (IBM) Exploring Magnum and Senlin integration
- 2. Contributors Exploring Magnum and Senlin integration2 Zilian Ji, Linjuan Xia, Shijia E, Peiyu Lin, Tianyuan Zhuang, Ji Jiang, Yongkui Wang
- 3. Talk Outline 1. Use cases 2. Magnum 3. Senlin 4. How it works 5. Demo Exploring Magnum and Senlin integration3
- 4. Use cases Exploring Magnum and Senlin integration4
- 5. Use case • Shared COE cluster – Autoscale on some policies (utilization, etc.) – Scale number of nodes • Multiple apps on cluster – Autoscale on some policies (request rate, etc.) – Scale number of containers • Example: – 2, 3-tiered web apps: handle requests – SLA determines policy Exploring Magnum and Senlin integration5
- 6. Scaling scenarios (1) Exploring Magnum and Senlin integration6 node c c c node c cc node c scale app scale cluster Single app on cluster • scale out containers as needed • scale cluster by utilization • coordination = utilization
- 7. Scaling scenarios (2) Exploring Magnum and Senlin integration7 node c node c c node c c c c scale 2 apps Two apps on cluster • scale out containers as needed • scale cluster by utilization • coordination = utilization scale cluster unlimited scale cluster limited
- 8. Scaling scenarios (3) Exploring Magnum and Senlin integration8 node c c node scale 2 apps scale cluster limited c node c c c c c c Two apps on cluster, limited cluster size • blue scales first, retains containers • red scales later, fails • maybe acceptable • what if red is critical and blue is not • inverted priority: blue < red
- 9. Scaling scenarios (4) Exploring Magnum and Senlin integration9 node c c scale 2 apps Limit on apps • Place max on blue • blue scales, red does not scale • cannot use full resources scale cluster limited c nodenode c limited
- 10. Scaling scenarios (5) Exploring Magnum and Senlin integration10 node c c node scale 2 apps scale cluster limited c node c c c c c cc c c c Priority on apps • blue scales first, red scales later • blue scales back to free up resource for red • 2 ways linkage: container cluster host cluster
- 11. Magnum Exploring Magnum and Senlin integration11
- 12. Instance Magnum Overview Exploring Magnum and Senlin integration12 • Provision • Kubernetes • Docker Swarm • Mesos • Scale • Add instances • Remove instances • Manage • Pod • Service • Replication Controller • Container Nova Instance container container Magnum CLI Magnum Heat
- 13. Magnum Architecture Exploring Magnum and Senlin integration13
- 14. Magnum: Autoscale Bay 1. Raw data: Magnum pulls raw data from Bays. 2. Metrics: Metrics, like memory utilization, are computed and sent them to Ceilometer. 3. Alarm: Ceilometer triggers alarms based on user-specified scenario and metrics. 4. Scale: Heat scales the bays. Exploring Magnum and Senlin integration14 Ceilometer Magnum Heat Bay 1) Raw data 2) Metrics 3) Alarm 4) Scale
- 15. Kubernetes: Autoscale Pod 1. Metrics: The autoscaler periodically queries pods’ metrics. 2. Scale: Based on collected metrics, the autoscaler uses a built-in algorithm to trigger the scaling. 3. Resize: Replication Controller resize the pods. Exploring Magnum and Senlin integration15 Horizontal Pod Autoscaler Pods 1) Metrics 3) Resize Replication Controller 2) Scale
- 16. Future direction • Magnum AutoScaler – A new component in Magnum • Scaling Bays • Scaling Applications – Interacts with AutoScaler from COEs • Kubernetes AutoScaler as a driver for container-level scaling • Magnum responsible for scaling RCs without Kubernetes autoscaler – Customizability • data sources • metrics and thresholds • standby bay nodes (i.e. resource pool) Exploring Magnum and Senlin integration16
- 17. Two-Levels of AutoScaling Exploring Magnum and Senlin integration17 Working K8S Bay NodeNode K8S Master Heapster Kublet cAdvisor Kublet cAdvisor Storage Backend RC AutoScaler Pod Magnum AutoScaler Service K8S Driver Collector Analyzer Operator Swarm Driver Mesos Driver Senlin Magnum Conductor Standby
- 18. Senlin Exploring Magnum and Senlin integration18
- 19. Clustering - A Missing Service in OpenStack Exploring Magnum and Senlin integration19 Scalable Load-Balanced Highly-Available Manageable ...... of any (OpenStack) objects
- 20. Senlin Architecture Exploring Magnum and Senlin integration20 REST RPC Senlin Database Policies Profiles
- 21. Clustering/AutoScaling/LoadBalancing Exploring Magnum and Senlin integration21 Nova Kubernetes Heat Ironic BareMetal VMs Stacks Containers Senlin placement deletion scaling health load-balance batching
- 22. Scaling A Cluster using Senlin Exploring Magnum and Senlin integration22 type: os.nova.server version: 1.0 properties: image: fedora-21 flavor: 3 networks: - network: private userdata: ... Profile Cluster type: senlin.policy.scaling version: 1.0 properties: event: CLUSTER_SCALE_IN adjustment: type: CHANGE_IN_CAPACITY number: 1 min_step: 1 best_effort: True attach Policy type: senlin.policy.scaling version: 1.0 properties: event: CLUSTER_SCALE_OUT adjustment: . . . attach Policy webhook RESIZE
- 23. Policies Exploring Magnum and Senlin integration23 type: senlin.policy.scaling version: 1.0 properties: adjustment: min_step: 1 best_effort: true number: 1 type: CHANGE_IN_CAPACITY event: CLUSTER_SCALE_OUT • Declarative • Can be mapped to from a standard • Example: SUR_Cluster (Add k8s Minion / Add Spark worker node)
- 24. Trigger Exploring Magnum and Senlin integration24 type: CeilometerThresholdAlarm version: 1.0 rule: meter_name: memory_util comparison_operator: gt threshold: 70 period: 60 evaluation_periods: 1 statistic: avg query: - field: resource_metadata.cluster op: eq value: SUR_Cluster • Generic abstraction • alarms in OpenStack monitoring services • e.g. Ceilometer Alarm
- 25. How it works Exploring Magnum and Senlin integration25
- 26. Design goals 1. Coordinate scaling at apps and cluster level • Control flows in both direction • Apps -> Cluster: utilization, other methods • Cluster -> Apps: priority, other methods 2. Integrate scaling support from COE if available • Kubernetes autoscaler 3. Policy driven • Easy to express, manage complex cases Exploring Magnum and Senlin integration26
- 27. template Outer Stack Autoscaling: OpenStack-Only Solution (Today) 27 Exploring Magnum and Senlin integration AutoScalingGroup Stack Heat Ceilometer AH PUp PDn AL ASGS1 S2 S3 update ASG
- 28. Senlin profile Autoscaling: OpenStack-Only Solution (New) 28 Exploring Magnum and Senlin integration Cluster Ceilometer AH PUp PDn AL SCALE_IN SCALE_OUT Node1 Node2 Node3 webhookspolicies triggers
- 29. Demo Exploring Magnum and Senlin integration29
- 30. Autoscaling with Kubernetes 30 Exploring Magnum and Senlin integration VMs K8S Bay NodeNode Heapster Kublet cAdvisor Kublet cAdvisor Pod Senlin Magnum profile CeilometerWorkload policy
- 31. Questions Exploring Magnum and Senlin integration31
- 32. Thank You Exploring Magnum and Senlin integration32