Red Hat Summit 2017: Wicked Fast PaaS: Performance Tuning of OpenShift and Docker
4 likes1,468 views
This document summarizes performance tuning techniques for OpenShift 3.5 and Docker 1.12. It discusses optimizing components like etcd, container storage, routing, metrics and logging. It also describes tools for testing OpenShift scalability through cluster loading, traffic generation and concurrent operations. Specific techniques are mentioned like using etcd 3.1, overlay2 storage and moving image metadata to the registry.
![if [ “$containers” = “linux” ];
then
echo “fundamentals don’t change”
fi](https://image.slidesharecdn.com/rh-summit-2017-wicked-fast-paas-performance-tuning-of-openshift-and-docker-v2-170503190334/85/Red-Hat-Summit-2017-Wicked-Fast-PaaS-Performance-Tuning-of-OpenShift-and-Docker-5-320.jpg)
![OpenShift 3.5: Installation
https://docs.openshift.org/latest/scaling_performance/install_practices.html
[defaults]
forks = 20
gathering = smart
fact_caching = jsonfile
fact_caching_timeout = 600
callback_whitelist = profile_tasks
[ssh_connection]
ssh_args = -o ControlMaster=auto -o
ControlPersist=600s
control_path = %(directory)s/%%h-%%r
pipelining = True
timeout = 10](https://image.slidesharecdn.com/rh-summit-2017-wicked-fast-paas-performance-tuning-of-openshift-and-docker-v2-170503190334/85/Red-Hat-Summit-2017-Wicked-Fast-PaaS-Performance-Tuning-of-OpenShift-and-Docker-10-320.jpg)
Red Hat Summit 2017: Wicked Fast PaaS: Performance Tuning of OpenShift and Docker
- 1. Wicked fast PaaS: Performance tuning of OpenShift 3.5 and Docker 1.12 Red Hat OpenShift Engineering Jeremy Eder and Mike Fiedler, 2017-05-03
- 3. Awesome titles such as ...
- 5. if [ “$containers” = “linux” ]; then echo “fundamentals don’t change” fi
- 7. Terminology Overview Control Plane Compute Nodes and Storage Tier Infrastructure master and etcd master and etcd master and etcd registry and router registry and router LB
- 8. Let’s just get this out of the way ● We’re talking about OCP 3.5 or later ● Slides: https://www.slideshare.net/jeremyeder/ ● Code: https://github.com/openshift/svt ● There’s no video recording of this.
- 9. Deploying 2048 OpenShift nodes on the CNCF Cluster https://www.cncf.io/blog/2017/03/28/deploying-2048-openshift-nodes-cncf-cluster-part-2/
- 10. OpenShift 3.5: Installation https://docs.openshift.org/latest/scaling_performance/install_practices.html [defaults] forks = 20 gathering = smart fact_caching = jsonfile fact_caching_timeout = 600 callback_whitelist = profile_tasks [ssh_connection] ssh_args = -o ControlMaster=auto -o ControlPersist=600s control_path = %(directory)s/%%h-%%r pipelining = True timeout = 10
- 11. Docker Graph Driver: devicemapper vs overlay2 ● RHEL 7.4 ● SELinux ● Overlay2
- 12. Container Native Storage ● Dynamic ● Hyperconverged ● Scalable ● Performant CNS Session Thu 11:30a, Rm 157A NODENODENODE NODE POD POD POD POD POD POD POD POD POD POD RHGS RHGS RHGS POD POD POD MASTER
- 13. Container Native Storage: StorageClasses apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: cnsclass provisioner: kubernetes.io/glusterfs parameters: resturl: "http://172.25.87.92:8080" restuser: "admin" apiVersion: storage.k8s.io/v1beta1 kind: StorageClass metadata: name: ec2class provisioner: kubernetes.io/aws-ebs parameters: type: io1 zone: us-west-2b
- 14. Container Native Storage ● Dynamic ● Hyperconverged ● Scalable ● Performant
- 16. distributed key value store that provides a reliable way to store data across a cluster of machines
- 17. OpenShift 3.5: etcd-3.1.x ● etcd-2.x limited node scalability ● etcd-3.x gets us to 2000+ nodes comfortably ● Image metadata moved from etcd to registry in 3.4.z and 3.5. https://www.cncf.io/blog/2017/03/28/deploying-2048-openshift-nodes-cncf-cluster-part-2/
- 18. OpenShift 3.5: Image Metadata moved to Registry
- 19. OpenShift 3.5: etcd-3.1, storage mode v2, 5K projects Quorum Reads added in OCP 3.4 for data safety
- 20. OpenShift 3.5: etcd-3.1.x, 1k proj/4k pods
- 21. OpenShift 3.5: Metrics ● Bump scalability limits 12,000 → 25,000 pods ● METRICS_DURATION=7, METRICS_RESOLUTION=30 ● Capacity Planning and Scalability docs Heapster Hawkular Cassandra (collection) (metrics) (storage)
- 22. OpenShift 3.5: Logging (EFK) ● Logging Sizing Guidelines systemd/Fluentd Elasticsearch Kibana (collection) (storage) (visualization)
- 23. OpenShift 3.5: Routing/Network Ingress Tier ● HAProxy-based ingress tier (haproxy runs as a pod)
- 24. OpenShift 3.5: Routing/Network Ingress Tier
- 25. OpenShift 3.5: Alpha Support for GPUs ● Works fine ● Mostly manual for now ● GA gated on finalizing resource management https://blog.openshift.com/use-gpus-openshift-kubernetes/
- 26. Tooling
- 27. “I want an environment with thousands of deployments, pods (with persistent storage), build configurations, routes, services, secrets and more…” http://sandeen.net/wordpress/uncategorized/coming-clean-on-o_ponies/
- 28. “I want an environment with thousands of deployments, pods (with persistent storage), build configurations, routes, services, secrets and more…” http://sandeen.net/wordpress/uncategorized/coming-clean-on-o_ponies/
- 29. OpenShift Scalability Testing ● Cluster horizontal scale ○ # of nodes, # of running pods across all nodes ○ application traffic ● Node vertical scale ○ # of pods running on a single node ○ work that 1 node can support (applications, builds, storage) ● Application scalability ○ Scale # of application replicas up/down
- 30. OpenShift Performance Tests ● Resource usage/response times for scenarios ○ Application workload and access performance ○ Builds (OpenShift) ○ Metrics and Log collection ● OpenShift infrastructure performance ○ Resource usage of processes under load ○ Network (SDN) throughput ○ Routing ○ Storage (EBS, Ceph, Gluster, Cinder, etc)
- 31. Tools ● https://github.com/openshift/svt ○ cluster load-up ○ traffic generation ○ concurrent builds, deployments, pod start/stop ○ reliability testing ○ network performance ○ logging and metrics tests
- 32. Cluster loader ● cluster-loader - python tool to quickly load clusters according to a YAML test specification. ● Can be used with Kubernetes or OpenShift projects: - num: 1000 basename: nginx-explorer tuning: default templates: - num: 10 file: cluster-loader/nginx.yaml - num: 20 file: cluster-loader/explorer-pod.yaml
- 33. Demo