Engage 2020 - Kubernetes for HCL Connections Component Pack - Build or Buy?
0 likes1,095 views
The document discusses the considerations for building or buying a Kubernetes cluster, highlighting the differences between on-premises and cloud deployments. It covers essential aspects such as planning, storage, network configurations, deployment options, and support limitations, with a focus on using AWS or Azure. Additionally, it provides cost estimation for various resources and offers links for further reference on setting up Kubernetes in the cloud.
Engage 2020 - Kubernetes for HCL Connections Component Pack - Build or Buy?
- 1. Kubernetes - Build or Buy Martin Schmidt – Beck et al. Christoph Stoettner – panagenda 1#engageug On Premises Cloud
- 2. Agenda • Short Intro on containers and orchestration • Disclaimer (actual support statement) • Building a Kubernetes cluster • Planning • Network (Overlay, DNS)* • Storage (HA)* • Backup / Restore • On Premises caveats • Using a Cloud provider • Sizing / Costs 2#engageug * On Premises only On Premises Cloud
- 3. Martin Schmidt • Senior IT Architect at Beck et al. • IBM / HCL Connections since 2007 • Experience in • Deployment, Migration, Operation and Customization • Kubernetes EKS and AWS, Container • Focusing on • enhancing with own containers • make the most out of customizer • More and more • DevOps, Automation 3#engageug Cloud
- 4. Christoph Stoettner • Senior Consultant at panagenda • IBM Connections since 2009 • Experience in • Migrations, Deployments • Kubernetes, Container • Performance Analysis, Security • Focusing in • Security, Deployments • More and more • DevOps, Automation, Ansible 4#engageug On Premises
- 7. Kubernetes is containers on steroids 7#engageug
- 8. What's Kubernetes? 8#engageug On Premises Cloud key value: Key-Value Store Controller Api-Server Scheduler Master Node Kube-proxy Kublet Pod Pod Node Kube-proxy Kublet Pod Pod Node Kube-proxy Kublet Pod Pod User CLI/APIs Dashboard
- 9. Infrastructure example 9#engageug On Premises Cloud
- 10. Distributions / Cloud provider • AWS EKS • Google GCE • Azure • Digital Ocean • Kubeadm • The Hard Way (from Scratch) • Kops • OKE/OpenShift • Rancher • Ubuntu Kubernetes 10#engageug https://www.cncf.io/certification/software-conformance On Premises Cloud
- 11. Disclaimer • HCL currently supports the documented reference installation only. • Other installations are not officially supported (but they will not refuse just you are not on the reference platform) • In the future this will change…. 11#engageug
- 12. Supported Kubernetes for ComponentPack 6.5 • Red Hat 7.6 and CentOS 7.6 • Docker 17.03 (EE/CE + devicemapper storage) Docker 18.06.2+ (CE + devicemapper storage) • Kubernetes version 1.11.9 • Calico v3.3 used as the network add-on • Helm v2.11.0 12#engageug This limits us a little bit and of 90 compliant options only few are left On Premises Cloud The Component Pack was validated on a Kubernetes v1.11.6 cluster that was set up on virtual machines using the kubeadm tool.
- 13. Supported Cloud provider with version 7 13#engageug
- 14. Deployment Options – Testserver • All services on one server (master and worker): • 16 CPU, 2.x GHZ, 64GB memory, 100GB disk. • 50GB+ for Device Mapper block storage 14#engageug 16 CPU cores 64 GB memory 150 GB disk Additional requirements for Customizer 1 - Reverse proxy: for Customizer: 4 CPU, 2.x GHZ, 4GB memory - 100GB disk On Premises
- 15. Deployment Options – Standard • 1 - Master: 4 CPU, 2.x GHZ, 16GB memory, 100GB+ disk • 50GB+ for Device Mapper block storage. • 1 – Generic Worker • 8 CPU, 2.x GHZ, 24GB memory, 100GB disk. • 50GB+ for Device Mapper block storage • 1 – Infrastructure Worker • 8 CPU, 2.x GHZ, 24GB memory, 100GB disk. • 50GB+ for Device Mapper block storage. • Storage: Persistent volumes for Elasticsearch, Customizer, MongoDB, Zookeeper, and Solr indexes, 100GB disk. 15#engageug 20 CPU cores 64 GB memory 550 GB disk Additional requirements for Customizer 1 - Reverse proxy: for Customizer: 4 CPU, 2.x GHZ, 4GB memory - 100GB disk On Premises
- 16. Deployment Options – high available • 3 - Masters • 4 CPU, 2.x GHZ, 16GB memory, and at least 100GB disk. • Add 50GB+ per master for Device Mapper block storage. • 3 - Generic Workers • 6 CPU, 2.x GHZ, 24GB memory, and 100GB disk. • Add 50GB+ per master for Device Mapper block storage. • 3 - Infrastructure Workers • 6 CPU, 2.x GHZ, 24GB memory, and 100GB disk. • Add 50GB+ per master for Device Mapper block storage. • Storage (available on all nodes) • Elasticsearch, Customizer, MongoDB, Zookeeper, and Solr indexes - 150GB disk. • Loadbalancer (haproxy, i5, nginx) 16#engageug 48 CPU cores 156 GB memory 2.4 TB disk Additional requirements for Customizer 1 - Reverse proxy: for Customizer: 4 CPU, 2.x GHZ, 4GB memory - 100GB disk On Premises
- 18. What do we want to achieve? How to start? • Get a cloud like experience • Automate the deployment process • Choose a networking solution • Choose a storage solution • Handle security and authentication • Ansible can help automating the installation • Prepare VM • Disable Swap • Install Prerequisits • ... • Deploy Kubernetes 18#engageug On Premises
- 19. Build machines • Bare Metal or Virtual Machines • If you plan high availability workers and nodes • Cluster must be distributed across multiple hypervisors • Hardware errors can happen • Think about dumb machines which doesn't hold data • Just running containers (pods) 19#engageug On Premises
- 20. Operating System • Supported: Red Hat 7.6 or CentOS 7.6 with devicemapper • Better optimized • Fedora CoreOS. • Automatically updating Linux OS for containerized workloads. • RancherOS • lightweight, secure Linux, built from containers to run containers • Evolution from Infrastructure as Code • No separate Packer -> Terraform -> Ansible -> Kubernetes • Provide a pxeboot.cfg and ignition file on a webserver • VM grabs config from pxe and ignition file • To update or repair, just reboot into new image 20#engageug On Premises
- 21. Networking • Supported: Calico • CNI – Container Network Interface • Calico - a layer 3 virtual network • Weave - a multi-host Docker network • VMware NSX – enables automated NSX L2/L3 networking and L4/L7 LB • Many more 21#engageug On Premises
- 22. High available network storage • Replication • Ceph • GlusterFS • Rancher Longhorn • Amount of disk space is needed multiple times • Hardware storage • NAS / SAN • High available 22#engageug On Premises
- 23. Kubeadm • Script/Binary to prepare Kubernetes environment • Saves from copying “hundreds” of certificates • Manually deploy and update masters and nodes 23#engageug On Premises
- 24. Alternatives • Kubespray • Ansible roles to build Kubernetes Cluster • Rancher • Run it free or buy support • OpenShift • Special • Secure, but special • Not for ComponentPack, but k3s.io even runs on Raspberry Pi 24#engageug On Premises
- 25. Updates • You only can upgrade • from one MINOR version to the next MINOR version 1.16 → 1.17 • between PATCH versions of the same MINOR 1.16.1 → 1.16.5 • So going from 1.11.6 to 1.17.3 means • 1.11.6 → 1.12.x → 1.13.x → 1.14.x → 1.15.x → 1.16.x → 1.17.3 • The upgrade workflow at high level is the following: • Upgrade the primary control plane node • Upgrade additional control plane nodes • Upgrade worker nodes • One at a time • Zero downtime • In place or add temporary additional nodes 25#engageug On Premises
- 26. Our goal • We should end with • Supported Kubernetes Deployment • Supported Connections / ComponentPack • Kubernetes supports • Last 3 minor versions • Today: 1.15, 1.16, 1.17 • Several companies sell support contracts for Kubernetes • Or build the skills inhouse 26#engageug On Premises
- 28. Buy Kubernetes from a Cloud Provider • Kubernetes Cluster can be purchased from different provider • AWS * • Azure * • Google * • Digital Ocean • IBM Cloud • … • They are all a little bit different and you need to check if the products fit your needs. • None of them matches the “Reference Implementation” => currently not supported -> * v7 will change this 28#engageug Cloud
- 29. Component Pack on AWS • AWS has the most complete offer (as far as I know) • Azure was tested as well (Component Pack 6.0.0.7) • We have tested Component Pack 6.0.0.7 – 6.0.0.9 and 6.5.0.0 on EKS. • Other services can also be used • Elasticsearch Service (for metrics and type ahead search of classic infrastructure. For OrientMe it does not work – open PMR) 29#engageug Cloud
- 31. Installation Pro • With a single command you get an operational Kubernetes Cluster. (eksctl) • With a single command you get an operational Elasticsearch cluster. Con • You get only dedicated Kubernetes / Elasticsearch versions. • Limited individual setup possible. • Minimum Infrastructure Requirements must be available. 31#engageug Cloud
- 32. Operations Pro • AWS is managing the Master / Cluster for you. • Restart / Recreate / Add / Remove Kubernetes node / Elasticsearch node is just 1 command. Con • In case of problems, you have no access possibilities to the Kubernetes Master other than API. • In case of problems, you can not install patches or fixes on Kubernetes Nodes. • In case of problems, you have no direct access to the Elasticsearch Nodes. 32#engageug Cloud
- 33. Support Pro • AWS can help you with the Kubernetes Cluster / Elasticsearch Cluster (Professional Services) • The infrastructure can be build reproducible without effort (e.g. Cloudformation, eksctl) • Can be easily integrated with other products (e.g. Lambda, CodePipline, CodeBuild, CloudWatch, ...) Con • Officially not (yet) supported by HCL 33#engageug Cloud
- 34. Update Pro • Update to a later Kubernetes version is just one command. • Update to a later Elasticsearch version is just one command. • AWS makes sure these updates work. Con • Updates are only possible to dedicated versions. • AWS removes support for older versions of Kubernetes and you must update to a newer version. • AWS removes support for older versions of Elasticsearch and you must update to a newer version. 34#engageug Cloud
- 35. Used Services Pro • Kubernetes (EKS) • Elasticsearch* • Load Balancer (Classic Load Balancer) • NFS Server (efs) for persistent storage • Docker Registry (ECR) • DNS (Route53) Con • Every service has a price tag. 35#engageug * Elasticsearch for Metrics and Type Ahead only. Cloud
- 36. AWS cost estimation • Based on • 1year reserved instances, no upfront • Instances hosted in Ireland (zone eu-west-1) • HA deployment • Elastic Search Server on Kubernetes (Not recommended due to storage problems but currently AWS Elasticsearch does not work for OrientMe) • EC2 m5.xlarge are smaller than recommended and m5.2xlarge are larger than recommended • Price calculation from 27. February 2020 • Prices for network traffic, backup storage and AWS support is not included • No free tier included • Classic infrastructure is not included 36#engageug Cloud
- 37. Rough Price Estimation * AWS resource Size Number required Price per unit / per resource Price per month EKS Master 1 $ 0.10 $ 73.0 EKS Worker Node m5.xlarge 6 $ 0.1500 $ 657.00 HDD 100 GB 6 $ 0.11 $ 66.00 EFS Share 200 GB 1 $ 0.33 $ 66.00 Classic LB 2 $ 20.50 $ 20.50 Admin Host 20% t3.medium 1 $ 0.0408 $ 6.71 Admin Host HDD 30GB 1 $ 0.11 $ 3.30 ECR 30GB 1 $ 0.10 $ 3.00 Sum $ 895.51 37#engageug Cloud * no responsibility is taken for the correctness of this information.
- 38. Summary • In case your company already has AWS or Azure know how but lacks Kubernetes know how EKS or AKS is a good start. • Due to the lack of support we can’t recommend it for production use (yet) but for test, development or prove of concept this infrastructure works fine. 38#engageug Cloud
- 39. Links and references • Setup Component Pack in the cloud: https://becketalservices.github.io/beas-cnx-cloud/ • Kubernetes reference implementation: https://help.hcltechsw.com/connections/v65/admin/install/c p_prereqs_kubernetes.html • Kubernetes: https://kubernetes.io • K8s documentation (Tutorials, Basics, Start): https://kubernetes.io/docs/home/ • Cloud Native Computing Foundation (CNCF): https://cncf.io 39#engageug On Premises Cloud