Rancher 2.x first step before deep dive
5 likes2,689 views
- Rancher 2.X stores all data as Kubernetes custom resources (CRDs), allowing Rancher to manage Kubernetes clusters. - Management controllers run in Rancher and handle cross-cluster resources, while user controllers run in each cluster and sync data between Rancher and that cluster. - Cluster and node agents proxy API requests to Kubernetes clusters and report state changes back to Rancher. This allows Rancher to manage clusters that it does not run directly on.
Rancher 2.x first step before deep dive
- 1. Rancher 2.X First Step before Deep Dive LINE Corporation, Verda2 Yuki Nishiwaki
- 2. Who you are? Name: ● Yuki Nishiwaki Working Experience: ● Private Cloud Development (OpenStack) Jan 2015 -> Now ● Kubernetes as a Service Development (Rancher) July 2018 -> Now ● Function as a Service Development (Knative) Sep 2018 -> Now
- 3. Rancher and Me ● User and Developer ● Have contributed few patches to rancher 2.X related projects ○ rancher/types ■ https://github.com/rancher/types/pull/525 ○ rancher/machine ■ https://github.com/rancher/machine/pull/12 ○ rancher/norman ■ https://github.com/rancher/norman/pull/201 ■ https://github.com/rancher/norman/pull/202 ■ https://github.com/rancher/norman/pull/203 ○ rancher/rancher ■ https://github.com/rancher/rancher/pull/15909 ■ https://github.com/rancher/rancher/pull/15991 ■ https://github.com/rancher/rancher/pull/16044 Not so many contributions yet
- 4. Today’s scope ● We have only 20-30 min ● Rancher have bunch of features. It will take half of day if I talk all parts ● Try to explain whole picture ● Check following article for more detail ○ https://www.slideshare.net/linecorp/lets-unbox-rancher-20-v200 ○ https://github.com/ukinau/rancher-analyse
- 5. Rancher 2.X ? All Data for Rancher are stored in Kubernetes => Depend on Kubernetes to run Rancher 1 Manage Kubernetes Cluster by multiple Provider ● Google Container Engine ● Amazon EKS ● Microsoft Azure ● Any VM Provider like OpenStack (RKE) 2 RBAC Configuration Configure/Deploy Additional Tools on Cluster Addon Management 4 Resource Resource Resource Admin member RBAC Configuration 3 Sync RBAC configuration
- 6. Rancher 2.X
- 7. Rancher 2.X architecture API Controller Cluster Agent Node Agent Node Agent Node Agent Cluster Agent Node Agent Node Agent Node Agent Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster
- 8. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent Don’t do actual provisioning
- 9. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent
- 10. Rancher API is just kind of proxy for Kubernetes API ClusterA NodeA UserA GroupA ・・・ Create Cluster Store All data as a Kubernetes by using CRD CRD for Cluster CRD for Node CRD for User Call Kubernetes API
- 11. Custom Resource Definition(CRD) in Kubernetes? Kubernetes Native Resource Type Custom Resource Type CustomResourceDefinition ConfigMap Pod Nginx App A Nginx Config Cluster Node Cluster Node Cluster A Cluster B Node A Node B Kubernetes allow user to create custom resource type in addition to natively supported resource.
- 12. > kubectl get crd clusters.management.cattle.io -o yaml apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: 2018-10-26T13:49:37Z generation: 1 name: clusters.management.cattle.io resourceVersion: "1278" selfLink: /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/clusters.management.cattle.io uid: fa628204-d925-11e8-b840-fa163e305e2c spec: group: management.cattle.io names: kind: Cluster listKind: ClusterList plural: clusters singular: cluster scope: Cluster version: v3 > kubectl get cluster NAME AGE local 1d Example of CRD for Rancher Resource: Cluster CRD for Cluster Cluster Resource
- 13. Question: Do we really needed to have API server? Create Create Why not Create Cluster Resource directly ? Resource information tend to be large and some of the attributes are only needed internally and don’t have to be exposed to user. Thus, Rancher mutate resource before try to create resource in k8s to add some internal information and also mutate resource before try to return resource to user in order to drop some attributes. Cluster A Cluster A Cluster A some attributes some attributes Add
- 14. Engineer for Rancher believe their approach is common ● Develop API Server Framework for Kubernetes API(CRD) ○ https://github.com/rancher/norman ● Support ○ Generate API function based on CRD schema ○ Determine path based on CRD schema ○ Override function to do some additional work like overriding attributes, drop attributes…
- 15. Resource Creation process is same as Kubernetes Resource API Controller ClusterA Watch Kubernetes ClusterReconcile Get latest information from kube-apiserver Check if any difference Between desired and actual states Do something to make actual state desired Reconcile Loop Cluster Agent Node Agent
- 16. Rancher Controllers Management Controllers Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster User Controllers User Controllers User Controllers Cluster Controller Node Controller Catalog Controller ・・・・ Alert Controller RABC Controller Endpoint Controller ・・・・
- 17. ● Run 1 Controller in a Rancher Server ● Do something across Kubernetes Clusters deployed ○ Catalog for Rancher User to deploy helm chart onto Kubernetes Clusters ○ Authentication for Rancher ○ Node Create/Update/Remove ○ Cluster Create/Update/Remove ○ Start User Controller for New Cluster Management Controllers Management Controllers Kubernetes Cluster Kubernetes Cluster
- 18. User Controllers ● Run 1 Controller for each Kubernetes Cluster deployed ● Do something for assigned Kubernetes Cluster ○ Sync data between Rancher Configuration and Kubernetes Cluster deployed ○ Propagate event happening in Kubernetes Cluster deployed to Rancher ○ Watch all workload(Pod...) in Kubernetes Cluster deployed and Do something User Controllers Kubernetes Cluster Kubernetes Cluster
- 19. After deployed, How to check/update Node/Cluster state? kube-apiserver rancher-server management controller user controller kubelet Pod A Pod B Target Resources Any New Pod? Anything Change in existing Pod? Pod A and B is alive! rancher cluster agent Target ResourcesTarget Resources Container (kubelet) File (certificates) Any change in Container, Files? Work as a TCP Proxy Please check my internal state as you want rancher node agent api-server
- 20. After deployed, How to check/update Node/Cluster state? kube-apiserver rancher-server management controller user controller kubelet Pod A Pod B Target Resources Any New Pod? Anything Change in existing Pod? Pod A and B is alive! rancher cluster agent Target ResourcesTarget Resources Container (kubelet) File (certificates) Any change in Container, Files? Work as a TCP Proxy Please check my internal state as you want rancher node agent api-server
- 21. Agent TCP Proxy is used when you call k8s API
- 22. Please look at endpoint closely https://<rancher-server-endpoint>/k8s/cluster/local All requests against k8s cluster are sent to Rancher first and Rancher proxy it into correct cluster
- 23. How Rancher proxy Kubernetes Cluster rancher cluster agent Kubernetes Cluster rancher cluster agent User can not call Kubernetes API While cluster-agent is down If rancher cluster agent got down for some reason
- 25. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Rancher Server play very very important role, if it is down, following features are not available 1. Create/Update/Delete Node, Cluster 2. All management function for Node, Cluster 3. Proxy Kubernetes API request to Cluster
- 26. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Rancher 2.0 is not allowed operator to run multiple rancher server because of websocket session handling logic.
- 27. Rancher 2.0 -> 2.1: Support multiple Rancher Server Kubernetes Cluster Kubernetes Cluster LB Leader From 2.1, Rancher’s websocket session handling logic got improved and now they allow us to run multiple rancher server for high availability and scalability.
- 28. Rancher 2.0 -> 2.X: Move User Controllers to... User Controllers User Controllers User Controllers Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster CPU/Memory Usage increase every time create cluster ・・・・ ・ ・ Management Controllers
- 29. Rancher 2.0 -> 2.X: Move User Controllers to... Kubernetes Cluster Kubernetes Cluster Kubernetes Cluster User Controllers User Controllers User Controllers Management Controllers CPU/Memory increasing ratio by new cluster got improved than before because new user controller for new cluster will be running on new cluster itself