"Look Ma, no hands! Zero Touch Provisioning for OpenShift" DevConf.US 2021

Editor's Notes

• #2: Hi everybody! Thank you for joining our session, My name is Nir Magnezi, and with me, Freddy Rolland, we both are part of the Edge Pillar group here at Red Hat. Today we are going to talk about a new way to deploy OpenShift Clusters at scale, in an automated way. This method is called Zero Touch Provisioning, or ZTP for short.
• #3: After the container revolution, Kubernetes, which is the basis for OpenShift, became the de-facto standard for infrastructure management. Having multiple Kubernetes clusters in your organization, spread across data centers, regions, and multiple cloud providers, has become a reality. And with that, when the number of clusters increases, the need for administrators to have robust and reliable tools to manage their fleets of clusters became more apparent.
• #4: Open Cluster Management (AKA Red Hat Advanced Cluster Management) is an open source solution for managing muliple clusters. It offers solutions for challenges like: Policy enforcement Application management And also cluster lifecycle management. Where you may deploy, upgrade or deprovision your clusters. Open Cluster Managment is available for download as an operator from the operator hub website. Now, the notion of a cluster being this gigantic thing with hundreds of nodes, and a large footprint with muti tenancy, still exists and we do see that, but less and less. What we start to see is smaller clusters, with new and more compact topologies such 3 masters, 3 workers type of scenario and as we are now getting closer to the edge, it comes as a requirement to support more lightweight infrastructure, such as for example, in 5G, the far edge network with the distributed units. So with that, we got to a point where the entire deployment needs to occupy the absolute smallest possible footprint, just a single server.
• #5: To achieve that, Red Hat engineers have been working to reduce the footprint of OpenShift, so it fits into these more constrained environments by putting the control plane and worker capabilities into a single node. OpenShift typically requires a temporary bootstrap machine, which is usually a separate machine, and of course a provisioning network, but edge deployments are often environments where there are no extra nodes to spare. However, for these use-cases that we are interested in, a new functionality provided by OpenShift called “Bootstrap-in-Place” eliminates the separate bootstrap node requirement for single-node deployments. So when installing a single-node OpenShift, you only need the node you wish to install onto. Yet, few things to keep in mind here: Single node openshift, or SNO for short, is still in developer preview in 4.8 It requires at least 8 vCPU cores, 32 GB of RAM, and at least 120 GB of storage It does not have the option to add additional hosts after you finished your installation. And as you can probably assume, Single-node OpenShift is not highly-available. Which means you cannot expect zero downtime for your Kubernetes API. So how do we deliver a single node openshift out to the edge, in large scale and do that with good performance.
• #6: First, let’s define our requirements: The topology will be a Hub/Spoke where the Hub is the cluster running the management application. We want the Spoke cluster installed on Bare Metal, like for example a server at the base of an antenna in a 5G far edge deployment. We should have a minimal performance impact on the Hub cluster when installing spoke clusters. Also, we want to be able to automate the process, to avoid errors. So a declarative API, via kubernetes CRDs is required. It will allow us to use GitOps oriented deployments. Customer environment will be disconnected, so no access to the internet. And we want to be able to install 1000 Spoke clusters, import them into OCM, apply policies, get a status of the policies and all of that in an acceptable time frame. In order to install clusters, OCM is using an open source library called Hive. Hive is an operator which runs as a service on top of Kubernetes. The Hive service can be used to provision and perform the initial configuration of OpenShift clusters. It supports several platforms like AWS, Azure,GCP, OpenStack, oVirt and vSphere. Hive also supports bare metal provisioning as provided by openshift-install, using IPI. However, this feature requires a separate pre-existing provisioning host to run the bootstrap node. Also this host will require specific network configuration. And for each cluster install, Hive will start an installation pod, taking 800 MB of memory on the hub cluster and will consume some storage for running the installer. So, with an additional node and additional workloads on the hub for each spoke cluster installation, this method cannot meet our scale requirements. How can we take that away from the hub?
• #7: So there is a way to install bare metal clusters, without the need of an additional bootstrap. The assisted installer is a SAAS hosted in Red Hat dot com, that enables the user to easily install Openshift on bare metal or VMs. It provides a UI where the user is guided to the process of providing the minimal input to create a Discovery ISO. The user will need to boot the servers with that ISO where an agent will report to the Service the hardware and other sanity checks. Then the installation can be kicked off by the user once all preflight checks are done. So, the good news are that: - Assisted Installer supports SNO - No need for a bootstrap node - The installation is run on the node itself, (with Bootstrap In Place for SNO). No need to run the installer on the hub cluster. And the bad news are that: - Assisted Installer is SAAS, nodes need access to the internet to communicate - It has a REST API and not a declarative API - And the user needs to boot the ISO himself. Alright, so we need to take the assisted installer from the cloud into the hub. How do we do that? We packed the Assisted Installer as an operator deployed on the Hub Cluster (without UI) We created Kubernetes APIs , based on the API defined by Hive so that the integration with OCM will be easy. And for booting the nodes, we can use existing capabilities of Metal Kube. The Bare Metal Operator is capable of booting a host given an URL of an ISO using BMC which is Baseboard Management Controller. By the way, if you want more information on the assisted installer, you can check out a talk we did about it earlier this year, a link is available in the last slide.
• #8: Now we have all the pieces: - Multi Cluster management on the Hub cluster with OCM - Cluster Provisioning API with hive - Agent Based Installation with Assisted Installer - Bare Metal ISO boot with Metal Kube and Bare Metal Operator And now, all we need is just connect everything together...
• #9: So let's walk through the high level flow. First the assisted installer will generate an ISO according to parameters defined in CRDs. Once the ISO is ready, the Bare Metal Operator will connect to the bare metal via the BMC interface and boot it with the ISO. Once booted, the assisted installer agent will start collecting hardware information and report back to the Assisted Installer service. Once required validations are made, checking for example that we have enough RAM and CPU, the Assisted Installer will kickstart the installation on the spoke cluster. Once Openshift is installed, Hive will report via CRD to OCM that the cluster is ready to work with. OCM will import the cluster, and deploy his agent (called Klusterlet). Then it will apply policies if any are defined and also configure Prometheus to report metrics. So now, let’s take a closer look at the API and CRDs that we use in the ZTP flows.
• #10: Note that some of them were already defined. ClusterDeployment and ClusterImageSet came from Hive. ClusterDeployment was enhanced to be able to plug an external installer. Using ClusterDeployment, gave us the ability to keep the existing Open Cluster Management interface. BareMetalHost is coming from the Metal Kube project and ManagedCluster and KlusterletAddonConfig came from Open Cluster Management. The InfraEnv, and NmstateConfig are the resources needed to create the Discovery Image. The two are linked with a label selector that allows InfraEnv to locate the relevant NMStateconfig, those contain network configuration, such as static IPs and more. Users may configure the infraenv with a SSHKey to debug the host in the discovery phase. The BareMetalHost contains the BMC connection information for the target Bare Metal machine. It will load and boot the Discovery image on that spoke. AgentClusterInstall specifies the cluster’s configuration such as networking, number of control plane, etc. The Agent contains hardware information about the target Bare Metal machine. It is created automatically on the Hub cluster once the Discovery Image on the machine is booted. You’ll see that in the demo. One additional note about ManagedCluster and KlusterletAddonConfig. both are Open Cluster Management CRDs. In order for the cluster to be managed by the Hub, it needs to be imported and known. ManagedCluster provides that interface. KlusterletAddonConfig contains the list of services (provided by the Hub) to be deployed to a ManagedCluster once imported into the hub.
• #11: Over the next few minutes, I'll demonstrate the deployment of an OpenShift cluster comprised of a single node. Such a low footprint cluster is useful for a variety of use cases, and here you'll see how such deployment is performed using the declarative API we mentioned previously in the session. I'll start by creating a cluster image set and pull secret, which are prerequisites for registering a cluster to assisted installer. Following up with creating cluster deployment and agent cluster install to register the cluster. Now, we can monitor the cluster events and agent cluster install conditions, to get better visibility to the installation process. I'll follow up by creating the InfraEnv resource, which contains configurations relevant for generating the Red Hat CoreOS discovery image, such as the SSH key, Ignition config, and more. Notice how this is immediately reflected in cluster events. We now get an ISO download URL which we could use to download the image and upload to our server, but instead we have the Assisted Installer that will monitor that URL, and when it becomes available, update the BareMetalHost Then the Bare Metal Operator will automatically provision the host using that image. I'll speed the recording so we can hop into the next step and see the host reports to assisted service. The assisted installer will approve the node for us and the cluster will start preparing for installation. The installation has now started. I'll speed up the recording once more, and we'll see the installation steps take place. The installation workflow consists of several steps. The cluster, which in our case is a single node that also serves as our bootstrap machine, starts the installation by writing the core OS image and cluster configuration to the disk. Then the machine is being rebooted and started from the disk. It no longer depends on that discovery ISO. Lastly, we move to the finalize phase, where we wait for the cluster to initialize the core OpenShift operators. The cluster installation is now completed. Let's validate that we are able to interact with the cluster by gaining CLI access using the admin kubeconfig available in a secret. Note that we can see our single node in a ready state - function both as a master and worker. That concludes the demo of a single cluster deployment”
• #12: Alright, now that saw how to deploy one SNO, are we ready to deploy 1000 SNO clusters? We used about one hundred physical nodes, and used libvirt on top of them in order to spin up VMs and simulate the spoke clusters”
• #13: Here are the result of one of the 1K runs We did 10 seconds staggering for provisioning. Meaning that we create all CRDs for one cluster, wait 10 seconds and then create the next one. So it takes about 3 hours to create all of them, and about an additional one hour to finish all the OCP installs. The blue line is “initialized”, meaning that CRs for a cluster are created and applied. The red one is “booted”, Bare metal operator booted the spoke machine with the discovery ISO. The green is "discovered": meaning that the agent CR was created on the hub, and that the hardware discovery phase is completed. The "provisioning" line, the one in purple, shows that ocp installation is in progress. "completed": means that OCP is installed and "managed" means OCM imported the cluster. The success rate is very high, and out of a little more than 1k clusters maybe about only 5 clusters did not finish the installation successfully. A lot of performance improvements were done on the way to be able to achieve this milestone.
• #14: In an ideal world we can just pull everything from the Internet, but in the real world, It is not that simple. Private Data centers are mostly disconnected from the internet for security reasons. Here are the components that are required to achieve a disconnected setup of ZTP. So, we will need an internal registry server, where we should mirror the OCP Release and also all the OLM containers. We will need an HTTP server to host the RHCOS LiveISO and the RootFS. these are needed for the assisted installer to create the discovery ISO. And of course we will need some networking configuration like Firewall rules, and make sure that DNS and DHCP are well configured. Exacts steps are available in the below GitHub repo.
• #15: Now, that we have all the ZTP APIs and use GitOps to deploy full sites and use the power of OCM policies to apply configurations. The source of truth is a set of git repos, that contains a site planning. A site planning is a set of yamls files that contain all the different settings for the clusters that we want to deploy. It will include parameters like cluster name, domain, IP ranges, and also definitions for the hardware like BMC credential, static network definitions and also additional operators that we want to be installed. All that combined with Kustomize we are producing the final output that will be a yaml definition of all the settings that we are passing to OCM, that will apply them, create the cluster and after the install applying the rest of the configuration via policies. For example a 5G profile will include: Machine config settings for NTP and SCTP Performance add on operator: for enable Real Time Kernel and other features SRIOV, if you want to interact with low level network functions And in the edge PTP configuration is also required. Below is a link to a repo with all you need to use GitOps with Argo CD and deploy ZTP with 5G profiles.
• #16: Here are some of the features planned for the ZTP component: We want to support additional cluster topologies, like 3 masters, or 3 masters and more workers. Also, the user may want to add an additional Worker Node to an installed cluster as a Day2 operation. The flow will be very similar, only that the starting point will be an existing cluster. Regarding Late Binding, the idea is that the persona that will be in charge of booting the hosts and maintaining the hardware inventory is not the same persona that will create the clusters. That way you can pre-boot the host and build pools of hosts that clusters can be built from. So, with the late binding option, an UI integration in OCM will allow the end user to select the hosts to create a new cluster. And finally, more improvement for scale support. What about 2000 SNO nodes? Maybe more?
• #17: Here are some useful links, with all the projects we mentioned, Open Cluster Managment, Hive, Assisted Installer, Metal Kube. The “ZTP the hard way repo” provides a walkthrough that will guide you if you want to try ZTP. A link to the assisted installer at redhat.com, we encourage you to try it. We also added some recordings of related talks.
• #18: This session is prerecorded, but we will be available for questions in the chat area right after it.
• #19: Thank you for listening, you are welcome to reach out and share feedbacks. Wishing you all successful installations. Thank you!