Red Hat OpenShift Container Platform - LPAR Deployment using Agent-based installer on IBM LinuxONE

This article objective is to be used as a complement to the Red Hat and IBM official documentation on how to deploy Red Hat OpenShift Container Platform (RHOCP) in a Logical Partition (LPAR) using the Agent-based installer deployment method.

Why use the Agent-based installer? It is a new available option and it requires less resources (no need for the bootstrap LPAR), and automate most of the install process, even when using User Provisioned Infrastructure (UPI). The Agent-based installer is one option out of already existing installation options available previously, like Assisted Installer, and full manual installation mode.

IBM LinuxONE ships with IBM Dynamic Partition Manager (DPM), which is a very easy to use interface to allow us to create Logical Partitions, and assign resources such as an Integrated Facility for Linux (IFLs), memory (RAM), and Storage (FCP/ECKD), network interfaces, etc... making this system not only a powerful enterprise Linux Server, but also easy to manage.

The environment we will create will consist of seven LPARs. An LPAR for a bastion server (where we will host our HAProxy load balancer), and three LPARs for the control nodes, and three LPARs for the compute nodes.

The image below will show you a high level architecture overview of the environment we used to document this install process:

From a storage perspective, on a real enterprise environment the storage administrator would provide the storage group zoning configuration for the LinuxONE team deploying RHOCP. The storage group zoning will not be covered in this article.

In our environment the Storage Area Network (SAN) will be connected to the IBM LinuxONE server using Fibre Channel via fabric layer connected to one or more I/O devices that will provide multiple paths of connection to avoid single point of failure when accessing the storage system. Each volume created inside the IBM Flash9500 storage system will have 4 paths.

Major Steps part of this article :

• Create Storage Request (DPM)

• Create the Logical Partitions (DPM)

• Setup the bastion server (HAProxy, Apache Web Server, Setup the Agent-based installer, etc.. (the bastion LPAR was provisioned with a RHEL OS)

• Initiate the Agent-based Install

• Login to the RHOCP web console

Let's begin:

Storage Request steps:

The process to open a request and later on configure disk volumes to be added to the respective LPARs is as follows:

Login to the Hardware Management Console (HMC) interface

Select the system that will be used for this configuration by selecting on the Systems Management option on the left:

In our case, we have many systems, but the one which uses the IBM DPM interface is the one identified as Hydra, select that system:

Creation of the Storage Request using the DPM Mode

Select Configure Storage option under Configuration in Tasks tab for the selected System:

This will open the following menu:

Create the requests directly by selecting the FICON & FCP icon. The environment used in this article uses SCSI disks on an IBM FS9500 Storage system, which means a FCP storage request will be created. Create a storage request for all LPARs part of this environment; control nodes (control 1, control 2 and control 3) and compute nodes (compute 1, compute 2, compute 3).

The following steps will show you how to create a storage request for a LPAR that will be used for the control nodes of RHOCP . The control LPARs will have just a single storage request of 120GB for the CoreOS root disk. The following example will show how to create a storage request that will be identified as control2-LPAR.

If this is a storage request for a control LPAR use the following:

The next step you will name this request:

Then confirm

Click Done

Do not click Finish yet, pay attention to the information about the WWPNs provided. The storage request creates a request that includes all the information needed by your storage system administrator that will be needed so the LUN zoning and disk allocation can be done for that particular request for that particular LPAR. In the Example above, shows the request for the control2-LPAR. Then click Finish for this request. The steps above only created the request for a single LPAR, in this example control2-LPAR, next open storage requests for the remaining LPARs: control1-LPAR, control3-LPAR, compute1-LPAR, compute2-LPAR and compute3-LPAR.

NOTE: the only difference when creating the request for the compute nodes, is that it will include 2 volumes instead of just one. One volume will be the 120GB for the CoreOS and the second volume will be at least 500GB. The additional volume for the compute nodes will be used later on as the foundation to install IBM Fusion Data Foundation software defined storage layer to provide the RHOCP with a dynamic allocation of volumes in either file, block or object storage within the RHOCP cluster. [The IBM Fusion Data Foundation will be covered in another article in the future]

Creating Logical Partitions (LPARs) on IBM LinuxONE

To create a new LPAR, log to the HMC, select Systems Management on the left and select the system you are working on. Once the system is selected, on the Task bar look for the configuration menu (at the bottom), look for the New Partition option:

When you select the New Partition you will be presented with the following screen below, click Next:

Next define the name of the LPAR. In this example we are creating the Control-3 LPAR with Partition Type Linux, then click Next:

Next define the number of logical IFLs this LPAR will have access to. In this example 4 logical IFLs will be used. Once CoreOS is installed, CoreOS will see 8 CPUs. Each Logical IFL is SMT2, so each logical IFL will provide 2 threads. Click Next:

Next define the amount of Memory (RAM) that will be assigned to this LPAR. In this example 16GB will be allocated for this LPAR.

Next define the network interface for this LPAR. Click on the plus (+) sign to open the menu with possible options:

Next provide a name for this network interface card, and select the correct adapter for your environment. In this example, a OSA-Express6S 10GbE was already pre-configured and the network already pre-defined for us, as this is a task for the network administrators, click okay button:

Click Next on this screen:

NOTE: The steps above demonstrated how to create an LPAR, Control-3. Now repeat the same steps for Control-1, Control-2, Compute-1, Compute-2 and Compute-3 LPARs.

At this point, we have almost all components to get our RHOCP install. Since this is an UPI (User Provisioned Infrastructure) install driven by the Agent-based installer we still need to provide the DNS records and an external load balancer.

The DNS records will be provided to us by our network administrator. This guide follows the recommendation from the official Red Hat OpenShift Documentation for the creation of the DNS records. If you need guidance on how to create and host your own DNS service, please follow the guidance of the following article.

This is the information provided to us by our network administrator about the DNS records for our environment:

This guide will cover the creation of a local load balancer that will be hosted inside our bastion server.

Setting up the Bastion server

The Bastion LPAR, was provisioned using RHEL OS and the process of provisioning this Linux system will not be covered although it will be very similar to the provisioning on CoreOS on the LPARs.

The Bastion server will have the following services:

• HAProxy

• Apache Web Server

Setting up the HAProxy:

Add the following lines to the end of the haproxy.cfg configuration file at /etc/haproxy/

Enable SELinux boolean:

Start and enable the haproxy service

Setting up the Apache Web Server:

Edit the httpd.conf configuration file:

Make sure the LISTEN parameter inside the httpd.conf looks like the following:

Enable SELinux boolean:

Start and enable the Apache server service:

Preparing the Red Hat OpenShift Agent-based installer process.

Install the nmstate package:

Download RHOCP installer. This article is using the RHOCP installer for LinuxONE version 4.19.1, but if you would like to download a different version, maybe a newer one, feel free to browse the Red Hat mirror repository.

Then

Download the RHOCP client

then

Copy the kubectl and oc files to the /usr/local/bin

Test oc client command line

Create the install-config.yaml file - This configuration file will create an RHOCP with 3 control plane nodes, and 3 compute nodes:

The pull secret can be copied from Red Hat Hybrid Cloud Console: https://console.redhat.com/openshift/install/ibmz/user-provisioned

To generate the sshKey key-pair:

Check the content of your generated public key:

Create the agent-config.yaml:

NOTE: This examples uses the master1/control1 as our rendezvous node as well. Think of it as our bootstrap server that will then be converted to the actual master1/control1 node. All this process is abstracted by the Agent-based installer.

NOTE: How to find you the MAC address for each OSA on each LPAR? Follow the steps below:

Click on the name of the LPAR, in this example Control-1:

A new tab will open with the Partition Details, select the Network section, and a NIC configuration will be presented. There is a section called MAC Address. To find out the Mac Address for all LPARs, repeat this process for all other LPARs part of the Red Hat OpenShift Cluster.

NOTE: Make sure to populate the agent-config.yaml with all the proper information. This example shows all items that will need to be changed for master1, make sure to change all the other sections with the relevant information for each particular node.

Generating the Agent-based Artifacts:

Requirements:

• install-config.yaml

• agent-config.yaml

• openshift-install

• openshift-client

Create a directory for the openshift files

This is a listing of all contents created so far:

Copy all yaml files to the opc-install directory and run the following command:

Check the contents of the ocp-install directory. notice you have the auth directory and the boot-artifacts directory. The auth directory contains your kubeadmin-password and your kubeconfig file:

The boot-artifacts directory contains the following:

The agent.s390x-generic.ins file contains the template for our .INS files that is required for the LPAR install. The method used in this article, renames these files to facilitate access during deployment. NOTE: You do not have to adopt this method, but I highly recommend.

Contents of agent.s390x-generic.ins:

The following example uses the agent.s390x-generic.ins as a template to create the control1.ins file that will be used for the LPAR that will host the control1/master1 node of the RHOCP.

control1.ins:

Do this process for all LPARs part of the RHOCP, each LPAR will have its own .INS file.

To re-enforce the task understanding, this is another example, now for control2.ins:

At the end of this task you should have the following:

Before move to the next section, as mentioned before, the contents of the .INS files were edited, so the actual files needs to comply with the same name and path. This guide uses a simple bash shell script to rename the files. Save the content below in a file named rename.sh and give it execution rights by issuing the chmod +x rename.sh command:

This is how the contents of the /root/ocp-install/boot-artifacts should look like:

NOTE: the file rootfs will be used by the next section, but for now, we will just move/copy the rootfs file to the following directory: /var/www/html/ocp:

NOTE: The next step will create PRM files for each LPAR. A .PRM file is used, during the instalation of the CoreOS, to specify kernel parameters and network configurations. These files help automate and streamline the installation process by providing initial setup details.

To create the PRM files for each LPAR. Inside the INS file there is a reference to a PRM file. For example, the control1.ins shows a reference to control1.prm file. In this section it will show how to create the PRM file for control1:

NOTE: The PRM file example above was formatted for readability. PRM files are a single line separated by spaces (There are no new lines on a PRM file)

Let's digest a few highlighted lines from example above:

This line references the rootfs file we moved from the boot-artifacts directory to the /var/www/html/ocp directory, which the Apache web server will serve via http when we initiate the install process using the Agent-based installer deployment type.

The line above, the first section 10.11.6.10 is the IP address of the control1/master1 node. The 10.11.0.1 is the IP of the gateway of the subnet this node resides, the 255.255.240.0 is the netmask of the subnet this node resides (in this case is /20).

Self explanatory. The line above makes reference to the IP address of the DNS server.

The lines above are references to the disk volume allocated to the LPAR where we will install control1/master1. We are using the FCP protocol, meaning, SCSI disks on a FS9500 storage system that offers multipath support. How do find out all these information? I will show you from the IBM DPM interface.

Where do I find this information about the FCP? Just as a review, this information is provided by the IBM DPM interface. Once logged in, select the partition you would like to see details about, in this case control1, then scroll down to the Storage Groups and select the storage group control1-LPAR:

The next screen will take you to the storage overview, where you will see the volumes that are part of this volume group. In this case, since this is a control/master node, you will only see a single volume of about 120GB:

Select Get Details option under Config Details:

The screen above shows all the paths to this volume, 4 paths for each controller on the storage system. All this information is likely to be provided to you by the Storage Administrator, and in this case when we look at the Target Port WWPN the last 2 digits identify what controller that path is from. So 88 is one, 89 is another. To complete the disk references on the PRM file, we will pick a path from each controller. In our case the Device No. 0005 was the lucky winner, so I got the path for that device number for each controller.

Pay attention that the way they are represented inside the PRM file, requires a certain format, so device number 0005 becomes 0.0.0005, WWPN 5005076813764789 becomes 0x5005076813764789 and LUN No. 0000000000000000 becomes 0x0000000000000000, to reference the disk you will also need to add the rd.zfcp=

The final result should look like this, for this example for control1/master1:

Repeat this process for all LPARs, remember to target just the 120GB volume. The second volume will be added to the compute/worker nodes on Day2.

The line above set the parameters needed to identify the logical addresses of the network interface (in this case a shared OSA device) that will be used by this LPAR.

To identify the logical network addresses for the network card, in this case a shared OSA, access the IBM DPM interface, select the LPAR control1 for example to get the details and scroll down to the section NICs:

Select the Adapter name (OSD 0110...), this will open the Adapter Details:

The section Network Interface Cards will show to what LPARs this network device is shared to, for example, for Control-1, the device number (the logical address) is 0001. This 0001 is "like a virtual network card".

To complete the PRM file with the proper format for the network logical addresses of this virtual network interface use the following as an example, the 0001 becomes 0.0.0001, and since these needs to be a triplet sequence, just continue the sequence as indicated, 0002 becomes 0.0.0002, 0003 becomes 0.0.0003 :

A little background on that:

On IBM LinuxONE systems, the network definition in a parameter (prm) file for Linux uses three logical addresses to identify a network device because of the architecture's use of channel-attached devices and the Channel Command Word (CCW) model. These three addresses correspond to the read, write, and data channels required for communication with the network device. Here's a detailed explanation:

Channel-Attached Devices in IBM LinuxONE:

IBM LinuxONE use a channel subsystem to manage I/O operations, including network communication. Network devices, such as OSA (Open Systems Adapter), are accessed through channel paths identified by device numbers.Each network device typically requires three distinct subchannels to handle different types of I/O operations: one for reading data, one for writing data, and one for control or status communication (often referred to as the data channel).

Three Logical Addresses:

The three logical addresses in the prm file (e.g., 0.0.0001,0.0.0002,0.0.0003) represent the device numbers assigned to the read, write, and data/control subchannels of the network device.

The IBM LinuxONE requires these three subchannels to ensure proper separation of I/O operations, which enhances reliability and performance in high-throughput LinuxONE environments.

The next task requires the creation of the PRM file for each LPAR. So each .INS file will have a .PRM file, apply the same concepts explained above to create the other files:

NOTE: the next step involves copying files you created to an FTP server. In our environment a FTP server was already available. If you do not have an FTP server, you can setup an FTP server on the bastion node.

Transfer all the .INS and PRM files to the FTP server:

From the boot-artifacts directory (/root/opc-install/boot-artifacts) and transfer the following files to the FTP server:

Listing the content of the FTP server should provide you a list of all .INS files, .PRM files and the initrd, initrd.addrsize and vmlinuz files:

Next modify each LPAR to search the FTP server and load the particular INS file for each LPAR. In the following example below we will modify the Control1 LPAR's configuration, so it will search the INS file from the FTP server previously populated with the content needed to proceed with this section. The Control1 LPAR will access the control1.ins file at the last section of the control1 LPAR configuration:

NOTE: Make sure to do this change to all the other partitions of your future Red Hat OpenShift Cluster; control2, control3, compute1, compute2, compute3 LPARs.

Once this change has been made to all the LPARs for this particular environment it is time to start each LPAR. Starting the LPAR is a very simple process, just select the LPAR you want to start, then from the Tasks menu at the bottom, select Start. A tab will open with that status of the starting process of the specific LPAR.

Repeat this process for all other LPARs; control2, control3, compute1, compute2 and compute3. The Agent-based install process, needs all LPARs initiated to complete the Red Hat OpenShift Cluster.

NOTE: This phase might take a while; around 20-30 minutes.

To follow the progress you can use the following command from your bastion system - this first command will show the progress until the bootstrap process is done:

After that, to monitor all other steps that the Agent-based installer will automate as well use the following command:

Once this process is done you can go to your bastion node or workstation open a browser and access the Red Hat OpenShift Web Console.

NOTE: The user login is kubeadmin and the password will be provided by the output of the install-complete command above, or you can go also find it inside your ocp-install directory under /root/ocp-install/auth directory in a file called: kubeadmin-password.

Red Hat OpenShift cluster is installed on the LPAR mode, both container workloads and virtual machine workloads can be used within this cluster. To leverage virtual machines, you will need to install the Red Hat OpenShift VIrtualization Operator, but before you do that, you will need to provide a persistent storage layer for OCP, so it can be used by the OpenShift Virtualization Operator and the VMs you will create.

The recommended solution for the persistent storage is either Red Hat ODF (which is bundled with Red Hat OpenShift Plus subscription) or IBM Fusion Data Foundation (FDF) as these options provide a variety of storage classes that covers file, block and object storage. IBM offers other options depending on the use case as well, such as IBM Fusion Scale.

This article is a guide to help you understand the process and possible challenges you will face when installing Red Hat OpenShift Container Platform on IBM LinuxONE on LPAR mode deployment and using the Agent-based installer install process.

References:

Red Hat Official Documentation

IBM Dynamic Partition Manager Documentation

Acknowledging contributors to this article:

Matt Mondics, Craig Harmon, Pat Fruth, Holger Wolf, Dominik Werle, Livio Sousa, Jay Brenneman, Richard Young, Wasif Mohammad

Legal Disclaimer: This content was provided for informational purposes only. The opinions and insights discussed are those of the authors and contributors and do not necessarily represent those of the IBM Corporation.

Nothing contained in these materials or the products discussed is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers, or altering the terms and conditions of any agreement you have with IBM.