Author: Brandon B. Jozsa

"Filling a need is not merely good for business; it's a basic attitude towards life."
- George Forman

Part I: Introduction

If someone were to ask me "What do you like best about working at Red Hat", I would summarize with a few thoughts:

• The open culture (it's in the company DNA)
• Great people (both technically and just overall)
• General willingness to discuss and listen to feedback
• Autonomy to find and solve gaps with open source software

All of the above describe the topic I'm going to write about this week. It's about a small project called aicli. This project was created and it primarily maintained by Karim Boumedhel, who's been with Red Hat quite a while. In general, the project's aim is to simplify the user experience for the Assisted-Service API. If you just need a basic cluster, or even a Single-Node OpenShift cluster (SNO), than the Assisted-Installer UI is hands-down the best option, and I'll cover why this is the case in this blog post.

There are already many different ways for automating cluster deployments via the Assisted-Service/Installer. Users could leverage the OpenShift Assisted-Installer Test Infrastructure Ansible Playbooks, or they could consider more telco-based scenarios (such as Edge, RAN, or disconnected installs) by using the Global Telco Team's Ansible Playbooks. But with so many resources available, there's a gap in which Karim's project solves: it's when a customer wants to either customize the deployment, or if they want to use static IP addresses with the Assisted-Installer service. These options are available through the API, as I have covered in previous articles, but they're not present in the UI. So let's cover how this project aims to solve this problem.

Part II: Installation

To install the aicli, you will need a couple of small python packages/dependancies. We'll use pip3 to install these packages.

sudo pip3 install -U aicli
sudo pip3 install -U assisted-service-client

The first package is pretty self-explanatory; it's the client utility, or what you will interact with. The second package, assisted-service-client, is the client library for the Assisted-Service project. If you haven't familiarized yourself with the Assisted-Service API yet, I would highly recommend that you review my previous article about Creating OpenShift Clusters with the Assisted-Service API. You may also want to review the Swagger models as well. These models will tell you want a CLI utility could provide, and it could help. you understand if any features need to be added to Karim's project.

Part III: Working with Clusters

Once installed, you're going to want to start using the client utility. As I described in my previous article, you will want to grab the OpenShift Offline Token. This is how you will authenticate against the Assisted-Service API at cloud.redhat.com.

Next, you will also need your OpenShift Pull Secret. By default, the aicli utility will look for a openshift_pull.json token file in the local current working directory (download to pwd). This naming is important, as you can see below.

❯ pwd
/Users/bjozsa/aicli-demo

❯ ls -asl
total 4
0 drwxr-xr-x    3 bjozsa staff   96 Sep 30 10:37 .
0 drwxr-xr-x+ 131 bjozsa staff 4192 Sep 30 10:37 ..
4 -rw-r--r--    1 bjozsa staff 2727 Sep 30 10:37 openshift_pull.json
❯

Next, export a variable as an OFFLINETOKEN. This offline token will be converted to a usable TOKEN. This TOKEN will be used when communicating to cloud.redhat.com in realtime. Here's an example of this below:

❯ export OFFLINETOKEN="eyJhbGciOiJIUzI1NiIsInR5cCIgOi..."

❯ alias aicli="aicli --offlinetoken $OFFLINETOKEN"

❯ aicli list clusters
Using https://api.openshift.com as base url
Storing new token in /Users/bjozsa/.aicli/token.txt
+------------+--------------------------------------+-------------------+-------------------+
|  Cluster   |                  Id                  |       Status      |     Dns Domain    |
+------------+--------------------------------------+-------------------+-------------------+
| calico-poc | 8a1e2c80-296c-4739-be1b-8209dcc8be42 | pending-for-input |     jinkit.com    |
|    demo    | 0aedcee8-3eb4-443a-acc0-e39027cecc4f |     installed     | ztp.telco.ocp.run |
+------------+--------------------------------------+-------------------+-------------------+

❯ 

If you noticed from above, the aicli is intelligent enough to convert this custom OFFLINETOKEN to a TOKEN automatically, which is then stored as $HOME/.aicli/token.txt.

Creating Clusters

As I covered earlier, a requirement for creating a cluster is to download your OpenShift Pull Secret locally and in the present working directory. Even though this is the default behavior of aicli, you also have the option to provide this pull-secret as a custom --param (or -P). This can be especially useful if you commonly keep your pull-secret in a specific location for reuse. Here's an example of using this flag:

❯ aicli create cluster aicli-demo -P pull_secret=$HOME/aicli-demo/openshift_pull.json
Using https://api.openshift.com as base url
Creating cluster aicli-demo
Storing new token in /Users/bjozsa/.aicli/token.txt

❯ aicli list clusters
Using https://api.openshift.com as base url
+------------+--------------------------------------+-------------------+-------------------+
|  Cluster   |                  Id                  |       Status      |     Dns Domain    |
+------------+--------------------------------------+-------------------+-------------------+
| aicli-demo | 6516458e-40bf-49f7-9f4c-1f753307cb5f | pending-for-input |   karmalabs.com   |
| calico-poc | 8a1e2c80-296c-4739-be1b-8209dcc8be42 | pending-for-input |     jinkit.com    |
|    demo    | 0aedcee8-3eb4-443a-acc0-e39027cecc4f |     installed     | ztp.telco.ocp.run |
+------------+--------------------------------------+-------------------+-------------------+

❯ 

One thing you may have noticed from the output above is that the domain for this new example cluster is karmalabs.com. This is coming from a set of default_cluster_params within the project. These defaults can be updated in a couple different ways. The most common would be to offer these parameters at the initial cluster creation time, like this:

aicli create cluster new-aicli-demo \
  -P pull_secret=$HOME/aicli-demo/openshift_pull.json \
  -P base_dns_domain=jinkit.com \
  -P network_type=OVNKubernetes \
  -P cluster_network_cidr=10.128.0.0/14 \
  -P service_network_cidr=172.30.0.0/16 \
  -P cluster_network_host_prefix=23 \
  -P vip_dhcp_allocation=false
  

However, you also have the option to update cluster settings as well, which I will cover in the next section.

Updating Clusters

You can use the update command to update a cluster that's already been created via the aicli:

aicli update cluster new-aicli-demo \
  -P pull_secret=$HOME/aicli-demo/openshift_pull.json \
  -P base_dns_domain=jinkit.com \
  -P network_type=OVNKubernetes \
  -P cluster_network_cidr=10.128.0.0/14 \
  -P service_network_cidr=172.30.0.0/16 \
  -P cluster_network_host_prefix=23 \
  -P vip_dhcp_allocation=false

Using Parameter Files

Another option is to leverage a parameter file. This can be particularly handy if you wish to repeatedly build the same type of cluster (for testing purposes, as an example). Let's try this using a params.yaml file like below:

aicli create cluster newest-aicli-demo \
  -P pull_secret=$HOME/aicli-demo/openshift_pull.json \
  --paramfile newest-aicli-demo.yaml

base_dns_domain: jinkit.com
network_type: OVNKubernetes
cluster_network_cidr: "10.128.0.0/14"
service_network_cidr: "172.30.0.0/16"
cluster_network_host_prefix: 23
vip_dhcp_allocation: false

Reviewing Clusters

Now that we've made all of these changes, you will likely want to review them. You can get quite a bit of information by reviewing a given cluster with the aicli command. For example, if you want to see what the progression state of a given cluster is, you could issue the following command:

❯ aicli info cluster new-aicli-demo | grep progress
progress: {'total_percentage': None, 'preparing_for_installation_stage_percentage': None, 'installing_stage_percentage': None, 'finalizing_stage_percentage': None}

I would  suggest taking a look at this information as you work with aicli,  because the parameters that you can update (either with a parameter file or inline parameters) can be viewed and also used during initial instantiation of a cluster, either when updating or initially creating clusters.

Static IP Addresses

Now we're getting into more "real-world" situations, and truthfully speaking, this is where the aicli command really shines when compared to using the UI-only Assisted-Installer.

I would love to see everyone leveraging DHCP for bare-metal deployments. After all, DHCP and DNS are very "cloudy" types of services. DHCP allows for dynamic auto-scaling (yes, even for bare metal). But not everyone can use DHCP, and many customers still use static IP addressing on their network.

Very important PSA: the curl examples I outlined in my previous blog post, custom Ansible playbooks, or by using the aicli utility are the only real ways (that I know of) which will be able to deploy static IP addressing via the Assisted-Installer. The UI on cloud.redhat.com just doesn't have any UI-elements to include static IP address information, as it stands today. This could chance in the future, but this is the way things are currently.

Considering this PSA, how can static IP addressing be achieved? The answer is with providing a static_network_config YAML document as part of the ISO generation process. Here is an example for a Single-Node OpenShift deployment. We will name this file host-x.yaml:

static_network_config:
 - dns-resolver:
     config:
       server:
       - 192.168.1.70
   interfaces:
   - name: eno1
     mac-address: 3c:ec:ef:43:4d:6c
     ipv4:
       address:
       - ip: 192.168.3.32
         prefix-length: 24
       dhcp: false
       enabled: true
     ipv6:
       enabled: false
     state: up
     type: ethernet
   - name: eno2
     mac-address: 3c:ec:ef:43:4d:6d
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno3
     mac-address: 3c:ec:ef:43:4d:6e
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno4
     mac-address: 3c:ec:ef:43:4d:6f
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno5
     mac-address: 3c:ec:ef:43:45:90
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno6
     mac-address: 3c:ec:ef:43:45:91
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno7
     mac-address: 3c:ec:ef:43:45:92
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   - name: eno8
     mac-address: 3c:ec:ef:43:45:93
     ipv4:
       enabled: false
     ipv6:
       enabled: false
     state: down
     type: ethernet
   routes:
     config:
     - destination: 0.0.0.0/0
       next-hop-address: 192.168.3.1
       next-hop-interface: eno1
       table-id: 254

You can update the cluster/host with these settings like this:

❯ aicli create iso newest-aicli-demo --paramfile host-x.yaml
Using https://api.openshift.com as base url
Creating Iso for Cluster newest-aicli-demo
Storing new token in /Users/bjozsa/.aicli/token.txt
Iso available at https://api.openshift.com/api/assisted-install/v1/clusters/fccd6a47-3044-4a36-89c0-f0c83eecf30f/downloads/image

Next, download the installation media (LiveISO) with the following command:

aicli download iso newest-aicli-demo

After issuing the command above, what you find an ISO in the current directory with the cluster_name included as part of the filename.

API and VIP IP Addresses

Once the cluster hosts have been checked into the GUI, you can update the AIP and VIP addresses with the following command. Keep in mind that api_vip and ingress_vip cannot be set until each of the nodes have checked into the Assisted-Service API. This is because the Assisted-Service API will validate each of the hosts, and ensure they are on the same subnet/broadcast domain.

aicli update cluster newest-aicli-demo \
 -P api_vip=192.168.3.230 \
 -P ingress_vip=192.168.3.231

Customizations

Let's take another scenario which can come up occasionally - custom OpenShift integrations. If you're familiar with User Provisioned Infrastructure (UPI) and have ever deployed a custom CNI, such as Calico or Cilium, you know that these CNIs need to be deployed along with the customer (or at the time of instantiation). This is achieved by dropping custom manifests into the manifests directory, which is created with the openshift-install command. The Assisted-Service isn't much different, in the fact that it leverages a manifests directory where all of the core OpenShift/Kubernetes manifests are stored.  Just like with UPI, the Assisted-Service allows administrators to uploaded new/custom manifests into this directory.

What makes this process a bit more challenging is that the Assisted-Service requires each of these manifests to be base64 encoded prior to uploading them. I describe this process in my previous blog posts (both for Calico and for Cilium). Where the aicli improves on this process is that this is done automatically as part of the aicli update cluster process. Let's describe this process below.

Custom Manifests

Since we're working with the manifests folder, first create the directory locally. For the sake of our demonstration and to keep things simple, create this directory in the current working directory (where you've been keeping your other parameters files, etc).

mkdir -p ./manifests

Let's use Calico's documentation for the next few steps, just as an example. However, this can be done with any CNI provider, like Cilium as well. Just copy these customized CNI manifests into to the manifests directory. Here's an example of this (as of 2021-09-30):

curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-apiserver.yaml -o manifests/01-crd-apiserver.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-installation.yaml -o manifests/01-crd-installation.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-imageset.yaml -o manifests/01-crd-imageset.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/01-crd-tigerastatus.yaml -o manifests/01-crd-tigerastatus.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_bgpconfigurations.yaml -o manifests/crd.projectcalico.org_bgpconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_bgppeers.yaml -o manifests/crd.projectcalico.org_bgppeers.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_blockaffinities.yaml -o manifests/crd.projectcalico.org_blockaffinities.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_clusterinformations.yaml -o manifests/crd.projectcalico.org_clusterinformations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_felixconfigurations.yaml -o manifests/crd.projectcalico.org_felixconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_globalnetworkpolicies.yaml -o manifests/crd.projectcalico.org_globalnetworkpolicies.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_globalnetworksets.yaml -o manifests/crd.projectcalico.org_globalnetworksets.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_hostendpoints.yaml -o manifests/crd.projectcalico.org_hostendpoints.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamblocks.yaml -o manifests/crd.projectcalico.org_ipamblocks.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamconfigs.yaml -o manifests/crd.projectcalico.org_ipamconfigs.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ipamhandles.yaml -o manifests/crd.projectcalico.org_ipamhandles.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_ippools.yaml -o manifests/crd.projectcalico.org_ippools.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_kubecontrollersconfigurations.yaml -o manifests/crd.projectcalico.org_kubecontrollersconfigurations.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_networkpolicies.yaml -o manifests/crd.projectcalico.org_networkpolicies.yaml
curl https://docs.projectcalico.org/manifests/ocp/crds/calico/kdd/crd.projectcalico.org_networksets.yaml -o manifests/crd.projectcalico.org_networksets.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/00-namespace-tigera-operator.yaml -o manifests/00-namespace-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-rolebinding-tigera-operator.yaml -o manifests/02-rolebinding-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-role-tigera-operator.yaml -o manifests/02-role-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-serviceaccount-tigera-operator.yaml -o manifests/02-serviceaccount-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-configmap-calico-resources.yaml -o manifests/02-configmap-calico-resources.yaml
curl https://docs.projectcalico.org/manifests/ocp/tigera-operator/02-tigera-operator.yaml -o manifests/02-tigera-operator.yaml
curl https://docs.projectcalico.org/manifests/ocp/01-cr-installation.yaml -o manifests/01-cr-installation.yaml
curl https://docs.projectcalico.org/manifests/ocp/01-cr-apiserver.yaml -o manifests/01-cr-apiserver.yaml

In this example, since we're changing the default CNI to Calico, I need to update the parameter for network_type to Calico. This can be done with. the following command:

aicli update cluster calico-poc -P network_type=Calico

Next, update the cluster with the files you've just added into the manifest directory:

aicli create manifests calico-poc --dir manifests

When the upload is complete, you can review each of the files by listing them out and validating that they have in fact been uploaded properly with the following command:

aicli list manifests calico-poc

Example output:

❯ aicli list manifests calico-poc
Using https://api.openshift.com as base url
Retrieving manifests for Cluster calico-poc
Storing new token in /Users/bjozsa/.aicli/token.txt
+----------------------------------------------------------+-----------+
|                           File                           |   Folder  |
+----------------------------------------------------------+-----------+
|            00-namespace-tigera-operator.yaml             | manifests |
|                   01-cr-apiserver.yaml                   | manifests |
|                 01-cr-installation.yaml                  | manifests |
|                  01-crd-apiserver.yaml                   | manifests |
|                   01-crd-imageset.yaml                   | manifests |
|                 01-crd-installation.yaml                 | manifests |
|                 01-crd-tigerastatus.yaml                 | manifests |
|            02-configmap-calico-resources.yaml            | manifests |
|               02-role-tigera-operator.yaml               | manifests |
|           02-rolebinding-tigera-operator.yaml            | manifests |
|          02-serviceaccount-tigera-operator.yaml          | manifests |
|                 02-tigera-operator.yaml                  | manifests |
|       crd.projectcalico.org_bgpconfigurations.yaml       | manifests |
|           crd.projectcalico.org_bgppeers.yaml            | manifests |
|        crd.projectcalico.org_blockaffinities.yaml        | manifests |
|      crd.projectcalico.org_clusterinformations.yaml      | manifests |
|      crd.projectcalico.org_felixconfigurations.yaml      | manifests |
|     crd.projectcalico.org_globalnetworkpolicies.yaml     | manifests |
|       crd.projectcalico.org_globalnetworksets.yaml       | manifests |
|         crd.projectcalico.org_hostendpoints.yaml         | manifests |
|          crd.projectcalico.org_ipamblocks.yaml           | manifests |
|          crd.projectcalico.org_ipamconfigs.yaml          | manifests |
|          crd.projectcalico.org_ipamhandles.yaml          | manifests |
|            crd.projectcalico.org_ippools.yaml            | manifests |
| crd.projectcalico.org_kubecontrollersconfigurations.yaml | manifests |
|        crd.projectcalico.org_networkpolicies.yaml        | manifests |
|          crd.projectcalico.org_networksets.yaml          | manifests |
+----------------------------------------------------------+-----------+

Part IV: Conclusion

Hopefully through these aicli examples, you can see how powerful the utility is. If you've already been working with the Assisted-Service API, either on your own or by running through some of the examples I've suggested on this blog, then you will quickly realize that this utility greatly improves the user experience!

In closing, I'd like to get back to the part where I spoke about the great people I work with at Red Hat - I really commend Karim. He recognized that there was a gap with the user experience with the Assisted-Installer. Even though the Assisted-Installer/Service is an incredible project on it's own, it really benefits and it's capabilities are only highlighted more from his project.

I would highly suggest you check out Karim's project. Use it if you install bare metal OpenShift clusters. Perhaps more importantly, if you have ideas that you'd like to share, submit an issue or pull-request. I've asked him for assistance recently and he was incredibly responsive, and even offered to help out using tmate (which I'm a huge fan of).

So try out the project and let us know what you think of it. For me it's a no-brainer. Any clusters that I create using the Assisted-Service API will be using the aicli. Big shout out to Karim for finding opportunities to improve on AI. I'm really happy to work with such great people.