OpenShift-Install Agent: ZTP Manifests

Author: Brandon B. Jozsa

"I would not give a fig for the simplicity this side of complexity, but I would give my life for the simplicity on the other side of complexity."
- Oliver Wendall Holmes Jr.

So you've read the post where I talking about ZTP for OpenShift Alpha Clusters, but you may have been thinking "I didn't see any ZTP manifests in that post". This is what we're going to cover in today's article, of course with a demonstration, and discuss why this could be beneficial.

Part I: Introduction

Most of the how required to generate an ISO from an install-config.yaml and agent.yaml were already discussed in the previous blog post, but I want to discuss some other considerations for generating installation media using openshift-install agent commands.

Part II: Parts of the Deployment

You may already be familiar with what Zero-Touch Provisioning is, and how Red Hat is leading this concept of git-instantiated bare metal deployments. But if you're unfamiliar with it, have a look at a generic ZTP deployment from a git repository (simply meaning that no secrets or pull secrets are included). You'll notice that a kustomization.yaml is used to control which manifests are deployed to production. Within the ztp-sno15 folder you'll find several important deployment manifests. Let me cover the important ones in more detail:

Yes, there are others but I will cover these in a more ZTP-focused post later. The manifests above are important, because we'll be using these for our demonstration today.

01-unsealed-pull-secret.yaml
This file is simply the pull secret used to pull images for OpenShift deployments. If you've ever deployed OpenShift before, than you have used this pull-secret in one way or another.

03-agentclusterinstall.yaml
AgentClusterInstall, is an object managed by the Hive Operator. What we're declaring in this manifest are a few key items for describing our cluster deployment. For example, we're declaring the version of OpenShift, what CNI we want to leverage along with CIDR ranges, what SSH public key will have access to the cluster during/after the deployment.

04-clusterdeployment.yaml
ClusterDeployment, is also an object managed by the Hive Operator. This manifest describes the name of the cluster along with base-DNS name, it declares the pull-secret used to obtain OpenShift container images, and most importantly it declares what platform the cluster will be deployed within, in this case bare metal.  

07-nmstate.yaml
NMStateConfig is a declarative network configuration model, which is controlled through the OpenShift/Kubernetes API. This is used to configure the networking on an RHCOS host.

08-infraenv.yaml
InfraEnv is managed via the Infrastructure Operator, or as we typically call it the Assisted-Service (Assisted-Installer). InfraEnv can include all sorts of things from ignition overrides to proxy overrides, but in general this object tells the assisted-installer the basics of the installation request.

There you have it! These are arguably the most important artifacts required to deploy a cluster. Why did we cover these? Because we're going to create simple variations of these to deploy a sample cluster without the need for an install-config.yaml or agent.yaml.

For this one, we're going to jump right into the demonstration, but I'll continue to link other helpful posts where applicable.

Part III: Demonstration

I'm going to deploy the same exact cluster as I did in my previous blog post. The reason for doing this is to make things easy to read and follow along. So, our cluster name will still be ztp-ocpsno58. Let's create these deployment files exactly the same way, so you have some level of success, and then you can modify them to your own needs (I'll also cover some caveats with this MVP build of openshift-install agent utility.

Make sure you follow the same exact instructions prior to creating/using these manifests (consider these prerequisites):

Step 1: Download the Binary

gdown 1-rJzzyVa9rWvw3yeAuGsTfaaSFLcMY6x
tar zxvf openshift-install.tgz
sudo chmod +x openshift-install
sudo chown root:root openshift-install
sudo mv openshift-install /usr/local/bin/openshift-install
PATH=$PATH:/usr/local/bin 

Step 2: Install Perquisite Packages

sudo yum install genisoimage nmstate -yy

Step 3: Create Deployment Directory Structure

Slightly different from before, you will need to create a directory structure like this for the deployment manifests:

[root@aio demo]# mkdir -p {ztp-ocpsno58/cluster-manifests,ztp-ocpsno58-backup/cluster-manifests}
[root@aio demo]# tree .
.
├── ztp-ocpsno58
│   └── cluster-manifests
└── ztp-ocpsno58-backup
    └── cluster-manifests

4 directories, 0 files
[root@aio demo]# 

Step 4: Create the Following Manifests

The following manifests are mandatory. I will explain some adaptations, and where the state of things are with this MVP later in the post.

PullSecret

In the same directory above (view the tree output), write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/pull-secret.yaml
---
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: pull-ztp-ocpsno58
  namespace: cluster0
stringData:
  .dockerconfigjson: "$OPENSHIFT_PULL_SECRET"
EOF

IMPORTANT: Make sure that you've replaced $OPENSHIFT_PULL_SECRET with your actual pull secret.

AgentClusterInstall

In the same directory above (view the tree out) write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/agent-cluster-install.yaml
---
apiVersion: extensions.hive.openshift.io/v1beta1
kind: AgentClusterInstall
metadata:
  name: ztp-ocpsno58
  namespace: cluster0
spec:
  clusterDeploymentRef:
    name: ztp-ocpsno58
  imageSetRef:
    name: openshift-4.11
  networking:
    clusterNetwork:
      - cidr: "10.128.0.0/14"
        hostPrefix: 23
    serviceNetwork:
      - "172.31.0.0/16"
    machineNetwork:
      - cidr: "192.168.3.0/24"
  provisionRequirements:
    controlPlaneAgents: 1
    workerAgents: 0
  sshPublicKey: "$SSH_PUBLIC_KEY"
EOF 

IMPORTANT: Make sure that you've replaced $SSH_PUBLIC_KEY with your actual SSH public key.

ClusterDeployment

In the same directory above (view the tree out) write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/cluster-deployment.yaml
---
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
  name: ztp-ocpsno58
  namespace: cluster0
spec:
  baseDomain: telco.ocp.run
  clusterName: ztp-ocpsno58
  controlPlaneConfig:
    servingCertificates: {}
  installed: false
  clusterInstallRef:
    group: extensions.hive.openshift.io
    kind: AgentClusterInstall
    name: ztp-ocpsno58
    version: v1beta1
  platform:
    agentBareMetal:
      agentSelector:
        matchLabels:
          cluster-name: "ztp-ocpsno58"
  pullSecretRef:
    name: pull-ztp-ocpsno58
EOF 

ClusterImageSet

In the same directory above (view the tree out) write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/cluster-image-set.yaml
---
apiVersion: hive.openshift.io/v1
kind: ClusterImageSet
metadata:
  name: openshift-4.11
spec:
  releaseImage: quay.io/openshift-release-dev/ocp-release:4.11.0-x86_64
EOF 

InfraEnv

In the same directory above (view the tree out) write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/infraenv.yaml
---
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: ztp-ocpsno58
  namespace: cluster0
  annotations:
    argocd.argoproj.io/sync-options: Validate=false
spec:
  clusterRef:
    name: ztp-ocpsno58
    namespace: cluster0
  sshAuthorizedKey: '$SSH_PUBLIC_KEY'
  pullSecretRef:
    name: pull-ztp-ocpsno58
  nmStateConfigLabelSelector:
    matchLabels:
      cluster-name: ztp-ocpsno58
EOF 

NMStateConfig

In the same directory above (view the tree out) write the following file:

cat << EOF > ./ztp-ocpsno58/cluster-manifests/nmstateconfig.yaml
---
apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: vm-ocp-node58
  namespace: openshift-machine-api
  labels:
    cluster-name: ztp-ocpsno58
spec:
  config:
    interfaces:
      - name: enp1s0
        type: ethernet
        state: up
        ipv4:
          enabled: true
          address:
            - ip: 192.168.3.58
              prefix-length: 24
          dhcp: false
        ipv6:
          enabled: false
    dns-resolver:
      config:
        search:
          - telco.ocp.run
        server:
          - 192.168.3.5
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 192.168.3.1
          next-hop-interface: enp1s0
          table-id: 254
  interfaces:
    - name: "enp1s0"
      macAddress: "0e:00:00:03:30:58"
EOF 

The result should look like the following tree output:

[root@aio demo]# tree .
.
├── ztp-ocpsno58
│   └── cluster-manifests
│       ├── agent-cluster-install.yaml
│       ├── cluster-deployment.yaml
│       ├── cluster-image-set.yaml
│       ├── infraenv.yaml
│       ├── nmstateconfig.yaml
│       └── pull-secret.yaml
└── ztp-ocpsno58-backup
    └── cluster-manifests

4 directories, 6 files
[root@aio demo]# 

It's at this time where you may want to do yourself a favor, and backup the manifests into the ztp-ocpsno58-backup that I had you create earlier:

[root@aio demo]# cp ztp-ocpsno58/cluster-manifests/* ztp-ocpsno58-backup/cluster-manifests/
[root@aio demo]# tree .
.
├── ztp-ocpsno58
│   └── cluster-manifests
│       ├── agent-cluster-install.yaml
│       ├── cluster-deployment.yaml
│       ├── cluster-image-set.yaml
│       ├── infraenv.yaml
│       ├── nmstateconfig.yaml
│       └── pull-secret.yaml
└── ztp-ocpsno58-backup
    └── cluster-manifests
        ├── agent-cluster-install.yaml
        ├── cluster-deployment.yaml
        ├── cluster-image-set.yaml
        ├── infraenv.yaml
        ├── nmstateconfig.yaml
        └── pull-secret.yaml

4 directories, 12 files
[root@aio demo]# 

Step 5: Generate ISO Installation Media

Now that all the hard work is complete, let's generate the installation media from our ZTP manifests. Before we do this though, make sure you use a variable to override the default image that's hard-coded in the openshift-install agent subcommand.

Set the following variable:

export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE="quay.io/openshift-release-dev/ocp-release:4.11.0-x86_64"

Notice that this image is the same as what we declared in the cluster-image-set.yaml file. Keep this in mind if you ever want to change OpenShift versions, or if you get an error that mentions the following:

DEBUG Using internal constant for release image quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4
ERROR failed to write asset (Agent Installer ISO) to disk: image reader not available
FATAL failed to fetch Agent Installer ISO: failed to load asset "ClusterImageSet Config": invalid ClusterImageSet configuration: Spec.ReleaseImage: Forbidden: value must be equal to quay.io/openshift-release-dev/ocp-release@sha256:300bce8246cf880e792e106607925de0a404484637627edf5f517375517d54a4

Now that we've covered this override variable, let's build the ISO installation media with the following final command:

/usr/local/bin/openshift-install agent create image --log-level debug --dir ztp-ocpsno58 

And that should do it! After everything has run your directory should contain the ISO installation media and auth directory.

[root@aio demo]# tree .
.
├── ztp-ocpsno58
│   ├── agent.iso
│   └── auth
│       └── kubeconfig
└── ztp-ocpsno58-backup
    └── cluster-manifests
        ├── agent-cluster-install.yaml
        ├── cluster-deployment.yaml
        ├── cluster-image-set.yaml
        ├── infraenv.yaml
        ├── nmstateconfig.yaml
        └── pull-secret.yaml

4 directories, 8 files
[root@aio demo]# 

Now you can follow the rest of my instructions, if you're using a virtual machine or you can attempt to boot to this ISO using bare metal. Once you've tried this a couple of times, and understand the workflow, you can customize this deployment for your environment. There are a couple of caveats right now, as the binary is still in dev-preview and currently in MVP (as of 2022/08/22).

  • The file names need to be exact at this time. This will change in the near future.
  • Don't try to use your typical ZTP manifests, if you're currently using ZTP. There is currently a bug open which will prevent this from working (some of the keys for ZTP/ACM aren't being ignored during the YAML marshal process). This too should change very soon.

Final Thoughts

So let me know what you think about using ZTP manifests for a localized openshift-install agent deployment. I see some very interesting use cases brewing as a result of this work, but really, I'd like to get your thoughts! If you want to find me, I'm on Twitter or eMail.

Thanks for taking the time to read my post!