Cilium Installation: OpenShift (Assisted-Installer)

Topic: Assisted-Installer Service May 29, 2021

Author: Brandon B. Jozsa

eBPF is the real deal. Thomas Graf has worked really hard to make sure that you've heard of eBPF and how it can improve Kubernetes networking significantly. He and his team have been laying down this foundation, with incredible foresight into the future, for years. While others were saying, "you have to patch the kernel", or "you have to wait for kernel version", he was slowing waiting for this day to come. Chances are that you're ready for what he and the team have been building this entire time.

While I'm not going to get into the amazing benefits of eBPF or XDP, or what makes it so incredible (I'll cover this a bit deeper later), I do want to show you how to install it for OpenShift Assisted-Installer. I highly encourage you to explore Cilium, and understand what makes eBPF (and the story of Isovalent) so incredible. So let's get started.

This blog post will describe how to install the following:

  • OpenShift 4.8.0-fc.3
  • Single or Multi-node OpenShift (introduced in 4.8.x)
  • Cilium as the default CNI
  • All via the Assisted-Installer over REST calls (think automation)

To access the assisted-installer, log into your cloud.redhat.com account. This will work for 60-day evaluations as well. If you have questions, please leave them in the comments below. So let's get started!

Assisted-Service Red Hat Cloud
Red Hat Token

Infrastructure Requirements

Networking Setup (Static IP Addressing)
Networking Setup (DHCP - Automatic)
Load Balancing Setup (HAProxy Example)

Part I - Create a Cluster in Assisted-Service

In order to use the Assisted-Service API via cloud.redhat.com, you will need to create a bearer token. Red Hat provides a user-level "OpenShift Cluster Manager API Token" which can be used to provide a bearer token.

  1. Once you have created an OpenShift Cluster Manager API Token, use the "Copy to clipboard" function and provide it as the variable OFFLINE_ACCESS_TOKEN.
OFFLINE_ACCESS_TOKEN="<PASTE_TOKEN_HERE>"

export TOKEN=$(curl \
--silent \
--data-urlencode "grant_type=refresh_token" \
--data-urlencode "client_id=cloud-services" \
--data-urlencode "refresh_token=${OFFLINE_ACCESS_TOKEN}" \
https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token | \
jq -r .access_token)

IMPORTANT: This token will expire often (5 minutes). So if you recieve a 400 - Token is expired, reissue the command above.

VERIFY: You can check to see if the TOKEN variable is set via the following conditional:

if [ -z ${TOKEN+x} ]; 
  then echo "Token is undefined. Please check formatting before continuing\!"; else echo "Token is ready\!"; 
fi
  1. Review the following contents, and modify them as needed. These variables will be used throughout the rest of the demonstration and have been test to work:
ASSISTED_SERVICE_API="api.openshift.com"
CLUSTER_VERSION="4.7"
CLUSTER_IMAGE="quay.io/openshift-release-dev/ocp-release:4.7.9-x86_64"
CLUSTER_NAME="cilium-poc"
CLUSTER_DOMAIN="jinkit.com"
CLUSTER_NET_TYPE="Cilium"
CLUSTER_CIDR_NET="10.128.0.0/14"
CLUSTER_CIDR_SVC="172.30.0.0/16"
CLUSTER_HOST_NET="192.168.3.0/24"
CLUSTER_HOST_PFX="23"
CLUSTER_WORKER_HT="Enabled"
CLUSTER_WORKER_COUNT="0"
CLUSTER_MASTER_HT="Enabled"
CLUSTER_MASTER_COUNT="0"
CLUSTER_SSHKEY='<PUT-PUBLIC-SSH-KEY-HERE-AND-LEAVE-SINGLE-QUOTES>'

VERIFY: Verify that you can correctly talk with the API with the following curl command (install jq before running this command):

curl -s -X GET "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -H "accept: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq -r

If you are able to verify/return a response via the command above, you can continue to the next steps and deploy a cluster with modifications!

  1. Next, download your pull_secret from the following URL: https://cloud.redhat.com/openshift/install/pull-secret

HINT: All of these instructions are intended to be run from the current directory. Considering this, make sure that the pull-secret.txt and installation are where you want them to be before continuing.

  1. Next, create a variable with the raw contents of your pull-secret.txt file. This is important, because escape characters should be included as part of this output.
PULL_SECRET=$(cat pull-secret.txt | jq -R .)
  1. Now create an Assisted-Service deployment .json file:
cat << EOF > ./deployment.json
{
  "kind": "Cluster",
  "name": "$CLUSTER_NAME",
  "openshift_version": "$CLUSTER_VERSION",
  "ocp_release_image": "$CLUSTER_IMAGE",
  "base_dns_domain": "$CLUSTER_DOMAIN",
  "hyperthreading": "all",
  "cluster_network_cidr": "$CLUSTER_CIDR_NET",
  "cluster_network_host_prefix": $CLUSTER_HOST_PFX,
  "service_network_cidr": "$CLUSTER_CIDR_SVC",
  "user_managed_networking": true,
  "vip_dhcp_allocation": false,
  "host_networks": "$CLUSTER_HOST_NET",
  "hosts": [],
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF

HINT: If you recieve an error that you cannot overwrite the file, then make sure to run setopt clobber in your shell.

HINT: There's a new option in OpenShift v4.8.x that allows you to create a single node OpenShift cluster (referred to as SNO). If this is what you want, then you will need to add the following lines to the ./deployment.json file:

  "high_availability_mode": "None"

As an example, a SNO deployment will look like this:

cat << EOF > ./deployment.json
{
  "kind": "Cluster",
  "name": "$CLUSTER_NAME",
  "openshift_version": "$CLUSTER_VERSION",
  "ocp_release_image": "$CLUSTER_IMAGE",
  "base_dns_domain": "$CLUSTER_DOMAIN",
  "hyperthreading": "all",
  "cluster_network_cidr": "$CLUSTER_CIDR_NET",
  "cluster_network_host_prefix": $CLUSTER_HOST_PFX,
  "service_network_cidr": "$CLUSTER_CIDR_SVC",
  "user_managed_networking": true,
  "vip_dhcp_allocation": false,
  "high_availability_mode": "None",
  "host_networks": "$CLUSTER_HOST_NET",
  "hosts": [],
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF
  1. Create the cluster via the Assisted-Service API:
curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -d @./deployment.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.id'
  1. IMPORTANT: that this will generate a CLUSTER_ID, which will need to be exported for future use. Export this variable from the output of the previous command in Step 5:
curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters" \
  -d @./deployment.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.id'
"0da7cf59-a9fd-4310-a7bc-97fd95442ca1"

CLUSTER_ID="0da7cf59-a9fd-4310-a7bc-97fd95442ca1"
  1. Now update the cluster install-config via the Assisted-Service API:
curl \
  --header "Content-Type: application/json" \
  --request PATCH \
  --data '"{\"networking\":{\"networkType\":\"Cilium\"}}"' \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/install-config"

VERIFY: You can review your changes by issuing the following curl request:

curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/install-config" \
  | jq -r

Part II - Generate and Download the Assisted-Service ISO

  1. Create another json file, this time called iso-params.json which will be used to generate the deployment ISO:
cat << EOF > ./iso-params.json
{
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF
  1. Now use the following command to POST a request for Assisted-Service to build the deployment ISO:
curl -s -X POST "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \
  -d @iso-params.json \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  | jq '.'
  1. Use curl to download the ISO just generated. This ISO will be used to build the OpenShift cluster, as with any other Assisted-Service deployment:
curl \
  -H "Authorization: Bearer $TOKEN" \
  -L "http://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \
  -o ai-liveiso-$CLUSTER_ID.iso
  1. Lastly, boot the bare metal instance from this ISO just downloaded.

Part III - Prepare and Deploy the Cilium Manifests

  1. Download the manifests for Cilium v1.9.7, which is the latest OLM supported version (at time of writing)
GIT_BRANCH="master"
CILIUM_VERSION="v1.9.7"
curl -s https://codeload.github.com/cilium/cilium-olm/tar.gz/"${GIT_BRANCH}" | \
  tar -xz --strip=2 cilium-olm-"${GIT_BRANCH}"/manifests/cilium."${CILIUM_VERSION}"

NOTE: If tar doesn't work for the command above (and you're using MacOS), be sure to have coreutils installed and mapped properly. See this StackExchange post for more details.

  1. Create a CiliumConfig with the provided variables:
cat << EOF > ./cilium."${CILIUM_VERSION}"/cluster-network-07-cilium-ciliumconfig.yaml
---
apiVersion: cilium.io/v1alpha1
kind: CiliumConfig
metadata:
  name: cilium
  namespace: cilium
spec:
  ipam:
    mode: "cluster-pool"
    operator:
      clusterPoolIPv4PodCIDR: "$CLUSTER_CIDR_NET"
      clusterPoolIPv4MaskSize: "$CLUSTER_HOST_PFX"
  nativeRoutingCIDR: "$CLUSTER_CIDR_NET"
  cni:
    binPath: "/var/lib/cni/bin"
    confPath: "/var/run/multus/cni/net.d"
  prometheus:
    serviceMonitor: {enabled: false}
  hubble:
    tls: {enabled: false}
EOF

NOTE: If you get receive an error that file exists, then use setopt clobber before writing the file, and setopt noclobber afterwards.

  1. Next, we need to base64 encode each of the Cilium manifests, and upload each of them to the Assisted-Installer to be included in the deployment manifests directory. Before starting, verify if any manifests have been previously uploaded for this cluster. If brackets are returned, continue (this is what we want):
curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"
  1. Set variables for the Cilium version and where the manifests are located. It's been expected that you're running each of these commands from the same directory each time. If this is the case, you don't need to change anything below:
CILIUM_VERSION="v1.9.7"
MANIFESTS=(cilium."${CILIUM_VERSION}"/*.yaml)
  1. Run the following BASH/ZSH loop to POST each base64 encoded manifest to the Assisted-Service API automatically:
total=${#MANIFESTS[@]}
i=0
for file in "${MANIFESTS[@]}"; do
    i=$(( i + 1 ))
    eval "CILIUM_B64_MANIFEST=$(cat $file | base64 -w 0)";
    eval "BASEFILE=$(basename $file)";
    printf "Processing file: $file \n"
    printf "Basename of file: $BASEFILE \n"
    curl  \
    --header "Content-Type: application/json"  \
    --request POST  \
    -H "Authorization: Bearer $TOKEN" \
    --data "{\"file_name\":\"$BASEFILE\", \"folder\":\"manifests\", \"content\":\"$CILIUM_B64_MANIFEST\"}"  \
    "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"
done
printf "Total Manifests: $total \n"
  1. Lastly, verify that each of the custom manifests have been uploaded to the cluster. You should not see brackets this time:
curl -s -X GET \
  --header "Content-Type: application/json" \
  -H "Authorization: Bearer $TOKEN" \
  "https://$ASSISTED_SERVICE_API/api/assisted-install/v1/clusters/$CLUSTER_ID/manifests"

The final step is to navigate to the WebUI of the assisted-installer, select the cluster you've been customizing, click "Next" with NO modifications (this is very important), and click on "Install".

That's all you have to do!

Tags

Brandon B. Jozsa

Recommended for you