LVM Storage Operator for Single-Node OpenShift (SNO)
Author: Brandon B. Jozsa
"It is not the beauty of a building you should look at; its the construction of the foundation that will stand the test of time."
- David Allen Coe
- Part I: Introduction
- Part II: Preparation
- Part III: Operator Installation
- Part IV: LVM Deployment
- Part V: StorageClass Enhancements (Optional)
- Final Thoughts
Part I: Introduction
It can be deceivingly easy to install and use OpenShift until you need something more advanced that requires additional file, block, or object storage, like a full-featured Network Observability Operator deployment (INSERT REFERENCE HERE).
Today we're going to be taking a look at running a more realistic storage solution as part of our SNO series, and the only requirement is to have a second disk; a pretty reasonable requirement.
Part II: Preparation
It's always wise to wipe any disk before reusing them, but what if you want to do this within RHCOS? Since RHCOS is considered an appliance, due in part to it's immutable nature, we'll use commands that preserve user privilege access and creates accurate accounting logs by using oc commands.
-
As always, verify that you're connected to the correct SNO environment and review the disk topology. Let's use
oc debugcommands to achieve this task, rather than usingsshdirecty to the node (because it's inherently impossible ot know who thecoreuser is, similar to logging in asroot).Check to make sure you're communicating with the correct environment.
❯ oc get nodes NAME STATUS ROLES AGE VERSION roderika Ready b200-m5-large-worker,control-plane,master,master-rt,worker 25d v1.28.7+f1b5f6cRun the following commands to get information about your attached disks.
NODE_NAME=$(oc get no -o name) echo $NODE_NAME cat <<EOF | oc debug $NODE_NAME chroot /host lsblk -o NAME,ROTA,SIZE,TYPE,WWN,SERIAL EOFThe
lsblkwith output list (-o) will provide some useful information. For example, I have two disks in this system, and both have different OpenShift installations on them. My primary deployment is on/dev/sda, while a second disk also has an old OpenShift installation configured on/dev/sdb. In this case, let's wipe/dev/sdbso we can leverage this device/disk for our LVM deployment.sda 1 931G disk |-sda1 1 1M part |-sda2 1 127M part |-sda3 1 384M part `-sda4 1 930.5G part sdb 1 7.3T disk |-sdb1 1 1M part |-sdb2 1 127M part |-sdb3 1 384M part `-sdb4 1 7.3T part sr0 1 1024M rom -
You can wipe the disks similarly with another direct
oc debugcommand. PLEASE TAKE CARE to change the drive to match your environment, especially if you copy/paste these commands.cat <<EOF | oc debug $NODE_NAME chroot /host sudo wipefs -af /dev/sdb sudo sgdisk --zap-all /dev/sdb sudo dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync sudo blkdiscard /dev/sdb EOF
Part III: Operator Installation
Installation of the LVM Operator couldn't be any easier or straight forward either. Since we already wiped the disk in our previous section, let's get right to it.
-
Please review the following manifest below. Note the three primary parts included in this manifest.
- A
Namespaceobject (orns) - An
OperatorGroupobject (orog) - A
Subscriptionobject (orsub)
You shouldn't need to change any of the options below, but there are two fields you may want to simply review in the
Subscriptiondeployment.spec.installPlanApprovalspec.channel
These are perfectly sane defaults across any OpenShift version, but if you want to target different versions or wish to manually upgrade the operator, you can review these CRD notes by running the following command (which is provided simply as an example):
oc explain sub.spec.installPlanApprovalFor the sake of this demonstration, please keep things simple and deploy the following manifest "as is". The first is the operator specifically for LVM, and this is mandatory.
oc apply -f - <<EOF --- apiVersion: v1 kind: Namespace metadata: labels: openshift.io/cluster-monitoring: "true" pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged name: openshift-storage --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-storage-operatorgroup namespace: openshift-storage spec: targetNamespaces: - openshift-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: lvms namespace: openshift-storage spec: installPlanApproval: Automatic name: lvms-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF - A
-
OPTIONAL: While this next deployment is optional (below), I recommend installing the Local Storage Operator (or, LSO) as well. This can be useful if you ever wish to partition drives on a system, and use those partitions for other things. This step isn't required for LVM, so if you don't need it or don't know if you need it you are safe to skip this step.
cat <<EOF | oc apply -f - --- apiVersion: v1 kind: Namespace metadata: name: openshift-local-storage --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: local-operator-group namespace: openshift-local-storage spec: targetNamespaces: - openshift-local-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: local-storage-operator namespace: openshift-local-storage spec: channel: stable installPlanApproval: Automatic name: local-storage-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF -
Now you've deployed the LVM operator. It's really that simple! This is the general workflow for installing all operators in OpenShift. Now you can see a list of the operators within your environment, their general release, and what channel they're associated with.
❯ oc get sub -A NAMESPACE NAME PACKAGE SOURCE CHANNEL openshift-cnv hco-operatorhub kubevirt-hyperconverged redhat-operators stable openshift-nmstate kubernetes-nmstate-operator kubernetes-nmstate-operator redhat-operators stable openshift-sriov-network-operator sriov-network-operator sriov-network-operator redhat-operators stable openshift-storage lvms lvms-operator redhat-operators
Part IV: LVM Deployment
With the operator installed, it's time to deploy the custom resource (CR). The CR is what triggers the operator to install or configure your deployment. So we obviously need to trigger a deployment using your custom disks and other configuration options.
-
Carefully look at the following example CR. The LVM operator allows you to leverage one or more disks as part of an LVM deployment, but you need to know which disks you want to target. In addition to this, it's best to target a disk that is consistent. To do this, we are going to target a disk by it's PCIe path, rather than by name. This is due to the fact that disk names (i.e.
/dev/sda,/dev/sdbetc) can actually change upon reboot. By targeting the diskby-path, we provide a consistent target for LVM.So let's explore the various attached disks using their PCIe address. To do this, run the following command:
NODE_NAME=$(oc get no -o name) cat <<EOF | oc debug $NODE_NAME chroot /host ls -aslc /dev/disk/by-path/ EOFThe results should look something like the following:
sh-5.1# ls -aslc /dev/disk/by-path/ sh-5.1# exit total 0 0 drwxr-xr-x. 2 root root 260 Sep 7 00:30 . 0 drwxr-xr-x. 9 root root 180 Sep 7 00:30 .. 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8.0 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:0:0 -> ../../sda 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:1:0 -> ../../sdc 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:2:0 -> ../../sdb 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:3:0 -> ../../sdd 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0 -> ../../sde 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part1 -> ../../sde1 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part2 -> ../../sde2 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part3 -> ../../sde3 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part4 -> ../../sde4 Removing debug pod ... -
Using the output above, I'm going to use the following
/dev/disk/by-pathfor my LVM configuration./dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0/dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0
-
Now I'm going to add this to my
LVMClusterconfiguration.cat <<EOF | oc apply -f - apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMCluster metadata: name: lvmcluster namespace: openshift-storage spec: storage: deviceClasses: - deviceSelector: paths: - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0 # < -- /dev/sdc - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0 # < -- /dev/sdd name: vg1 thinPoolConfig: name: thin-pool-1 overprovisionRatio: 10 sizePercent: 90 EOFYou can take this a step further by adding a label to your node, and limiting your LVM deployment to the targeted disks only when the correct label is applied to the node. This can be done like the following example.
NODE_NAME=$(oc get no -o name) oc label $NODE_NAME topology.kubernetes.io/lvm-disk=sdb oc apply -f - <<EOF apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMCluster metadata: name: lvmcluster namespace: openshift-storage spec: storage: deviceClasses: - deviceSelector: paths: - '/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0' fstype: xfs name: vg1 nodeSelector: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/lvm-disk operator: In values: - sdb thinPoolConfig: chunkSizeCalculationPolicy: Static name: thin-pool-1 overprovisionRatio: 10 sizePercent: 90 EOF -
With the
LVMClusterCR deployed, your LVM disk should be configured and progressing. Soon a newStorageClassnamedlvms-vg1will be available for your deployments.❯ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE lvms-vg1 topolvm.io Delete WaitForFirstConsumer true 5d4h
Part V: StorageClass Enhancements (Optional)
One thing you may have noticed about your new StorageClass is that lvms-vg1 is configured for WaitForConsumer. For deployments to work correctly, either a storageclass will need to be defined manually, or you will need to manually create a PersistentVolume that can bing to a given workload (or consumer). But if this sounds a little tedious, there's a work-around that I think many people will like/prefer, although it can be situational.
-
We're going to create a new
StorageClassthat lives side-by-side with the defaultlvms-vg1one created by the LVMS operator. We'll call thislvms-vg1-immediate, because we're going to allow workloads to automatically bind to corresponding PVs. In addition, we're going to make this newlvms-vg1-immediateStorageClassour default storageclass with the correct annotations.cat <<EOF | oc apply -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: lvms-vg1-immediate annotations: description: Provides RWO and RWOP Filesystem & Block volumes storageclass.kubernetes.io/is-default-class: 'true' storageclass.kubevirt.io/is-default-virt-class: 'true' provisioner: topolvm.io parameters: csi.storage.k8s.io/fstype: xfs topolvm.io/device-class: vg1 reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate EOF
Final Thoughts
This is a really simple blog post; perhaps more simple and direct than some of my previous posts. However, the LVM Operator is used so much in my deployments as of late, that I've pretty much been using it all the time. As such, I figured it would be a great base for my other blog posts, so I don't have to repeat this process for every other blog post going forward.
There's a much further condenced version of this blog post on Gist that you are free to use as well.
Thanks for reading my post! Hopefully this post can help you or someone you know when working with OpenShift.
