LVM Storage Operator for Single-Node OpenShift (SNO)
Author: Brandon B. Jozsa
"It is not the beauty of a building you should look at; its the construction of the foundation that will stand the test of time."
- David Allen Coe
- Part I: Introduction
- Part II: Preparation
- Part III: Operator Installation
- Part IV: LVM Deployment
- Part V: StorageClass Enhancements (Optional)
- Final Thoughts
Part I: Introduction
It can be deceivingly easy to install and use OpenShift until you need something more advanced that requires additional file, block, or object storage, like a full-featured Network Observability Operator deployment (INSERT REFERENCE HERE).
Today we're going to be taking a look at running a more realistic storage solution as part of our SNO series, and the only requirement is to have a second disk; a pretty reasonable requirement.
Part II: Preparation
It's always wise to wipe any disk before reusing them, but what if you want to do this within RHCOS? Since RHCOS is considered an appliance, due in part to it's immutable nature, we'll use commands that preserve user privilege access and creates accurate accounting logs by using oc
commands.
-
As always, verify that you're connected to the correct SNO environment and review the disk topology. Let's use
oc debug
commands to achieve this task, rather than usingssh
directy to the node (because it's inherently impossible ot know who thecore
user is, similar to logging in asroot
).Check to make sure you're communicating with the correct environment.
❯ oc get nodes NAME STATUS ROLES AGE VERSION roderika Ready b200-m5-large-worker,control-plane,master,master-rt,worker 25d v1.28.7+f1b5f6c
Run the following commands to get information about your attached disks.
NODE_NAME=$(oc get no -o name) echo $NODE_NAME cat <<EOF | oc debug $NODE_NAME chroot /host lsblk -o NAME,ROTA,SIZE,TYPE,WWN,SERIAL EOF
The
lsblk
with output list (-o
) will provide some useful information. For example, I have two disks in this system, and both have different OpenShift installations on them. My primary deployment is on/dev/sda
, while a second disk also has an old OpenShift installation configured on/dev/sdb
. In this case, let's wipe/dev/sdb
so we can leverage this device/disk for our LVM deployment.sda 1 931G disk |-sda1 1 1M part |-sda2 1 127M part |-sda3 1 384M part `-sda4 1 930.5G part sdb 1 7.3T disk |-sdb1 1 1M part |-sdb2 1 127M part |-sdb3 1 384M part `-sdb4 1 7.3T part sr0 1 1024M rom
-
You can wipe the disks similarly with another direct
oc debug
command. PLEASE TAKE CARE to change the drive to match your environment, especially if you copy/paste these commands.cat <<EOF | oc debug $NODE_NAME chroot /host sudo wipefs -af /dev/sdb sudo sgdisk --zap-all /dev/sdb sudo dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync sudo blkdiscard /dev/sdb EOF
Part III: Operator Installation
Installation of the LVM Operator couldn't be any easier or straight forward either. Since we already wiped the disk in our previous section, let's get right to it.
-
Please review the following manifest below. Note the three primary parts included in this manifest.
- A
Namespace
object (orns
) - An
OperatorGroup
object (orog
) - A
Subscription
object (orsub
)
You shouldn't need to change any of the options below, but there are two fields you may want to simply review in the
Subscription
deployment.spec.installPlanApproval
spec.channel
These are perfectly sane defaults across any OpenShift version, but if you want to target different versions or wish to manually upgrade the operator, you can review these CRD notes by running the following command (which is provided simply as an example):
oc explain sub.spec.installPlanApproval
For the sake of this demonstration, please keep things simple and deploy the following manifest "as is". The first is the operator specifically for LVM, and this is mandatory.
oc apply -f - <<EOF --- apiVersion: v1 kind: Namespace metadata: labels: openshift.io/cluster-monitoring: "true" pod-security.kubernetes.io/enforce: privileged pod-security.kubernetes.io/audit: privileged pod-security.kubernetes.io/warn: privileged name: openshift-storage --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: openshift-storage-operatorgroup namespace: openshift-storage spec: targetNamespaces: - openshift-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: lvms namespace: openshift-storage spec: installPlanApproval: Automatic name: lvms-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
- A
-
OPTIONAL: While this next deployment is optional (below), I recommend installing the Local Storage Operator (or, LSO) as well. This can be useful if you ever wish to partition drives on a system, and use those partitions for other things. This step isn't required for LVM, so if you don't need it or don't know if you need it you are safe to skip this step.
cat <<EOF | oc apply -f - --- apiVersion: v1 kind: Namespace metadata: name: openshift-local-storage --- apiVersion: operators.coreos.com/v1 kind: OperatorGroup metadata: name: local-operator-group namespace: openshift-local-storage spec: targetNamespaces: - openshift-local-storage --- apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: local-storage-operator namespace: openshift-local-storage spec: channel: stable installPlanApproval: Automatic name: local-storage-operator source: redhat-operators sourceNamespace: openshift-marketplace EOF
-
Now you've deployed the LVM operator. It's really that simple! This is the general workflow for installing all operators in OpenShift. Now you can see a list of the operators within your environment, their general release, and what channel they're associated with.
❯ oc get sub -A NAMESPACE NAME PACKAGE SOURCE CHANNEL openshift-cnv hco-operatorhub kubevirt-hyperconverged redhat-operators stable openshift-nmstate kubernetes-nmstate-operator kubernetes-nmstate-operator redhat-operators stable openshift-sriov-network-operator sriov-network-operator sriov-network-operator redhat-operators stable openshift-storage lvms lvms-operator redhat-operators
Part IV: LVM Deployment
With the operator installed, it's time to deploy the custom resource (CR). The CR is what triggers the operator to install or configure your deployment. So we obviously need to trigger a deployment using your custom disks and other configuration options.
-
Carefully look at the following example CR. The LVM operator allows you to leverage one or more disks as part of an LVM deployment, but you need to know which disks you want to target. In addition to this, it's best to target a disk that is consistent. To do this, we are going to target a disk by it's PCIe path, rather than by name. This is due to the fact that disk names (i.e.
/dev/sda
,/dev/sdb
etc) can actually change upon reboot. By targeting the diskby-path
, we provide a consistent target for LVM.So let's explore the various attached disks using their PCIe address. To do this, run the following command:
NODE_NAME=$(oc get no -o name) cat <<EOF | oc debug $NODE_NAME chroot /host ls -aslc /dev/disk/by-path/ EOF
The results should look something like the following:
sh-5.1# ls -aslc /dev/disk/by-path/ sh-5.1# exit total 0 0 drwxr-xr-x. 2 root root 260 Sep 7 00:30 . 0 drwxr-xr-x. 9 root root 180 Sep 7 00:30 .. 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:00:17.0-ata-8.0 -> ../../sr0 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:0:0 -> ../../sda 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:1:0 -> ../../sdc 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:2:0 -> ../../sdb 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:3:0 -> ../../sdd 0 lrwxrwxrwx. 1 root root 9 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0 -> ../../sde 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part1 -> ../../sde1 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part2 -> ../../sde2 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part3 -> ../../sde3 0 lrwxrwxrwx. 1 root root 10 Sep 7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part4 -> ../../sde4 Removing debug pod ...
-
Using the output above, I'm going to use the following
/dev/disk/by-path
for my LVM configuration./dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0
/dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0
-
Now I'm going to add this to my
LVMCluster
configuration.cat <<EOF | oc apply -f - apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMCluster metadata: name: lvmcluster namespace: openshift-storage spec: storage: deviceClasses: - deviceSelector: paths: - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0 # < -- /dev/sdc - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0 # < -- /dev/sdd name: vg1 thinPoolConfig: name: thin-pool-1 overprovisionRatio: 10 sizePercent: 90 EOF
You can take this a step further by adding a label to your node, and limiting your LVM deployment to the targeted disks only when the correct label is applied to the node. This can be done like the following example.
NODE_NAME=$(oc get no -o name) oc label $NODE_NAME topology.kubernetes.io/lvm-disk=sdb oc apply -f - <<EOF apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMCluster metadata: name: lvmcluster namespace: openshift-storage spec: storage: deviceClasses: - deviceSelector: paths: - '/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0' fstype: xfs name: vg1 nodeSelector: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/lvm-disk operator: In values: - sdb thinPoolConfig: chunkSizeCalculationPolicy: Static name: thin-pool-1 overprovisionRatio: 10 sizePercent: 90 EOF
-
With the
LVMCluster
CR deployed, your LVM disk should be configured and progressing. Soon a newStorageClass
namedlvms-vg1
will be available for your deployments.❯ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE lvms-vg1 topolvm.io Delete WaitForFirstConsumer true 5d4h
Part V: StorageClass Enhancements (Optional)
One thing you may have noticed about your new StorageClass
is that lvms-vg1
is configured for WaitForConsumer
. For deployments to work correctly, either a storageclass will need to be defined manually, or you will need to manually create a PersistentVolume
that can bing to a given workload (or consumer). But if this sounds a little tedious, there's a work-around that I think many people will like/prefer, although it can be situational.
-
We're going to create a new
StorageClass
that lives side-by-side with the defaultlvms-vg1
one created by the LVMS operator. We'll call thislvms-vg1-immediate
, because we're going to allow workloads to automatically bind to corresponding PVs. In addition, we're going to make this newlvms-vg1-immediate
StorageClass
our default storageclass with the correct annotations.cat <<EOF | oc apply -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: lvms-vg1-immediate annotations: description: Provides RWO and RWOP Filesystem & Block volumes storageclass.kubernetes.io/is-default-class: 'true' storageclass.kubevirt.io/is-default-virt-class: 'true' provisioner: topolvm.io parameters: csi.storage.k8s.io/fstype: xfs topolvm.io/device-class: vg1 reclaimPolicy: Delete allowVolumeExpansion: true volumeBindingMode: Immediate EOF
Final Thoughts
This is a really simple blog post; perhaps more simple and direct than some of my previous posts. However, the LVM Operator is used so much in my deployments as of late, that I've pretty much been using it all the time. As such, I figured it would be a great base for my other blog posts, so I don't have to repeat this process for every other blog post going forward.
There's a much further condenced version of this blog post on Gist that you are free to use as well.
Thanks for reading my post! Hopefully this post can help you or someone you know when working with OpenShift.