Quick Overview: LVM Storage Operator for Single-Node OpenShift (SNO)

Author: Brandon B. Jozsa

This article is part of a larger OpenShift Operators series, with the goal of providing quick information about installing using Red Hat's official operators.

Table of Contents

- Part I: Introduction
- Part II: Preparation
- Part III: Operator Installation
- Part IV: LVM Deployment
- Part V: StorageClass Enhancements (Optional)
- Final Thoughts

Part I: Introduction

It can be deceivingly easy to install and use OpenShift until you need something more advanced that requires additional file, block, or object storage, like a full-featured Network Observability Operator deployment (INSERT REFERENCE HERE).

Today we're going to be taking a look at running a more realistic storage solution as part of our SNO series, and the only requirement is to have a second disk; a pretty reasonable requirement.

Part II: Preparation

It's always wise to wipe any disk before reusing them, but what if you want to do this within RHCOS? Since RHCOS is considered an appliance, due in part to it's immutable nature, we'll use commands that preserve user privilege access and creates accurate accounting logs by using oc commands.

  1. As always, verify that you're connected to the correct SNO environment and review the disk topology. Let's use oc debug commands to achieve this task, rather than using ssh directy to the node (because it's inherently impossible ot know who the core user is, similar to logging in as root).

    Check to make sure you're communicating with the correct environment.

    ❯ oc get nodes
    NAME       STATUS   ROLES                                                        AGE   VERSION
    roderika   Ready    b200-m5-large-worker,control-plane,master,master-rt,worker   25d   v1.28.7+f1b5f6c
    

    Run the following commands to get information about your attached disks.

    NODE_NAME=$(oc get no -o name)
    
    echo $NODE_NAME
    
    cat <<EOF | oc debug $NODE_NAME
    chroot /host
    lsblk -o NAME,ROTA,SIZE,TYPE,WWN,SERIAL
    EOF
    

    The lsblk with output list (-o) will provide some useful information. For example, I have two disks in this system, and both have different OpenShift installations on them. My primary deployment is on /dev/sda, while a second disk also has an old OpenShift installation configured on /dev/sdb. In this case, let's wipe /dev/sdb so we can leverage this device/disk for our LVM deployment.

    sda                                                   1   931G disk
    |-sda1                                                1     1M part
    |-sda2                                                1   127M part
    |-sda3                                                1   384M part
    `-sda4                                                1 930.5G part
    sdb                                                   1   7.3T disk
    |-sdb1                                                1     1M part
    |-sdb2                                                1   127M part
    |-sdb3                                                1   384M part
    `-sdb4                                                1   7.3T part
    sr0                                                   1  1024M rom
    
  2. You can wipe the disks similarly with another direct oc debug command. PLEASE TAKE CARE to change the drive to match your environment, especially if you copy/paste these commands.

    cat <<EOF | oc debug $NODE_NAME
    chroot /host
    sudo wipefs -af /dev/sdb
    sudo sgdisk --zap-all /dev/sdb
    sudo dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct,dsync
    sudo blkdiscard /dev/sdb
    EOF
    

Part III: Operator Installation

Installation of the LVM Operator couldn't be any easier or straight forward either. Since we already wiped the disk in our previous section, let's get right to it.

  1. Please review the following manifest below. Note the three primary parts included in this manifest.

    • A Namespace object (or ns)
    • An OperatorGroup object (or og)
    • A Subscription object (or sub)

    You shouldn't need to change any of the options below, but there are two fields you may want to simply review in the Subscription deployment.

    • spec.installPlanApproval
    • spec.channel

    These are perfectly sane defaults across any OpenShift version, but if you want to target different versions or wish to manually upgrade the operator, you can review these CRD notes by running the following command (which is provided simply as an example): oc explain sub.spec.installPlanApproval

    For the sake of this demonstration, please keep things simple and deploy the following manifest "as is". The first is the operator specifically for LVM, and this is mandatory.

    oc apply -f - <<EOF
    ---
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        openshift.io/cluster-monitoring: "true"
        pod-security.kubernetes.io/enforce: privileged
        pod-security.kubernetes.io/audit: privileged
        pod-security.kubernetes.io/warn: privileged
      name: openshift-storage
    
    ---
    apiVersion: operators.coreos.com/v1
    kind: OperatorGroup
    metadata:
      name: openshift-storage-operatorgroup
      namespace: openshift-storage
    spec:
      targetNamespaces:
      - openshift-storage
    
    ---
    apiVersion: operators.coreos.com/v1alpha1
    kind: Subscription
    metadata:
      name: lvms
      namespace: openshift-storage
    spec:
      installPlanApproval: Automatic
      name: lvms-operator
      source: redhat-operators
      sourceNamespace: openshift-marketplace
    EOF
    
  2. Now you've deployed the LVM operator. It's really that simple! This is the general workflow for installing all operators in OpenShift. Now you can see a list of the operators within your environment, their general release, and what channel they're associated with.

    ❯ oc get sub -A
    NAMESPACE                          NAME                          PACKAGE                       SOURCE             CHANNEL
    openshift-cnv                      hco-operatorhub               kubevirt-hyperconverged       redhat-operators   stable
    openshift-nmstate                  kubernetes-nmstate-operator   kubernetes-nmstate-operator   redhat-operators   stable
    openshift-sriov-network-operator   sriov-network-operator        sriov-network-operator        redhat-operators   stable
    openshift-storage                  lvms                          lvms-operator                 redhat-operators
    

Part IV: LVM Deployment

With the operator installed, it's time to deploy the custom resource (CR). The CR is what triggers the operator to install or configure your deployment. So we obviously need to trigger a deployment using your custom disks and other configuration options.

  1. Carefully look at the following example CR. The LVM operator allows you to leverage one or more disks as part of an LVM deployment, but you need to know which disks you want to target. In addition to this, it's best to target a disk that is consistent. To do this, we are going to target a disk by it's PCIe path, rather than by name. This is due to the fact that disk names (i.e. /dev/sda, /dev/sdb etc) can actually change upon reboot. By targeting the disk by-path, we provide a consistent target for LVM.

    So let's explore the various attached disks using their PCIe address. To do this, run the following command:

    NODE_NAME=$(oc get no -o name)
    
    cat <<EOF | oc debug $NODE_NAME
    chroot /host
    ls -aslc /dev/disk/by-path/
    EOF
    

    The results should look something like the following:

    sh-5.1# ls -aslc /dev/disk/by-path/
    sh-5.1# exit
    total 0
    0 drwxr-xr-x. 2 root root 260 Sep  7 00:30 .
    0 drwxr-xr-x. 9 root root 180 Sep  7 00:30 ..
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:00:17.0-ata-8 -> ../../sr0
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:00:17.0-ata-8.0 -> ../../sr0
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:0:0 -> ../../sda
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:1:0 -> ../../sdc
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:2:0 -> ../../sdb
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:3:0 -> ../../sdd
    0 lrwxrwxrwx. 1 root root   9 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0 -> ../../sde
    0 lrwxrwxrwx. 1 root root  10 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part1 -> ../../sde1
    0 lrwxrwxrwx. 1 root root  10 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part2 -> ../../sde2
    0 lrwxrwxrwx. 1 root root  10 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part3 -> ../../sde3
    0 lrwxrwxrwx. 1 root root  10 Sep  7 00:30 pci-0000:1a:00.0-scsi-0:2:4:0-part4 -> ../../sde4
    
    Removing debug pod ...
    
  2. Using the output above, I'm going to use the following /dev/disk/by-path for my LVM configuration.

    • /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0
    • /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0
  3. Now I'm going to add this to my LVMCluster configuration.

    cat <<EOF | oc apply -f -
    apiVersion: lvm.topolvm.io/v1alpha1
    kind: LVMCluster
    metadata:
      name: lvmcluster
      namespace: openshift-storage
    spec:
      storage:
        deviceClasses:
          - deviceSelector:
              paths:
                - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:1:0 # < -- /dev/sdc
                - /dev/disk/by-path/pci-0000:1a:00.0-scsi-0:2:3:0 # < -- /dev/sdd
            name: vg1
            thinPoolConfig:
              name: thin-pool-1
              overprovisionRatio: 10
              sizePercent: 90
    EOF
    

    You can take this a step further by adding a label to your node, and limiting your LVM deployment to the targeted disks only when the correct label is applied to the node. This can be done like the following example.

    NODE_NAME=$(oc get no -o name)
    
    oc label $NODE_NAME topology.kubernetes.io/lvm-disk=sdb
    
    oc apply -f - <<EOF
    apiVersion: lvm.topolvm.io/v1alpha1
    kind: LVMCluster
    metadata:
      name: lvmcluster
      namespace: openshift-storage
    spec:
      storage:
        deviceClasses:
          - deviceSelector:
              paths:
                - '/dev/disk/by-path/pci-0000:03:00.0-scsi-0:2:1:0'
            fstype: xfs
            name: vg1
            nodeSelector:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: topology.kubernetes.io/lvm-disk
                      operator: In
                      values:
                        - sdb
            thinPoolConfig:
              chunkSizeCalculationPolicy: Static
              name: thin-pool-1
              overprovisionRatio: 10
              sizePercent: 90
    EOF
    
  4. With the LVMCluster CR deployed, your LVM disk should be configured and progressing. Soon a new StorageClass named lvms-vg1 will be available for your deployments.

    ❯ oc get sc
    NAME                           PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    lvms-vg1                       topolvm.io                        Delete          WaitForFirstConsumer   true                   5d4h
    

Part V: StorageClass Enhancements (Optional)

One thing you may have noticed about your new StorageClass is that lvms-vg1 is configured for WaitForConsumer. For deployments to work correctly, either a storageclass will need to be defined manually, or you will need to manually create a PersistentVolume that can bing to a given workload (or consumer). But if this sounds a little tedious, there's a work-around that I think many people will like/prefer, although it can be situational.

  1. We're going to create a new StorageClass that lives side-by-side with the default lvms-vg1 one created by the LVMS operator. We'll call this lvms-vg1-immediate, because we're going to allow workloads to automatically bind to corresponding PVs. In addition, we're going to make this new lvms-vg1-immediate StorageClass our default storageclass with the correct annotations.

    cat <<EOF | oc apply -f -
    kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: lvms-vg1-immediate
      annotations:
        description: Provides RWO and RWOP Filesystem & Block volumes
        storageclass.kubernetes.io/is-default-class: 'true'
        storageclass.kubevirt.io/is-default-virt-class: 'true'
    provisioner: topolvm.io
    parameters:
      csi.storage.k8s.io/fstype: xfs
      topolvm.io/device-class: vg1
    reclaimPolicy: Delete
    allowVolumeExpansion: true
    volumeBindingMode: Immediate
    EOF
    

Final Thoughts

This is a really simple blog post; perhaps more simple and direct than some of my previous posts. However, the LVM Operator is used so much in my deployments as of late, that I've pretty much been using it all the time. As such, I figured it would be a great base for my other blog posts, so I don't have to repeat this process for every other blog post going forward.

There's a much further condenced version of this blog post on Gist that you are free to use as well.

Thanks for reading my post! Hopefully this post can help you or someone you know when working with OpenShift.