Expanding the size of the tenantfs in your existing installation.

Sravya Bandari
Sravya Bandari
  • Updated

Author: Sravya Bandari

Date: October 12, 2023

Audience: Everyone

Environmental details: For KOTS customers with on-premises setups.

Problem/Summary:

If the customer faces issues when attempting to resize the TenantFS (PVC) in the Jama application. Despite updating the 'Asset Size' value through the Admin UI and redeploying, the PVC capacity remains unchanged, and the error message "Ignoring the PVC: didn't find a plugin capable of expanding the volume" is observed.

kubectl describe pvc/tenantfs

Solution:

Follow these steps to resolve the TenantFS PVC resize issue:

  • Ensure PVC is Not in Use: The filesystem within the PVC cannot be resized while in use by any pods. To resolve this, scale down the core deployment to 0 pods to ensure that nothing actively uses the tenantfs volume.

    kubectl scale statefulset core --replicas=0

  • Adjust PVC Size: After scaling down, make the necessary changes to the PVC size through the Admin UI. Ensure that the Longhorn storage provisioner can recognize and expand the volume.
  • Verify PVC Size: Confirm that the PVC size has been updated by checking its details.

    kubectl describe pvc tenantfs
  • Scale Up Core Deployment: Once the PVC size is confirmed to be updated, scale up the core deployment to its original state.

    kubectl scale statefulset core --replicas=1
  • NOTE:
    To identify which pod is currently using the PVC named "tenantfs," follow these steps:
    • List all PVCs, look for the PVC named "tenantfs" and note its status and the volume it is bound to.
      # kubectl get pvc

      NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE

      kotsadm-postgres-kotsadm-postgres-0   Bound    pvc-191b93cd-9aeb-455b-9e36-1d96f43a08b7   1Gi        RWO            longhorn       16d

      tenantfs                              Bound    pvc-721b1494-e065-46e4-a499-f70ad3de2a53   7Gi        RWX            longhorn       16d

      volume-activemq-0                     Bound    pvc-95e6f75f-c662-49c6-9a87-61639cb8ed70   5Gi        RWO            longhorn       16d

      volume-core-0                         Bound    pvc-e9c4465c-d39b-476c-922e-97452a796470   3Gi        RWO            longhorn       16d

      volume-diff-0                         Bound    pvc-c1d21ffd-9a87-4153-b339-be54cc7ecf12   2Gi        RWO            longhorn       16d

      volume-elasticsearch-0                Bound    pvc-7ecfd6df-889f-4abf-b23b-8607c383ec5f   10Gi       RWO            longhorn       16d

      volume-hazelcast-0                    Bound    pvc-50bfb0af-22c5-47c6-a37f-cb15560af456   2Gi        RWO            longhorn       16d

      volume-nginx-0                        Bound    pvc-cfc6e0f0-3e10-4ee6-9a9b-a0eb255f7c6d   2Gi        RWO            longhorn       16d

      volume-oauth-0                        Bound    pvc-8a38f409-ddf5-40f3-b674-c713019beba1   2Gi        RWO            longhorn       16d

      volume-saml-0                         Bound    pvc-a7001f52-f97f-4c73-9b6e-5bf3aec496bb   2Gi        RWO            longhorn       16d

      volume-search-0                       Bound    pvc-b4266399-2086-4647-8564-6e20e3467185   2Gi        RWO            longhorn       16d
    • Describe the "tenantfs" PVC to find out which pod is using it:
      # kubectl describe pvc tenantfs

      Name:          tenantfs

      Namespace:     default

      StorageClass:  longhorn

      Status:        Bound

      Volume:        pvc-721b1494-e065-46e4-a499-f70ad3de2a53

      Labels:        app.kubernetes.io/instance=application

                     app.kubernetes.io/managed-by=Helm

                     app.kubernetes.io/name=core

                     app.kubernetes.io/version=8.25.3

                     helm.sh/chart=core-0.0.0

                     kots.io/app-slug=jama-k8s

                     kots.io/backup=velero

      Annotations:   kots.io/app-slug: jama-k8s

                     meta.helm.sh/release-name: application

                     meta.helm.sh/release-namespace: default

                     pv.kubernetes.io/bind-completed: yes

                     pv.kubernetes.io/bound-by-controller: yes

                     volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io

                     volume.kubernetes.io/storage-provisioner: driver.longhorn.io

      Finalizers:    [kubernetes.io/pvc-protection]

      Capacity:      7Gi

      Access Modes:  RWX

      VolumeMode:    Filesystem

      Used By:       core-0

      Events:        <none>
      in the describe details of the PVC, there's a Used By: field that describes which pods are currently using the tenantfs volume

      So, the PVC named "tenantfs" is currently being used by the pod named "core-0." If you need to resize this volume, you may need to scale down the "core" deployment to 0 replicas, resize the PVC, and then scale the deployment back up.

Cause:

Resizing storage in Kubernetes using Longhorn can be tricky if the PVC is still in use. To fix this, scaling down the core deployment to zero pods is recommended, ensuring the tenantfs PVC is not actively being used. It's important to resize the PVC when no pods use it, as Longhorn can't adjust the filesystem size when it's in use. 

 

Additional Documentation and Resources for Support Engineers:

  • Request #72370 and corresponding Replicated Issue #257

Feedback: Did this article help you resolve expanding tenantfs PVC size? Please leave feedback in the comments below.

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.