Remember to expand the size of the tenant file system in your existing installation

Author: Sravya Bandari

Updated: June 2024

Audience: Everyone

Environmental: Self-hosted, Replicated - KOTS 

Summary

Suppose you encounter problems while resizing the TenantFS (PVC) in the Jama application. In that case, you may find that the PVC capacity remains unchanged despite updating the 'Asset Size' value through the Admin UI and redeploying. Additionally, you may observe the error message "Ignoring the PVC: didn't find a plugin capable of expanding the volume."

kubectl describe pvc/tenantfs

Solution

Follow these steps to resolve the TenantFS PVC resize issue:

• Ensure PVC is Not in Use: The filesystem within the PVC cannot be resized while in use by any pods. To resolve this, scale the core deployment to 0 pods to ensure that nothing actively uses the tenantfs volume.
  
  kubectl scale statefulset core --replicas=0
  
  
• Adjust PVC Size: After scaling down, make the necessary changes to the PVC size through the Admin UI. Ensure that the Longhorn storage provisioner can recognize and expand the volume.
  
• Verify PVC Size: Confirm that the PVC size has been updated by checking its details.
  
  kubectl describe pvc tenantfs

• Scale Up Core Deployment: Once the PVC size is confirmed to be updated, scale up the core deployment to its original state.
  
  kubectl scale statefulset core --replicas=1

NOTE:
To identify which pod is currently using the PVC named "tenantfs," follow these steps:

• • List all PVCs, look for the PVC named "tenantfs," and note its status and the volume it is bound to.
    

    # kubectl get pvc
    
    NAME                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    
    kotsadm-postgres-kotsadm-postgres-0   Bound    pvc-191b93cd-9aeb-455b-9e36-1d96f43a08b7   1Gi        RWO            longhorn       16d
    
    tenantfs                              Bound    pvc-721b1494-e065-46e4-a499-f70ad3de2a53   7Gi        RWX            longhorn       16d
    
    volume-activemq-0                     Bound    pvc-95e6f75f-c662-49c6-9a87-61639cb8ed70   5Gi        RWO            longhorn       16d
    
    volume-core-0                         Bound    pvc-e9c4465c-d39b-476c-922e-97452a796470   3Gi        RWO            longhorn       16d
    
    volume-diff-0                         Bound    pvc-c1d21ffd-9a87-4153-b339-be54cc7ecf12   2Gi        RWO            longhorn       16d
    
    volume-elasticsearch-0                Bound    pvc-7ecfd6df-889f-4abf-b23b-8607c383ec5f   10Gi       RWO            longhorn       16d
    
    volume-hazelcast-0                    Bound    pvc-50bfb0af-22c5-47c6-a37f-cb15560af456   2Gi        RWO            longhorn       16d
    
    volume-nginx-0                        Bound    pvc-cfc6e0f0-3e10-4ee6-9a9b-a0eb255f7c6d   2Gi        RWO            longhorn       16d
    
    volume-oauth-0                        Bound    pvc-8a38f409-ddf5-40f3-b674-c713019beba1   2Gi        RWO            longhorn       16d
    
    volume-saml-0                         Bound    pvc-a7001f52-f97f-4c73-9b6e-5bf3aec496bb   2Gi        RWO            longhorn       16d
    
    volume-search-0                       Bound    pvc-b4266399-2086-4647-8564-6e20e3467185   2Gi        RWO            longhorn       16d
    

  • Describe the "tenantfs" PVC to find out which pod is using it:
    

    # kubectl describe pvc tenantfs
    
    Name:          tenantfs
    
    Namespace:     default
    
    StorageClass:  longhorn
    
    Status:        Bound
    
    Volume:        pvc-721b1494-e065-46e4-a499-f70ad3de2a53
    
    Labels:        app.kubernetes.io/instance=application
    
                   app.kubernetes.io/managed-by=Helm
    
                   app.kubernetes.io/name=core
    
                   app.kubernetes.io/version=8.25.3
    
                   helm.sh/chart=core-0.0.0
    
                   kots.io/app-slug=jama-k8s
    
                   kots.io/backup=velero
    
    Annotations:   kots.io/app-slug: jama-k8s
    
                   meta.helm.sh/release-name: application
    
                   meta.helm.sh/release-namespace: default
    
                   pv.kubernetes.io/bind-completed: yes
    
                   pv.kubernetes.io/bound-by-controller: yes
    
                   volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
    
                   volume.kubernetes.io/storage-provisioner: driver.longhorn.io
    
    Finalizers:    [kubernetes.io/pvc-protection]
    
    Capacity:      7Gi
    
    Access Modes:  RWX
    
    VolumeMode:    Filesystem
    
    Used By:       core-0
    
    Events:        <none>

    in the describe details of the PVC, there's a Used By: field that describes which pods are currently using the tenantfs volume
    

    So, the PVC named "tenantfs" is currently being used by the pod named "core-0." If you need to resize this volume, you may need to scale down the "core" deployment to 0 replicas, resize the PVC, and then scale the deployment back up.

Cause

Resizing storage in Kubernetes using Longhorn can be tricky if the PVC is still in use. To fix this, scaling down the core deployment to zero pods is recommended, ensuring the tenantfs PVC is not actively being used. It's important to resize the PVC when no pods are used, as Longhorn can't adjust the filesystem size when it's in use. 

Additional Documentation and Resources for Support Engineers

• Request #72370 and corresponding Replicated Issue #257

Feedback: Please leave feedback in the comments below.