KOTS - Fix PVC failing to mount

James Ressler
James Ressler
  • Updated

Author: James Ressler

Date: December 15, 2023

Audience: Everyone

Environmental details: Self hosted

Summary

Sometimes pods will be blocked from starting if their associated PVC fails to mount. You can check by describing the affected pod.

kubectl describe pod <POD-NAME>

And look for an event like:

ControllerPublishVolume: err: rpc error: code = DeadlineExceeded desc = volume pvc-64a0b176-3982-4a75-83c9-7cc5857cb44b failed to attach to node

Solution

First, check the share manager pod in the 

kubectl get pods -n longhorn-system | grep "share-manager"

And check the status of the the share-manager pod. It should report 1/1 Running.

Scale down the statefulset associated with the failing pod and then force the share manager to recreate itself. After the share manager pod comes back up, scale the statefulset back up. In the example below, the core-0 pod is failing to mount its PVC so we scale down the core statefulset.

kubectl scale statefulset/core --replicas=0 -n default
kubectl delete pod <SHARE-MANAGER-POD-NAME> -n longhorn-system --grace-period=0 --force
kubectl scale statefulset/core --replicas=1 -n default

After the affected pod comes back up, you should see the PVC mount correctly this time.

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.