PVC failing to mount - how to fix

Author: James Ressler

Date: December 15, 2023

Audience: Everyone

Environment: Self-hosted, Replicated - KOTS

Summary

Sometimes, pods are blocked from starting if their associated PVC fails to mount. To check, describe the affected pod.

kubectl describe pod <POD-NAME>

And look for an event like:

ControllerPublishVolume: err: rpc error: code = DeadlineExceeded desc = volume pvc-64a0b176-3982-4a75-83c9-7cc5857cb44b failed to attach to node

Solution

First, check the share manager pod in the 

kubectl get pods -n longhorn-system | grep "share-manager"

Check the status of the share-manager pod. It should report 1/1 Running.

Scale down the statefulset associated with the failing pod and then force the share manager to recreate itself. After the share manager pod returns, scale the statefulset back up. In the example below, the core-0 pod fails to mount its PVC, so we scale down the core statefulset.

kubectl scale statefulset/core --replicas=0 -n default
kubectl delete pod <SHARE-MANAGER-POD-NAME> -n longhorn-system --grace-period=0 --force
kubectl scale statefulset/core --replicas=1 -n default

After the affected pod comes back up, you should see the PVC mount correctly this time.