Pod stuck in Init: Mount.nfs: Protocol (Not supported)

Author:  Kellen Fields

Updated: June 2024

Audience: Everyone

Environmental details: Self-hosted, Replicated - KOTS, versions pre-v9.6.x 

Summary

A known Longhorn storage issue can cause the mount.nfs error on certain pods may make Jama Connect® unavailable after updating the Linux kernel.

Solution

1. When describing a pod that is stuck in a state other than running, you may see an output such as:

   Warning  FailedMount  21h (x21 over 25h)   kubelet  Unable to attach or mount volumes: unmounted volumes=[tenantfs], unattached volumes=[tenantfs core-trusted-certificates core-licenses-itext kube-api-access-vk78c volume data-restore]: timed out waiting for the condition
     Warning  FailedMount  21h (x19 over 24h)   kubelet  Unable to attach or mount volumes: unmounted volumes=[tenantfs], unattached volumes=[data-restore tenantfs core-trusted-certificates core-licenses-itext kube-api-access-vk78c volume]: timed out waiting for the condition
     Warning  FailedMount  20h (x23 over 25h)   kubelet  Unable to attach or mount volumes: unmounted volumes=[tenantfs], unattached volumes=[core-licenses-itext kube-api-access-vk78c volume data-restore tenantfs core-trusted-certificates]: timed out waiting for the condition
     Warning  FailedMount  20h (x142 over 25h)  kubelet  MountVolume.MountDevice failed for volume "pvc-d4d3c2f3-706b-42e3-a9f1-a5aaba6d2910" : rpc error: code = Internal desc = mount failed: exit status 32
   Mounting command: /usr/local/sbin/nsmounter
   Mounting arguments: mount -t nfs -o vers=4.1,noresvport,soft,intr,timeo=30,retrans=3 ***HIDDEN***:/pvc-d4d3c2f3-706b-42e3-a9f1-a5aaba6d2910 /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-d4d3c2f3-706b-42e3-a9f1-a5aaba6d2910/globalmount
   Output: mount.nfs: Protocol not supported

2. This is due to an issue with NFS in certain Linux Kernel versions. 
3. You can find your current kernel version with the command uname -a and when it was installed.
4. You will need to downgrade your kernel if it has been recently updated.

Cause

This is an issue with NFS and Longhorn storage, which is the storage component used by KOTS pre-9.6.x. More information can be found here:

Troubleshooting: RWX Volume Fails to Be Attached Caused by `Protocol not supported`

[BUG] MountVolume.MountDevice failed for volume Output: mount.nfs: Protocol not supported #6887

Prevention

You can upgrade to KOTS 9.6.x or consult Longhorn documentation before patching the kernel.