Pods Not Starting After Ubuntu Kernel Update (kubectl version mismatch)

Erik Haake
Erik Haake
  • Updated

Author: Erik Haake
Date: September 10, 2025
Audience: Everyone
Environmental Details: Self-hosted, Replicated - KOTS

  • Jama Connect® version(s): 9.6.x and above
  • OS: Ubuntu 20

Summary 

This article discusses issues with pods not starting after updating the Ubuntu kernel in a Jama Connect® self-hosted environment. The primary symptoms include pods stuck in ContainerCreating and errors related to FailedCreatePodSandBox. The reason was that kubectl was upgraded after running an apt upgrade.

Solution 

  • Downgrade kubectl to match the server version. Run kubectl version to determine the current server version.
  • Put specific packages on hold to prevent automatic upgrades that could disrupt the Kubernetes cluster. The packages to put on hold are listed here: https://kurl.sh/jama-k8s-standardkots
  • Monitor the system after downgrading to ensure pods start correctly.

Cause 

The issue arose from upgrading Jama dependent packages during the Ubuntu kernel update, which led to compatibility problems within the Kubernetes cluster. The version mismatch between the kubectl client and server was the main culprit.

Why the mismatch caused problems

kubectl is the client binary used to talk to the Kubernetes API server that runs in your cluster. Each version of Kubernetes has a defined API, and the kubectl client is built to match it.

  • Kubernetes only guarantees compatibility between a kubectl client and API server if they are within one minor version of each other (e.g. kubectl 1.22 can reliably talk to servers 1.21–1.23).

  • When apt upgrade updated kubectl, it jumped to a newer version that was outside of the supported skew for your cluster’s API server.

  • The result: the client likely tried to use API calls or fields that the older server didn’t understand (or in some cases stopped supporting). This led to failures when kubectl tried to start or manage the pods.

Prevention 

Before performing OS updates, always put Jama dependent Kubernetes packages on hold to avoid disruptions. Regularly check for compatibility between client and server versions of kubectl. Consider testing updates in a staging environment before applying them to production.

Additional Information / Metadata /Resources 

Feedback:
Have suggestions or improvements? Please leave your feedback in the comments section below.

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.