KOTS Deployment Failed: Error: upgrade failed: cannot patch "tenant-manager" with kind Job: Job.batch "tenant-manager" is invalid

Author: Sravya Bandari
Date: September 5, 2025
Audience: Agents and Admins
Environment: Self-Hosted, Replicated - KOTS

Summary

When deploying a new sequence or redeploying an existing one from the KOTS Admin Console, the deployment may hang with a spinning wheel or display a Deploy Failed message. The deploy logs may show an error similar to the following:

Full error:

Error: UPGRADE FAILED: cannot patch "tenant-manager" with kind Job: Job.batch "tenant-manager" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string{"controller-uid":"ed8ddbce-4273-4cc6-a713-810b78dee7b9", "job-name":"tenant-manager", "kots.io/app-slug":"jama-k8s", "kots.io/backup":"velero"}, Annotations:map[string]string{"kots.io/app-slug":"jama-k8s"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"tenant-manager", Image:"registry.replicated.com/jama-k8s/tenantmanager:0023b2d2", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"APPLICATION_LICENSE", Value:"PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPGxpY2Vuc2UgdmVyc2lvbj0iMS4wIj48bGljZW5zZURhdGE+PGNsaWVudE5hbWU+TUljcm9zb2Z0IC0gRGV2aWNlczwvY2xpZW50TmFtZT48c2ZJZD4wMDE3MDAwMDAxMGFTa0lBQVU8L3NmSWQ+PGNsaWVudE9yZ2FuaXphdGlvbj5NSWNyb3NvZnQgLSBEZXZpY2VzPC9jbGllbnRPcmdhbml6YXRpb24+PHByb2R1Y3ROYW1lPkphbWE8L3Byb2R1Y3ROYW1lPjxudW1iZXJOYW1lZD40PC9udW1iZXJOYW1lZD48bnVtYmVyQ29uY3VycmVudD43PC9udW1iZXJDb25jdXJyZW50PjxudW1iZXJTdGFrZWhvbGRlcj41MDA8L251bWJlclN0YWtlaG9sZGVyPjxudW1iZXJOYW1lZFJldmlld2VyPjA8L251bWJlck5hbWVkUmV2aWV3ZXI+PG51bWJlclRlc3RSdW5uZXI+MDwvbnVtYmVyVGVzdFJ1bm5lcj48Y3JlYXRpb25EYXRlPjEyLzE4LzIwMjM8L2NyZWF0aW9uRGF0ZT48ZXhwaXJhdGlvbkRhdGU+MTIvMjMvMjAyMzwvZXhwaXJhdGlvbkRhdGU+PG1haW50ZW5hbmNlRGF0ZT4xMi8yMy8yMDIzPC9tYWludGVuYW5jZURhdGU+PHByb2R1Y3RFZGl0aW9uPkVudGVycHJpc2U8L3Byb2R1Y3RFZGl0aW9uPjxsaWNlbnNlVHlwZT5jb21tZXJjaWFsPC9saWNlbnNlVHlwZT48bW9kdWxlcz48bW9kdWxlIG5hbWU9IlJldmlldyBDZW50ZXIiIG51bWJlck9mU2VhdHM9Ii0xIi8+PC9tb2R1bGVzPjxudW1iZXJSZWFkT25seT4wPC9udW1iZXJSZWFkT25seT48bnVtYmVyRmxvYXRpbmdSZWFkT25seT4wPC9udW1iZXJGbG9hdGluZ1JlYWRPbmx5PjxudW1iZXJSZXZpZXdlcj41MDA8L251bWJlclJldmlld2VyPjwvbGljZW5zZURhdGE+PHNpZ25hdHVyZT5UMnpPS2pOeXpSa0xKd0FtQlhac3FFajlTZ2FaZzdRZytKWm14Zm5FbEZkWGpJQ1BnczgraDBuRC8vM08wWjh2L3Bja3RxRDhla2NFaVpSa1VQSEJvOFpSVGRLaGwyZmZISnRaNFVsTDVIMFVac1ZaS1l5aUh5WFN2KzRRUk01T2lhNzJMTVZsMTdSai9WK1A3aWhjN1VBUjlPN2FIbDRSbDVnelpPNVU2QU09PC9zaWduYXR1cmU+PC9saWNlbnNlPg==", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"HTTPS_PORT", Value:"443", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"HTTP_PORT", Value:"80", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"USE_SSL", Value:"false", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"BASE_URL_HOST", Value:"pilot.jamadev.microsoft.com", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"TENANT_NAME", Value:"jama", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"TENANT_API_TOKEN", Value:"4A9BALW4NTX9OYF72QGX9AMYWX4B7LOC", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"CLUSTER_NAME", Value:"JamaElasticSearch", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAMA_CORE_PORT", Value:"80", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAMA_CORE_HORIZONTAL_SCALING_ENABLED", Value:"True", ValueFrom:(core.EnvVarSource)(nil)}, core.EnvVar{Name:"JAMA_CORE_HOST", Value:"core", ValueFrom:(core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:134217728, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, Requests:core.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:134217728, scale:0}, d:resource.infDecAmount{Dec:(inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(core.Probe)(nil), ReadinessProbe:(core.Probe)(nil), StartupProbe:(core.Probe)(nil), Lifecycle:(core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"Never", TerminationGracePeriodSeconds:(int64)(0xc00d7211c8), ActiveDeadlineSeconds:(int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"tenant-manager", AutomountServiceAccountToken:(bool)(nil), NodeName:"", SecurityContext:(core.PodSecurityContext)(0xc01fa49a80), ImagePullSecrets:[]core.LocalObjectReference{core.LocalObjectReference{Name:"jama-k8s-tenant-manager-registry"}}, Hostname:"", Subdomain:"", SetHostnameAsFQDN:(bool)(nil), Affinity:(core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(int32)(nil), PreemptionPolicy:(core.PreemptionPolicy)(nil), DNSConfig:(core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable

Solution

On the application server, run the following commands to remove the conflicting resources:

kubectl delete sts/core sts/hazelcast sts/activemq job/tenant-manager

After deleting the resources, re-deploy the application from the KOTS Admin Console.

Cause

This issue typically occurs if:

A deployment was attempted twice consecutively, or
A redeployment was initiated while the tenant-manager job was still running.

In these cases, Kubernetes does not allow the immutable spec.template of the Job to be modified, resulting in a deployment failure.

Additional Information/Metadata

Support bundles can provide useful context for troubleshooting:

Use kubectl get pods to check if the tenant-manager pod is running and to view the age of the pods.
Review kotsadm logs for detailed error information.
If the deployment fails, the connect-drainer pod may also fail; check its logs for related errors.

References

Related to