Preventing Application from Starting After Server Reboot

Author: Chris Breaux

Date: 11/26/2024

Audience: Everyone

Environmental details: KOTS 9.6.x, 9.17.x Self-Hosted

Summary

Sometimes, either for troubleshooting or server maintenance, it is desired to stop the application from running and prevent it from re-starting after a server reboot. Typically, after a server reboot, the application will automatically attempt to restart all pods/services.

Solution

In the past, we've recommended using the `ekco` scripts to 'shutdown' the cluster. Two problems: (a) the application will auto-restart after reboot and (b) Replicated has advised against having users access these internal scripts.

/opt/ekco/shutdown.sh
/opt/ekco/startup.sh

A more sensible approach may be to scale down the relevant pods

kubectl scale sts activemq core diff elasticsearch hazelcast search --replicas=0 
kubectl scale deployments oauth saml --replicas=0

kubectl get pods
NAME                                  READY   STATUS    RESTARTS      AGE
ingress-nginx-controller-2wxt7        1/1     Running   0             50m
kotsadm-6985bdf857-dxf76              1/1     Running   1 (52m ago)   3d5h
kotsadm-rqlite-0                      1/1     Running   0             53m
kurl-proxy-kotsadm-7f7868fcc7-cfll4   1/1     Running   0             53m

Then scale them back up reboot, when db server is back up

kubectl scale deployments oauth saml --replicas=1
kubectl scale sts activemq core diff elasticsearch hazelcast search --replicas=1

(can't scale down the ingress-controller daemonset)

Cause

Intended design and functionality of KOTS.

Please feel free to leave feedback in the comments below.