Author: Jenna Zwick
Date: November 3, 2023
Environmental details: KOTS channel, Airgap, and Internet
Operating Systems: Ubuntu and RHEL
During troubleshooting, the statefulset for elasticsearch is deleted, and the pod won't return. Deploying fails, so the elasticsearch pod is stuck in a down state.
kubectl delete sts/elasticsearch
kubectl delete pod elasticsearch
- First, check the tenantstate table in the Jama database. TRUNCATE the table to remove any error messages that will stop it from re-provisioning properly.
Delete the local tenant config. Steps 2 through 4 should ideally be done in under 60 seconds to prevent deployment errors:
kubectl exec core-0 -- sh -c 'rm -rf /home/contour/tenant_properties /home/contour/tenant.properties /home/contour/setup.properties /home/contour/services.properties /home/contour/node.properties /home/contour/database.properties'
- Followed with
kubectl delete job/tenant-manager
kubectl delete sts/hazelcast sts/activemq
kubectl delete pod/core-0
- Then re-deploy. This removed the block preventing the deployment, which is required to rebuild the statefulset and back up the pod.
Re-creating the statefulset requires deployment to be run successfully. In this case, we couldn't re-deploy as the deployment continued to fail. We successfully re-deploy and re-created the statefulset and pod by removing the local tenant data.
Feedback: Please leave feedback in the comments below.