Updated: June 2024
Audience: Everyone
Environment: Self-hosted, Replicated - KOTS, Airgap, % Internet
Operating Systems: Ubuntu & RHEL
Summary
During troubleshooting, the statefulset for elasticsearch is deleted, and the pod won't return. Deploying fails, so the elasticsearch pod is stuck in a down state.
kubectl delete sts/elasticsearch
kubectl delete pod elasticsearch
Solution
- First, check the tenantstate table in the Jama database. TRUNCATE the table to remove any error messages that will stop it from being properly re-provisioned.
-
Delete the local tenant config. Steps 2 through 4 should ideally be done in under 60 seconds to prevent deployment errors:
kubectl exec core-0 -- sh -c 'rm -rf /home/contour/tenant_properties /home/contour/tenant.properties /home/contour/setup.properties /home/contour/services.properties /home/contour/node.properties /home/contour/database.properties'
- Followed with
kubectl delete job/tenant-manager
kubectl delete sts/hazelcast sts/activemq
kubectl delete pod/core-0
- Then re-deploy. This removed the block preventing the deployment, which is required to rebuild the statefulset and back up the pod.
Cause
Re-creating the statefulset requires deployment to be run successfully. In this case, we couldn't re-deploy as the deployment continued to fail. We successfully re-deployed and recreated the statefulset and pod by removing the local tenant data.
Please feel free to leave feedback in the comments.
Related to
Comments
0 comments
Please sign in to leave a comment.