After deleting the Elasticsearch statefulset, the deployment in KOTS fails and cannot be restarted.

Amanda Jennewein
Amanda Jennewein
  • Updated

Updated: June 2024

Audience: Everyone

Environment: Self-hosted, Replicated -  KOTS, Airgap, % Internet

Operating Systems: Ubuntu & RHEL

Summary

During troubleshooting, the statefulset for elasticsearch is deleted, and the pod won't return. Deploying fails, so the elasticsearch pod is stuck in a down state.

kubectl delete sts/elasticsearch

kubectl delete pod elasticsearch

Solution

  1. First, check the tenantstate table in the Jama database. TRUNCATE the table to remove any error messages that will stop it from being properly re-provisioned.
  2. Delete the local tenant config. Steps 2 through 4 should ideally be done in under 60 seconds to prevent deployment errors:
    kubectl exec core-0 -- sh -c 'rm -rf /home/contour/tenant_properties /home/contour/tenant.properties /home/contour/setup.properties /home/contour/services.properties /home/contour/node.properties /home/contour/database.properties'
  3. Followed with 
    kubectl delete job/tenant-manager
    kubectl delete sts/hazelcast sts/activemq
    kubectl delete pod/core-0
  4. Then re-deploy. This removed the block preventing the deployment, which is required to rebuild the statefulset and back up the pod.

Cause

Re-creating the statefulset requires deployment to be run successfully. In this case, we couldn't re-deploy as the deployment continued to fail. We successfully re-deployed and recreated the statefulset and pod by removing the local tenant data.

 

Please feel free to leave feedback in the comments.

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.