After deleting the Elasticsearch statefulset, the deployment fails in KOTS and cannot be restarted.

Jenna
Jenna
  • Updated

Author: Jenna Zwick

Date: November 3, 2023

Audience: Everyone

Environmental details: KOTS channel, Airgap, and Internet

Operating Systems: Ubuntu and RHEL

Summary:

During troubleshooting, the statefulset for elasticsearch is deleted, and the pod won't return. Deploying fails, so the elasticsearch pod is stuck in a down state.

kubectl delete sts/elasticsearch

kubectl delete pod elasticsearch

Solution:

  1. First, check the tenantstate table in the Jama database. TRUNCATE the table to remove any error messages that will stop it from re-provisioning properly.
  2. Delete the local tenant config. Steps 2 through 4 should ideally be done in under 60 seconds to prevent deployment errors:
    kubectl exec core-0 -- sh -c 'rm -rf /home/contour/tenant_properties /home/contour/tenant.properties /home/contour/setup.properties /home/contour/services.properties /home/contour/node.properties /home/contour/database.properties'
  3. Followed with 
    kubectl delete job/tenant-manager
    kubectl delete sts/hazelcast sts/activemq
    kubectl delete pod/core-0
  4. Then re-deploy. This removed the block preventing the deployment, which is required to rebuild the statefulset and back up the pod.

Cause:

Re-creating the statefulset requires deployment to be run successfully. In this case, we couldn't re-deploy as the deployment continued to fail. We successfully re-deploy and re-created the statefulset and pod by removing the local tenant data.

 

Feedback: Please leave feedback in the comments below.

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.