Increasing minio storage (KOTS)

Amanda Jennewein
Amanda Jennewein
  • Updated

Author: Chris Breaux

Date: October 25, 2023

Audience: Everyone

Environmental details: Self-Hosted KOTS

Summary:

Unable to generate new support bundles, Replicated snapshots, or create save new configurations.
Looking at the kotsadm logs, should see a reference to a XMinioStorageFull

{"level":"error","ts":"2023-10-18T18:56:57Z","msg":"failed to upsert app version: failed to create app version archive: failed to write archive: failed to upload to s3: XMinioStorageFull: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed.\n\tstatus code: 507, request id: 178F489B97C59C6B, host id: "}

Solution:

-- scale down kotsadm, registry and minio in this order
kubectl -n default scale deployment kotsadm --replicas=0
kubectl -n kurl scale deployment registry --replicas=0
kubectl -n minio scale deployment minio --replicas=0

-- increase the size of the minio volume
kubectl edit pvc -n minio minio-pv-claim
(you should see two places with stroage values you can update)

-- check the status of the pvc, wait for it to bind (may take a minute)
kubectl describe pvc -n minio minio-pv-claim (check a few times until new storage volume is 'Bound)
kubectl get pvc -n minio (should show updated storage value)

-- after this has been save, scale UP the deployments in the reverse order
kubectl -n minio scale deployment minio --replicas=1
kubectl -n kurl scale deployment registry --replicas=1
kubectl -n default scale deployment kotsadm --replicas=1

Cause:

The minio storage volume is full. The default is 10Gi It can quickly fill up from large snapshots with many assets or during a lengthy troubleshooting session with many support bundles or configuration changes.

Prevention:
We can default to a higher value in our installation scripts.

 

Additional documentation and Resources for Support Engineers:

  • Support Engineers, please take a look at the internal KOTS troubleshooting guide for additional information in addition to request #72775.

 

 

Related to

Was this article helpful?

0 out of 0 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.