Increasing minio storage (KOTS)

Author: Chris Breaux

Date: October 25, 2023

Audience: Everyone

Environmental details: Self-Hosted, Replicated - KOTS

Summary

Unable to generate new support bundles, Replicate snapshots, or create and save new configurations.
Looking at the kotsadm logs, you should see a reference to a XMinioStorageFull

{"level":"error","ts":"2023-10-18T18:56:57Z","msg":"failed to upsert app version: failed to create app version archive: failed to write archive: failed to upload to s3: XMinioStorageFull: Storage backend has reached its minimum free disk threshold. Please delete a few objects to proceed.\n\tstatus code: 507, request id: 178F489B97C59C6B, host id: "}

Solution

-- scale down kotsadm, registry and minio in this order
kubectl -n default scale deployment kotsadm --replicas=0
kubectl -n kurl scale deployment registry --replicas=0
kubectl -n minio scale deployment minio --replicas=0

-- increase the size of the minio volume
kubectl edit pvc -n minio minio-pv-claim
(you should see two places with stroage values you can update)

-- check the status of the pvc, wait for it to bind (may take a minute)
kubectl describe pvc -n minio minio-pv-claim (check a few times until new storage volume is 'Bound)
kubectl get pvc -n minio (should show updated storage value)

-- after this has been save, scale UP the deployments in the reverse order
kubectl -n minio scale deployment minio --replicas=1
kubectl -n kurl scale deployment registry --replicas=1
kubectl -n default scale deployment kotsadm --replicas=1

Cause

The minio storage volume is full. The default is 10Gi. It can quickly fill up from large snapshots with many assets or during a lengthy troubleshooting session with many support bundles or configuration changes.

Prevention We can default to a higher value in our installation scripts.

Additional documentation and Resources for Support Engineers

• Support Engineers, please look at the internal KOTS troubleshooting guide for additional information in addition to request #72775.