Failed to create fsnotify watcher: too many open files

Author: Chris Breaux

Updated: November 2024

Audience: Agents and Admins

Environmental details: Self-Host KOTS

Summary

This error can appear when following logs from the terminal.
e.g. kubectl logs -f core-0
It may appear that something has gone wrong, but it's just that the environment limits the number of open 'items' it can watch. Most often, it appears during new installations/upgrades.

Solution

Check value: sudo sysctl fs.inotify.max_user_watches

Raise value: sudo sysctl -w fs.inotify.max_user_watches=2099999999

Note: You may not need to raise the value this high. It may be sufficient to double or triple the existing value. Additionally, these other values may need to be checked and increased similarly.
sudo sysctl -w fs.inotify.max_user_instances=2099999999
sudo sysctl -w fs.inotify.max_queued_events=2099999999

Cause

The number of filesystem items that can be 'watched' has been exceeded.

Prevention

Raise fs.inotify.max_values. (Limited by available memory)

Please feel free to leave feedback in the comments below.