Too Many Open Files
https://github.com/fluent/fluent-bit/issues/1777
@aderuwe It might be more related to inotify mechanism - as by default (e.g. when using the default Docker image) this mechanism is used for tailing files in in_tail plugin:
https://linux.die.net/man/2/inotify_init1
Instead of looking at ulimit and available files, you might also check more specific inotify related settings such as:
/proc/sys/fs/inotify/max_user_watches /proc/sys/fs/inotify/max_user_instancesmaybe it works for you.
In my case in the end I have rebuilt the Docker image with -DFLB_INOTIFY=Off option off, so that instead of using more performant inofify mechanism, the plugin rather uses the more old-school stat mechanism for tailing files - and it works for me for now as a workaround - see #1778 - although it might have problems when using with symlinks probably.
The final solution is planned by @edsiper in this ticket to have a configuration option available in the upcoming https://github.com/fluent/fluent-bit/milestone/7 release.
sysctl -w fs.inotify.max_queued_events=524288
sysctl -w fs.inotify.max_user_instances=16383
sysctl -w fs.inotify.max_user_watches=524288