Add alert when fluentd queue buffers grow too large

[priteau]

No description provided.

[technowhizz]

Should we not warn at 100MiB as that is what the default Opensearch http.max_content_length is set as? Perhaps even warn at 99MiB

[priteau]

Should we not warn at 100MiB as that is what the default Opensearch http.max_content_length is set as? Perhaps even warn at 99MiB

I am open to suggestions, I just picked a random number.

[dougszumski]

Thanks Pierre.

I think Dawud has a point that until we fix https://stackhpc.atlassian.net/browse/INFRA-666, a lower threshold would be useful, but we should make https://stackhpc.atlassian.net/browse/INFRA-666 a priority.

[Alex-Welsh]

Something is better than nothing. Since the INFRA-666 task is still on the backlog, I propose we move forward with this and we can update it with something more fancy down the line.
Though, we should probably rebase this change onto antelope

[jovial]

Should we not warn at 100MiB as that is what the default Opensearch http.max_content_length is set as? Perhaps even warn at 99MiB

100 MiB seems like a sane default (99MiB could still technically be posted)