feat(mongodb): alert when compaction is needed

[delthas]

Follow-up to ZENKO-5285 (PR #2431), which bundled two alerts in its description and only shipped the first (createIndexes-failed). This adds the second — a fragmentation / compaction-needed signal — that @DarkIsDude explicitly asked be filed as a follow-up.

Two commits, deliberately split

1. mongodb: enable dbstatsfreestorage collector in exporter — one-line values.yaml change adding --collector.dbstatsfreestorage to metrics.extraArgs. The chart's exporter no longer uses --collect-all (dropped in 8414833 for ZENKO-5281), so each sub-collector is opted into individually in extraArgs (already lists dbstats, diagnosticdata, replicasetstatus, topmetrics). Without dbstatsfreestorage, freeStorageSize / totalFreeStorageSize only appear as part of the clunky per-host mongodb_dbstats_raw_<host>_* expansion instead of as top-level mongodb_dbstats_* series.

2. mongodb: alert when compaction is needed — the alert proper.

The alert

mongodb_dbstats_totalFreeStorageSize > 0.3 * mongodb_dbstats_fsTotalSize
for: 1h

Per-(pod, database) granularity. Severity warning. Threshold exposed as the compactionFreeStorageRatioThreshold x-input (default 0.3).

Expressing the threshold as a fraction of filesystem capacity (not of the DB's own storage) lets it scale across cluster sizes:

• 100 GB FS → fires around 30 GB reclaimable
• 1 TB FS → fires around 300 GB reclaimable
• 10 TB FS → fires around 3 TB reclaimable

Heuristic notes

A previous draft combined three legs (FS-pressure + fragmentation ratio + absolute floor). @francoisferrand's review pointed out that the fragmentation-ratio leg added noise without much signal and the FS-pressure leg could mask real waste on under-used disks. The single fs-scaled threshold above folds both concerns into one condition.

Per-DB granularity means the alert tells you pod={{ "{{ $labels.pod }}" }} and database={{ "{{ $labels.database }}" }} but does not identify the specific collection. To find that, an operator runs collStats on the alerting DB. The exporter has a --collector.collstats flag that could expose per-collection visibility, but at Artesca-scale cardinality (thousands of buckets per DB) that's expensive — deferred.

Safety against missing / zero values

• Missing metric for a (pod, database) tuple → PromQL produces empty → no alert.
• totalFreeStorageSize = 0 (no fragmentation) → 0 > 0.3 * fsTotalSize is always false.
• fsTotalSize = 0 would degenerate (0.3 * 0 = 0, then any positive freeStorage would fire), but it's physically impossible for an attached PVC.

Follow-up (not in this PR)

Matching compactionFreeStorageRatioThreshold config option in ZKOP, to expose the knob to ops. Not filed as a ticket yet; will track when this lands.

Related

• ZENKO-5293 — this ticket
• ZENKO-5285 — parent alert ticket; createIndexes-failed half shipped in feat(mongodb): alert on createIndexes command failures #2431
• BB-784 — Backbeat-side LifecycleIndexCreationFailed (feat(lifecycle): alert when index creation fails backbeat#2753)
• BB-783 — putBucketIndexesFailed metric (feat(lifecycle): emit putBucketIndexesFailed metric on index creation errors backbeat#2751)
• Parent epic ARTESCA-16740 — Empty bucket feature

Issue: ZENKO-5293

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[delthas]

@francoisferrand Heuristics are up for discussion -- I chose a mix of:

• Disk is being significantly used (to avoid saving 10GB on a disk that has 10TB free space)
• There is a signficiant number of GBs to save (to avoid alarms on tiny collections)
• There is a significant ratio of data to save (to avoid running compact just to save 10GB on a 10TB DB)

[francoisferrand]

• not sure about thresholds/computation: are these relevant default?
• not sure we should merge in 2.15, or target 2.16 and get time to "preview" this and avoid shipping alerts which would lead to support calls...

[delthas]

Keeping on 2.15 for now as 2.16 doesnt exist.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

The following reviewers are expecting changes from the author, or must review again:

• @francoisferrand

[delthas]

Intentionally set 30% by default (conservative), up for discussion

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[delthas]

/approve

[bert-e]

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

• ✔️ development/2.15

The following branches will NOT be impacted:

• development/2.10
• development/2.11
• development/2.12
• development/2.13
• development/2.14
• development/2.5
• development/2.6
• development/2.7
• development/2.8
• development/2.9

This pull request does not target the following hotfix branch(es) so they
will be left untouched:

• hotfix/2.13.5

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

• Any commit you add on the source branch will trigger a new cycle after the
  current queue is merged.
• Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/2.15

The following branches have NOT changed:

• development/2.10
• development/2.11
• development/2.12
• development/2.13
• development/2.14
• development/2.5
• development/2.6
• development/2.7
• development/2.8
• development/2.9

Please check the status of the associated issue ZENKO-5293.

Goodbye delthas.