Add support for time-based snapshot intervals

[pcholakov]

This change adds a basic time interval configuration option for triggering automated snapshots. When both time and record-based intervals are set, the logic for the trigger condition is an "and" between the two, so the record interval becomes a minimum requirement for time-based snapshotting.

Time interval-based snapshotting will get a lot more useful with incremental snapshot support, but this is already a big improvement as it allows operators to configure snapshots in slightly more intuitive terms of elapsed time.

The time interval is measured from the created-at timestamp of the published snapshot, which is based on the wall clock time of the producing node. Clock drift error is assumed to be relatively negligible compared to typical snapshotting intervals. A small amount of jitter is added to each partition's snapshot age check to spread snapshot uploads out over time.

Configuration examples

Periodic + minimum number of records

[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval-num-records = 10000
snapshot-interval = "30 min"

Translates into: "snapshot every 30 minutes, as long as the applied LSN has advanced by least 10,000 records since the last snapshot".

Periodic (unconditional)

[worker.snapshots]
destination = "s3://bucket/cluster-name"
snapshot-interval = "24 hours"

Translates into: "snapshot once every 24 hours even if zero new changes are committed".

[github-actions]

Test Results

  7 files  ±0    7 suites  ±0   2m 37s ⏱️ +7s
 47 tests ±0   47 ✅ ±0  0 💤 ±0  0 ❌ ±0 
200 runs  ±0  200 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit ba47e39. ± Comparison against base commit dbb33cf.

♻️ This comment has been updated with latest results.

[tillrohrmann]

Thanks for creating this PR @pcholakov. The changes look good to me. +1 for merging. Do we need to update our docs to also mention the snapshot_interval?

[tillrohrmann]

Maybe update the comment.

[tillrohrmann]

Is it important that we distinguish between created_at > now and created_at == now? If not, then we might get rid fo the error case.

[pcholakov]

No, collapsed to zero - good call to remove the error!

[pcholakov]

Thanks for taking a look, @tillrohrmann! Good points, addressed both. Docs will definitely need an update but there are a couple more PRs in this train and I'll take a pass all the changes are in.