Incremental Updates: honor events before changed_at

[oxzi]

As changes might not be committed in the database in a continuous linear fashions, elements out of order might get ignored. This change adds a window to the past for incrementalFetch, honoring older elements. The incrementalApplyPending function now compares the changed_at value for known elements, skipping those without a change.

[oxzi]

Something to note about this PR change is that it will retry the changes within each table for the last ten minutes. Since it will silently abort if the changed_at value has not changed or the item is deleted again, it does no real work, but it is not really clean, but more a hacky solution.

Another idea I toyed with was to use not only the last changed_at value, but also the current timestamp iff a changed_at value has already been fetched. For example, use min(changed_at - 10m, current_timestamp - 30m). In my naive thinking, this sliding time window would slowly catch up the fixed changed_at timestamp, while still being large enough to catch the weirdest of time travelers¹.

To be really sure, another layer could be added to query each known element if it has another changed_at or deleted value and to query for everything with an unknown primary key. This, however, should only be done once in a while.

Footnotes

1. In case of a long lasting split brain situations of a database cluster, other issues might arise as well. ↩

[Al2Klimov]

I.e this is a workaround for something web already could and should handle?

As such, it seems a bit over-engineered at first sight. There are other options:

• Write milliseconds where already allowed desired (Don't cut off milliseconds from current-time-in-milliseconds icinga-notifications-web#306)
• Write increasing values (Ensure new/updated rows having a changed_at greater than all previous ones icinga-notifications-web#311)

• If we decide to workaround this, here is something less complex: incrementalFetch(): fix race condition which would miss rows based on changed_at #288

But ideally I'd like web to handle this. I've already given them all instruments for that.

[julianbrost]

Reopened because I have no idea what's going on here, especially given there is now another PR over in the web project that wants to revert the PR that was used to close this PR.