Deployments should "wait" until deltas have been generated

[lawik]

I believe @joshk has plans around this for Releases. Some kinda of staging step.

But essentially when we switch target version for a deployment we immediately kick off the delta generation. But if you are doing 1000 concurrents that may well trigger 1000 deployments before the deltas are even available.

You can manually operate this by pausing the deployment while you switch and then wait a little bit for deltas to populate, then resume. And it should be all ready to go.

It is not terrible in the current way it works but it is a race between generating deltas and passing out firmware. Definitely not optimal. It would also nice to have an indication of how many delta jobs were fired off so you can watch them complete (whether success or failure). Ideally also report what the failures are for the versions that failed. Lots of potential improvements there.

On the flip side. The current thing is quite simple.

[joshk]

I have a few different ideas on how we can achieve this: (I'm also thinking out loud)

We could couple this together with DeploymentReleases. Its a natural fit but it would delay the solution as this already is a sizeable change, thus just adding to it.

Or we can start with adding a field to DeploymentGroup which indicates the 'delta status' / 'is this ready to send to devices'. This doesn't feel unreasonable, and the logic would be ported over to DeploymentRelease when we start on that feature.

You could possibly agrue that we add this to Firmware since deltas are generated from firmwares, but since its a DeploymentGroup setting it seems to make the most sense to have it bound it to the deployment. This also helps us answer the question of 'how many deltas should we generate', since we can now limit the deltas to the firmwares of devices connected to a DeploymentGroup, which is usually just 1 firmware version (unless some devices haven't updated yet as they haven't connected in a while)

This all said, I think we should add this to DeploymentGroup with a status field to indicate if the DeploymentGroup is 'ready' ('generating deltas', 'ready', 'error generating deltas').

[lawik]

So we currently fire off delta generation for all firmware version currently in use, like you describe. We'd need some book-keeping for when those complete and it may not be that they complete successfully. There are cases where making a delta makes no sense so firing off 5 delta generations might get us 3 completed deltas.

Does the orchestrator reset a bunch of state when a new deployment is started? I'm thinking maybe we could trigger the delta generation synchronously within the orchestrator and just grind through them. It would then know how they turned out and when the set has been processed.

[lawik]

So the handler for "deployments/update" could do this it seems. Or rather Orchestrator.trigger_update.

The orchestrator state can hold whether it has completed deltas. If deltas have already been generated, attempting them is really cheap, it is essentially just checking if they already exist. So on orchestrator restart or something it might re-check but it wouldn't do very much of the expensive work again.

[lawik]

I think this avoids getting state on the DeploymentGroup or Firmware that risks getting out of phase.

[joshk]

After talking this over with Lars in Slack, here is a brief summary of my thoughts:

• Track the status of FirmwareDelta generation:
  • Adjust the FirmwareDelta schema to include a 'status' ('generating', 'completed', 'failed', 'timed out')
  • Create the row before generation begins, update the row when it completes (including when it fails)
  • Show the FirmwareDelta status in the DeploymentGroup UI
  • Have a timer in the Orchestrator when makes sure delta generation is still happening, or marks it as timed out.
• When a FirmwareDelta generation finishes, have it emit a pubsub event which the Orchestrator listens for, which also allows it to decide to run now that the deltas have completed (or not). Additionally, these events can also be utilised for the UI.
• Add an overall 'status' to the DeploymentGroup, which can be used for the UI, and can be updated by the Orchestrator.

This can be split into three different PR, implemented in the order specified.

This should all be fairly defined and constrained, as well as giving us more data to work with.

[lawik]

I should be able to get on this a bit in the next few days.

Timer in the orchestrator or Oban clean-up job? I feel like the latter is more reliable.

[nshoes]

@lawik I vote Oban for cleaning up things, it's a pattern we already heavily follow.