Ensure that a processing job's status is always updated correctly

[annavik]

Summary

If something goes wrong with a job, we are seeing an error, but the job status still has the value "STARTED". Sometimes, we have the same problem for successful jobs (job is done, but the status is still "STARTED"). The problem affects the overall status, and also status for individual stages.

Detailed Description

Not getting the correct status makes it difficult to present correct information to users, for example adjust the status bar color and labels to highlight there is a problem.

How to Reproduce the Bug

Environment

• OS and/or device: Mac OS
• Browser Chrome
• Version: Production

Screenshots

[annavik]

After doing a retry on the job above (https://antenna.insectai.org/projects/107/jobs/1244), things seem to go better, except the job never changed its status to "SUCCESS" and still looks "STARTED".

Later, Michael manually updated the status for this job, so it now has status "SUCCESS". However last stage still has status "STARTED".

[annavik]

This job (see https://antenna.insectai.org/projects/107/jobs/1246) also stopped due to problems, but did not had the status updated, so it still looks like it's running:

[mihow]

This is related to #719

[mihow]

This is addresses & fixed by the changes in #910

[annavik]

I'm sorry, but I still have the same problems here. From a user perspective I wouldn't consider this as fixed. For example, this job I started this morning looks like it's running but is actually stuck. I have also recently seen jobs that has errors, but I can't understand from the status if still active or not. Can we keep this issue open until we have tested things a bit more?

[mihow]

Sorry this wasn't suppose to be closed. I must have used a keyword in my PR. It will be fixed in #910

The changes in #937 only affect the initial status being wrong, when the job is first created.

The job status will never be updated once the background task worker disappears, so I believe that's the cause of this issue.