workflow showing as stopped after restart

[oliver-sanders]

When workflows are restarted from the GUI, they don't always appear as running, even if you request a scan. But if you reload the GUI they show up as running straight away.

Probably either:

• A missing updated delta from the UIS.
• An internal state problem in the UI.

Transfer this issue to cylc-ui if it turns out to be the latter.

I've struggled to reproduce this locally, however, it reproduces intermittently when workflows are started on remote hosts.

[oliver-sanders]

To replicate:

• Install N workflows.
• Load the GUI, wait for them all to appear.
• Start the workflows.
• Request a scan from the GUI, wait for them to change state.

I've been using a script to reproduce this:

1. Load GUI
2. Run script
3. Once you see Continue ... all workflows are running.
4. Press enter and all workflows will stop then restart.
5. 3. Once you see Continue ... all workflows are running.
6. But not all will come back in the GUI...
7. Press enter again to repeat, press Ctrl+C to exit.

#!/usr/bin/env bash

set -euo pipefail

ITT=20
WFLOW=one  # <= Your workflow here (single task workflow is fine)

install () {
  for i in $(seq 1 $ITT); do
    local NAME="$WFLOW/$i"
    cylc install $WFLOW -n "$NAME" --no-run-name &
  done
  wait
  sleep 5
}

play () {
  for i in $(seq 1 $ITT); do
    local NAME="$WFLOW/$i"
    cylc play "$NAME" --pause &
  done
  wait
  sleep 5
}

stop () {
  for i in $(seq 1 $ITT); do
    local NAME="$WFLOW/$i"
    cylc stop "$NAME" &
  done
  wait
  sleep 5
}

cylc clean "$FLOW/*" -y
install
while true; do
  play
  echo
  echo -n 'Continue... '
  read mess
  stop
done

Some workflows remain in the stopped state, yet if you click on them, they will load correctly (but remain in the stopped state).

Example showing a 35% failure rate (on one occasion I got a 100% failure rate!!!):

From inspection, the 'status': 'running' update is missing from the deltas. Here's an example sequence for a workflow that did not update:

{"id": "1", "type": "data", "payload": {"data": {"deltas": {"id": "~me/generic/11/run1", "added": {"__typename": "Added"}, "updated": {"workflow": {"id": "~me/generic/11/run1", "status": "stopped", "statusMsg": "stopped", "owner": "me", "host": "", "port": 0, "__typename": "Workflow"}, "__typename": "Updated"}, "pruned": {"__typename": "Pruned"}, "__typename": "Deltas"}}}}
{"id": "1", "type": "data", "payload": {"data": {"deltas": {"id": "~me/generic/11/run1", "added": {"workflow": {"id": "~me/generic/11/run1", "status": "stopped", "statusMsg": "stopped", "owner": "me", "host": "", "port": 0, "stateTotals": {"waiting": 8, "expired": 0, "preparing": 0, "submit-failed": 0, "submitted": 0, "running": 0, "failed": 0, "succeeded": 0}, "latestStateTasks": {"failed": [], "preparing": [], "submit-failed": [], "submitted": [], "running": []}, "__typename": "Workflow"}, "__typename": "Added"}, "updated": {"__typename": "Updated"}, "pruned": {"__typename": "Pruned"}, "__typename": "Deltas"}}}}
{"id": "1", "type": "data", "payload": {"data": {"deltas": {"id": "~me/generic/11/run1", "added": {"__typename": "Added"}, "updated": {"workflow": {"id": "~me/generic/11/run1", "owner": "me", "host": "cazcylc6.spice.sc.metoffice.gov.uk", "port": 43271, "__typename": "Workflow"}, "__typename": "Updated"}, "pruned": {"__typename": "Pruned"}, "__typename": "Deltas"}}}}

So:

• updated: status=stopped ...
• added: status=stopped ...
• updated: port=1234 ...

The last delta contains details about the running scheduler, e.g. the port number, but not the state change stopped => running.

[oliver-sanders]

Did a bit of debugging, the workflow status is correct in the UIS data store, however, the status field (and likely others) are not reliably included in the deltas.

The failure rate seems to be somewhere between 25% and 33%! Quite high!

It looks like there is some sort of race condition going on. With this hack, the workflows show up as running in the GUI 100% of the time:

diff --git a/cylc/uiserver/data_store_mgr.py b/cylc/uiserver/data_store_mgr.py
index f88d76f..0e46121 100644
--- a/cylc/uiserver/data_store_mgr.py
+++ b/cylc/uiserver/data_store_mgr.py
@@ -476,6 +476,8 @@ class DataStoreMgr:
 
         if status is not None:
             flow.status = status
+        else:
+            flow.status = self.data[w_id]["workflow"].status
         if status_msg is not None:
             flow.status_msg = status_msg
         if pruned:

However, we wouldn't want to go ahead with this hack as we shouldn't need to repeat the status in this way and this hack only targets the status field, other changed fields could also be getting ignored.