Isolated start-up and shut-down graphs.

[hjoliver]

[UPDATE: likely to be superseded by a more general solution; partially addresses the retrospectively-written issue #7020]

Supersede #4912
Supersede #5036
(Really addresses the original #4903 but specifically just for start-up and shut-down graphs)

Implements isolated NON-CYCLING startup and shutdown graphs as special cycle points, to avoid the pain of eternal dependence on initial and final tasks.

[UPDATE from below] I initially called the new isolated graphs "alpha" and "omega" to avoid any confusion with the concepts of initial, start, final, stop, cycle points. However, that sounds a bit too quirky and I've switched to "startup graph" and "shutdown graph", which is descriptive, and it shouldn't be confusing if we consistently use "startup" and "shutdown" (not "start" and "stop").

How it works:

• at start-up, the startup graph loads and runs to completion
• then the main (cycling) graph loads, and runs to completion
• then the shutdown graph loads and runs to completion

Also:

• restart works as normal, in any of the graphs
• you can manually trigger tasks in any section, in which case (e.g.) startup/foo will coexist in the task pool with 3/bar etc.
  • if you need to retrigger an initial install task (e.g.) part way through a workflow run, you may or may not want to pause the main flow while it runs - to handle this we should implement flow pause (which is intended anyway, but not done yet)

EXAMPLE

[scheduling]
    [[graph]]
        startup= "a => b"
        shutdown= "x => y"
        R1 = "foo => bar"
[runtime]
    [[a, b, x, y, foo, bar]]

Result:

$ cylc cat-log junk | grep '=> running'
INFO - [startup/a submitted job:01 flows:1] => running
INFO - [startup/b submitted job:01 flows:1] => running
INFO - [1/foo submitted job:01 flows:2] => running
INFO - [1/bar submitted job:01 flows:2] => running
INFO - [shutdown/x submitted job:01 flows:3] => running
INFO - [shutdown/y submitted job:01 flows:3] => running

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg and conda-environment.yml.
• Tests are included (or explain why tests are not needed).
• CHANGES.md entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at Document isolated start-up and shut-down graphs. cylc-doc#559
• If this is a bug fix, PRs raised to both master and the relevant maintenance branch.

[hjoliver]

I'm tentatively putting this one back to 8.1.0.

• It's ready to go and shouldn't be hard to review (and the underlying principles were discussed at length and agreed)
• It's a nice feature that many users will like (it makes special behaviour at start much easier to understand and configure)
• It has some nasty conflict potential if left too long

[hjoliver]

Merged master and deconflicted.

[hjoliver]

@dpmatthews - pinging you for a big-picture review (given your involvement in the discussions that led to this).

[hjoliver]

Squashed and rebased.

[hjoliver]

(squashed, rebased, and deconflicted)

[dwsutherland]

Having a little play:

[meta]
    title = "alpha & omega graph sections"
[scheduler]
    UTC mode = True
    allow implicit tasks = True
[scheduling]
    initial cycle point = 20250301T00
    final cycle point = 20250310T00
    [[graph]]
        alpha = "prep => unpack"
        PT12H = "foo[-PT12H] => foo => bar"
        omega = "pack => unprep"
[runtime]
    [[root]]
        script = sleep $((3 + $RANDOM % 7))
    [[prep, unprep]]
        script = sleep 30

Nice that the UI doesn't care what a cycle looks like:

Might want to gracefully handle the mixing of tasks in different sections, i.e. if I change startup to contain foo

        alpha = "prep => unpack => foo"

    packages/metomi/isodatetime/parsers.py", line 435, in get_info
        keys, date_info = self.get_date_info(date)
      File "/home/sutherlander/.envs/flow/lib/python3.10/site-
    packages/metomi/isodatetime/parsers.py", line 395, in get_date_info
        raise ISO8601SyntaxError("date", date_string)
    metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601 date
    representation: alpha
CRITICAL - Workflow shutting down - Invalid ISO 8601 date representation:
    alpha

(I guess there just needs to be a validation catch)

[dwsutherland]

Also, as expected, if you use clock triggers in this section:

    [[xtriggers]]
        clock = wall_clock(offset=PT5M)
    [[graph]]
        alpha = """
@clock => prep
prep => unpack
"""

    packages/metomi/isodatetime/parsers.py", line 395, in get_date_info
        raise ISO8601SyntaxError("date", date_string)
    metomi.isodatetime.exceptions.ISO8601SyntaxError: Invalid ISO 8601 date representation: alpha
CRITICAL - Workflow shutting down - Invalid ISO 8601 date representation: alpha

let's see what happens if I use workflow_state:

    [[xtriggers]]
        poll_prep = workflow_state(%(workflow)s//%(point)s/prep)
    [[graph]]
        alpha = """
prep
@poll_prep => unpack
"""

(otherwise it would work I think haha).. Perhaps we can just use raw argument value if adjust_point_to_db throws this exception? or by pass for alpha and omega explicitly

Are these tasks even recorded in the task table, and with alpha as the cycle point?

[dwsutherland]

Are these tasks even recorded in the task table, and with alpha as the cycle point?

Well, stop/restart works while alpha active, so must be..

[dwsutherland]

Should alpha be translated under the hood to ICP?
I'm trying to think of cold-start tasks that need to work with cycle offsets to setup data in share/cycle...
Maybe the tasks will just have to be aware of the ICP via a env var (if not already)

[hjoliver]

Should alpha be translated under the hood to ICP?

Probably not. The alpha cycle is appropriate for stuff that doesn't care about the cycle point value, such as workspace setup and code deployment tasks (but they still have access to the ICP value via $CYLC_WORKFLOW_INITIAL_CYCLE_POINT if they need it).

The main cycling section still has an ICP for all the usual purposes.

[hjoliver]

(rebased onto current master)

[hjoliver]

I initially called the new isolated graphs "alpha" and "omega" to avoid any confusion with the concepts of initial, start, final, stop, cycle points. However, that sounds a bit too quirky and I've switched to "startup graph" and "shutdown graph", which is descriptive, and it shouldn't be confusing if we consistently use "startup" and "shutdown" (not "start" and "stop"). ... updated issue description.

[dwsutherland]

I initially called the new isolated graphs "alpha" and "omega" to avoid any confusion with the concepts of initial, start, final, stop, cycle points. However, that sounds a bit too quirky and I've switched to "startup graph" and "shutdown graph", which is descriptive, and it shouldn't be confusing if we consistently use "startup" and "shutdown" (not "start" and "stop"). ... updated issue description.

I don't want to throw a curve-ball in, but that nomenclature made me think.. Would it make sense to change:

[scheduling]
    [[graph]]
        startup graph = "a => b"
        shutdown graph = "x => y"
        R1 = "foo => bar"
[runtime]
    [[a, b, x, y, foo, bar]]

to this:

[scheduling]
    [[startup graph]] 
        R1 = "a => b"
    [[graph]]
        R1 = "foo => bar"
    [[shutdown graph]]
        R1/$ = "x => y"
        
[runtime]
    [[a, b, x, y, foo, bar]]

That way people can utilize whatever cycling, xtriggers (i.e. workflow_state), point reference they like in the start/shutdown graphs..

[oliver-sanders]

Would it make sense to change

Yes, but only if we extend the scope of this PR.

I don't want to throw a curve-ball in

Here's a curveball loosely based on a real example:

[scheduling]
    [[graph]]
        R1 = install => spinup => run => report

        [[[install]]]
            R1 = install<model>

        [[[spinup]]]
            cycling mode = integer
            initial cycle point = 1
            final cycle point = {{ SPINUP_LENGTH }}
            R1 = spinup<model>_cold => spinup<model>
            P1 = spinup<model> => spinup<model + 1>
            R1/$ = spinup<model> => make_bc

        [[[run]]]
            initial cycle point = {{ START_CYCLE }}
            final cycle point = {{ END_CYCLE }}
            P1D = foo & bar => run<model> => baz

        [[[report]]]
            R1 = pub => qux => report

Three part graphs (startup, run, teardown) are a tad restrictive. The real word example has a date-time cycling part, but an ungodly number of spinup tasks (currently implemented via parameters) that would better fit integer-cycling logic.

However, I don't think it would be possible to implement under the runahead limit model. Solving that one would involve clearing and rebuilding the task pool?

[hjoliver]

I quite like David's suggestion, but the problem with it is it suggests that non-R1 (i.e., cycling) is possible in the startup and shutdown graphs.

(That is perfectly possible and not much of a change to this PR, but I either decided or was persuaded against it two years ago ... I don't recall which at this point).

[hjoliver]

As for @oliver-sanders suggestion, it is also quite easy to change this to an arbitrary number of isolated graph sections, and my original experiments did that, but as I recall @dpmatthews persuaded me out of that. Not sure of the runahead implications at this point, would need to think through that.

[oliver-sanders]

(Note, my example above also supported both integer and datetime cycling)

[dwsutherland]

For the sake of our operation sync startup and removing our dependency on ICP tasks, we should get this going.

The idea I think will satisfy most parties with minimal backend headache is to have:

• Bespoke graph sections, whose members are not shared (so validation fails if the same named task is in multiple sections).
• All tasks will use the same task pools (runahead & active), and hence can have the same cycling sections (don't bother with different cycling modes at the moment).
• Sections are prioritised (in some way), so that if higher priority sections have active tasks the lower aren't released from runahead or not actioned.

e.g.

[scheduling]
    [[graph]]
        R1 = install => spinup => run => report

        [[[install]]]
            R1 = install<model>

        [[[spinup]]]
            R1 = spinup<model>_cold => spinup<model>
            R5/P1D = spinup<model> => spinup<model + 1>
            R1/+P6D = spinup<model> => make_bc

        [[[run]]]
            P1D = foo & bar => run<model> => baz

        [[[report]]]
            R1 = pub => qux => report

or more back compatible:

[scheduling]
    [[deploy]]
        priority = 1
        R1 = "a => b"
    [[graph]]
        priority = 2
        R1 = "foo => bar"
    [[final]]
        priority = 3
        R1/$ = "x => y"

(or [scheduling]priority = "deploy => graph => final" (or w/e), but would have to work with repeated section merging (i.e. from the same named graph in include files))

Internally this would be the easiest to implement (and perhaps build on later), you'd essentially create a couple of structures to check for runahead release in the Config/TaskPool:

config.graph_priority = {
    1: {'name': 'deploy', 'members': {'a', 'b'}},
    2: {'name': 'graph', 'members': {'foo', 'bar'}},
    3: {'name': 'final', 'members': {'x', 'y'}},
}

(and/or perhaps on the TaskDef)
with the following to populate with active task IDs, only releasing/actioning priority 2 task(s) if 1 is empty, and 3 if 1&2 are empty (effectively holding lower priority actions if you re-run a higher priority tasks)

pool.graph_priority = {
    1: set(),
    2: set(),
    3: set(),
}

IMHO this is the most simple back compatible approach that can be built on in the future..

[oliver-sanders]

IMHO this is the most simple back compatible approach that can be built on in the future..

Agreed that it's important to flesh out the direction of this feature now to ensure we don't close the door for future development.

We don't have to deliver the full vision upfront, just ensure that the syntax and implementation model we pick now is compatible with the longer term vision.

I'm not sure I'm following on the "priority" options above? Is that an alternative way of specifying the order in which the graph sections run, or a way of controlling the order if multiple graph sections were permitted to run in parallel?

Here are the implementation options I've thought of so far (there may be more). If we could work out which of these we want to aim for, we can settle the syntax, etc to leave room for this feature to grow as desired in the future:

Graph sections:

1. Pre defined graph sections
   • I.e, the alpha-omega approach as implemented.
   • Give users pre-defined graph sections they can make use of.
   • Answers the obvious cases like spinup teardown graphs.
2. Bespoke graph sections as pipelines
   • A generalisation to the above which allows more than three graph sections.
   • These will be run in strictly linear order, one at a time.
   • The user specifies the order in which these run (e.g, R1 = build => spinup => main_section => report => archive => teardown).
   • Removes some basic limitations allowing for more advanced use cases.
   • Makes it easier to turn off selected sections to support different run modes (e.g, skip the build graph section for production use).
   • Only one graph section can run at a time, which does limit the effectiveness of their use. E.g, in the above example, it would probs be better to implement build and spinip as families within the same section to allow them to run in parallel.
3. Bespoke graph sections as branched pipelines
   • A further generalisation of the above which allows for graphs to be parametrised.
   • E.g, R1 = spinup<model> => main_section => report<site> => archive<model> => teardown
   • Again, more flexible, but potentially trickier to support within the runahead limit model.
   • A common pattern we see looks like this: spinup<x, y> => spinup<x+1, y> => main_section.
4. Bespoke graphs as DAGs
   • A further generalisation of the above supporting wider Cylc syntax.
   • E.g,

      R1 = """
           build => spinup & install_remote => main_section
           install_cold => install_remote => main_section
           main_section => report => archive => teardown
      """

Cycling model:

1. Singluar
   • The approach as currently implemented.
   • The bespoke graph sections are strictly R1 graphs.
   • The ICP, FCP, cycling mode, etc are all global and only really apply to the main cycling section.
2. Freeform
   • Generalisation of the above which makes cycling specific to the graph section.
   • This allows us to combine different cycling regimes in the same workflow.
   • E.g, we might want to use integer cycling in the spinup section and datatime cycling in the main section.
   • Or we might want to have multiple main sections.
   • This cannot be implemented within a runahead model?
   • Incompatible with graph section option (4) above as different cycling regimes cannot coexist at the same time, least not without a substantial rethink of Cylc's implementation.

Interfaces:

• How should we expose these graph sections to the CLI / GUI?
• I haven't thought this through yet.
• What happens if you try to trigger, set or otherwise intervene with a task outside of the graph section the scheduler is currently operating on.
• Etc.

[hjoliver]

@dwsutherland - couple of initial responses:

All tasks will use the same task pools (runahead & active), and hence can have the same cycling sections (don't bother with different cycling modes at the moment).

Yes I was also considering that yesterday, as currently on this PR branch the special no-cycling points preclude (obviously) cycling, but they also require hacking any cycle point comparison code (e.g. for runahead limiting).

Sections are prioritised (in some way), so that if higher priority sections have active tasks the lower aren't released from runahead or not actioned.

The runahead limit itself would almost do the job, if we have the same kind of cycle points in each section (I mean for when startup tasks get retriggered mid-run; when run "naturally" the order is absolute)... but perhaps not quite (if the main graph front is not beyond the runahead limit with respect to the startup tasks) but users could just pause the workflow before retriggering startup tasks, to handle that.

Integer cycling in the startup graph of a datetime cycilng workflow should probably be supported, unfortunately. We might get away with a short datetime cycle (e.g. minutes) that the jobs just ignore the value of - but that's kinda nasty. f

Otherwise we'll need to think through the implications of a mixed datetime/integer task pool, if that's even possible. It might be better to keep the sections separate - if you trigger a startup graph again, the main graph will be suspended until the startup one finishes again.

[hjoliver]

@oliver-sanders -

I'm not sure I'm following on the "priority" options above? Is that an alternative way of specifying the order in which the graph sections run, or a way of controlling the order if multiple graph sections were permitted to run in parallel?

I think it's the latter, as one important use case for this is retriggering deployment tasks (in the startup graph) mid-run.

So for manual triggering purposes I guess we have to be able to:

• manage tasks from multiple separate graphs at once in the same task pool
• or manage multiple separate task pools at once
• or suspend the current task pool, replace it with another, then restore it again after

The same-task-pool model works fine on this branch, but only because the startup and shutdown graphs are non-cycling, and I have to carefully exclude those special cycle points from comparison operations and the like (which is not great).

How should we expose these graph sections to the CLI / GUI?

Reminded me, that's partly why I chose the special cycle point values "startup" and "shutdown" - it works with our existing UIs. It won't be so easy if we allow cycling there.

[dwsutherland]

I'm not sure I'm following on the "priority" options above? Is that an alternative way of specifying the order in which the graph sections run, or a way of controlling the order if multiple graph sections were permitted to run in parallel?

Yes, both really, this was just a way of implementing the section run order, i.e.:

[scheduling]
    R1 = "deploy => graph => final"
    [[deploy]]
        R1 = "a => b"
    [[graph]]
        R1 = "foo => bar"
    [[final]]
        R1/$ = "x => y"

(the under the hood priority would be 1, 2, 3 in this case)

I'd support bespoke graph section, and freeform cycling..
However, as a first implementation we just need to agree on the config structure/syntax..
Then future improvements like local cycling modes and ICP/FCP can be implemented on top.. i.e.

[scheduling]
    R1 = "deploy => spinup => graph => final"
    [[deploy]]
        R1 = "a => b"
    [[spinup]]
        cycling mode = integer
        initial cycle point = 1
        final cycle point = {{ SPINUP_LENGTH }}
        R1 = spinup<model>_cold => spinup<model>
        P1 = spinup<model> => spinup<model + 1>
    [[graph]]
        R1 = "foo => bar"
    [[final]]
        R1/$ = "x => y"

So the rest of the features will be mostly backend implementation in follow-on PRs..

The implementation I proposed (using existing pool) will allow us to get underway quickly, because we can then re-architect under the hood are release features piece wise after.

I think it's the latter, as one important use case for this is retriggering code deployment tasks (in the startup graph) mid-run.

Exactly, it's really a requirement that we're able to rerun these sections and have downstream/lower-priority section essentially freeze while it runs.

The runahead limit itself would almost do the job

Well we could just use the queuing system, so any actively waiting tasks in lower priority sections get queued, even if they are released from runahead, while higher priority sections have active tasks.
(over and above the existing queue)

Also, I like the fact we get to use cycling in the bespoke section with the inherited cycling mode and ICP/FCP.. And we get this for free by using the same pool (but restricting, by validation, the mixing of same named tasks between sections).

This might mean we need different runahead pool/internals for each graph section, otherwise a queued task in a lower priority section might runahead limit higher priority sections.

Must stress, it's important that the same named tasks can not be permitted to be in multiple graph sections..

How should we expose these graph sections to the CLI / GUI?

Perhaps - with only one graph section:

1
    . . .  
2
    . . .

with multiple:

startup
    1
        . . .
run
    1
        . . .  
    2
        . . .
final
    2
        . . .

(IMO - the visuals aren't as high a priority, can stay as is for now (cycle roots))

[oliver-sanders]

It would be a good idea to gather the use cases the feature is intended to solve.

E.g, what is difficult about ICP/FCP dependence under the current arrangement? What patterns of graphs can we identify? Etc.

[hjoliver]

Created a new issue for the general discussion - this old PR isn't really the right place for it: #7020

[hjoliver]

Converted to DRAFT - no point in continuing with this PR while we're revisiting the requirements.