Multiple isolated graphs.

[hjoliver]

A clean issue for a problem with a long and convoluted history (e.g. see #4903, #4912, #5036, and most "recently" #5090).

background

Cylc famously has no barrier between cycles. This is usually a very good thing, but there are certain situations where a hard barrier would be convenient because it stands in for ALL dependence on tasks prior to the barrier.

I'll illustrate for a simple workflow with:

• startup tasks that everything else depends on (e.g. to deploy stuff), and
• shutdown tasks that should not run until everything else has finished

how we currently achieve it (1/2)

⬆️ The blue bits represent perpetual dependence on the initial tasks, to ensure that nothing runs before they finish. This is reasonably intuitive and reflects real dependence - not a workaround - but it causes problems:

• it makes an unbelievable mess of graph visualizations for real workflows
• it makes retriggering the startup graph difficult or confusing (will it "flow on" again?)
• it can cause performance issues in the cycling computations, far from the ICP (is this still true?)

The red bits represent dependence on special dummy tasks that exist purely to ensure that final-cycle tasks wait on everything else - i.e. it's a workaround.

• it is not possible to do blue thing for the final cycle - that would require foo => bar[$]
• (however, the shutdown task requirement is much less common than startup)

how we currently achieve it (2/2)

⬆️ Pragmatically, using the workaround (red) at both ends is probably better than the proper solution (blue), and the skip task run mode makes it more attractive than it used to be, but:

• it can be unpleasant for some workflows (e.g. with lots of parentless tasks at the top of a cycle)
• evidence at ESNZ shows users don't naturally think of it (all of our workflows have the blue bits)
• (and it is still a workaround that should not be necessary)

a better solution

⬆️ For our simple example, it would be better to have 3 separate graphs that each run to completion before the next one starts, thus absolving us of the need to handle nasty perpetual dependencies.

High-level considerations

The separate graphs should naturally run in sequence (the whole point of this is to put a barrier between bits of graph in certain situations where that is actually helpful) but we need to be able to re-trigger tasks from earlier graphs, e.g. to redeploy code or data mid-run.

Also, generally we should - #5090 (comment)

• support more than 3 separate graphs, and
• support cycling within each graph (and probably different cycling types) - e.g. for model spinup

We need to consider future directions and be compatible with that vision (so far as is possible - #5090 (comment) - #5090 (comment)

Do we need to support parallel running of the different graphs, after manual retriggering? - probably NOT:

• if I retrigger tasks from earlier graphs that probably requires pausing or suspending the current graph
  • e.g. if I want to redeploy code that many later tasks depend on, I probably should not continue to run those later tasks during the redeployment process
• if not, then it seems a mixed task pool is not necessary
  • this removes the gnarly problem of comparing different kinds of cycle point within the task pool
  • instead, temporarily swap out the task pool and restore it after the retriggered other-graph tasks have completed?
  • have to wait for live tasks to complete first though?

implementation ideas and considerations

Isolate the first and last cycles by dynamically manipulating the runahead limit?

• ❌ it works but it restricts multi-cycling early in the main graph, because the isolated cycle is also the first main cycle
• (REJECTED)

Special startup and shutdown cycle points? - #5090

• ✅ works with existing UI, e.g. cylc trigger //startup/foo
• ❌ does not support cycling in the startup and shutdown graphs
• 🥹 mixed task pool works fine but requires hacking cycle computations (runahead limit and more) to handle the special points
• (works but probably too restrictive)

UI - how to identify tasks from different graphs (given that special cycle point values are insufficient if we want cycling)?

• the main graph is special, others automatically get a unique task-name prefix?
  • e.g. 1/startup_foo, 2/spinup_bar
  • works with existing UIs
  • clearly visible in GUI
• anything else would require new special options like --graph=spinup and have a visibility problem (how do I know which graph this task in the GUI belongs to?)?

[oliver-sanders]

Motivation for this feature

Comments on the points raised in the OP:

but it causes problems:

It makes an unbelievable mess of graph visualizations for real workflows

I'm not sure if this is causing issues at our end, haven't heard anything from users at least, perhaps you have more ICP dependencies in your workflows than we typically see in our workflows? Or just more resistance to dummy tasks?

I wonder if there could be other solutions to this problem, say something like the old [visualisation] section for configuring the graph?

it makes retriggering the startup graph difficult or confusing (will it "flow on" again?)

I'm not sure about this one, if "flow on" behaviour is confusing for R1 tasks, then it is confusing for all tasks. Now that the --flow option isn't needed for re-running graphs, I'm not sure how relevant this is (users are just confused by --flow in general IMO).

It can cause performance issues in the cycling computations, far from the ICP (is this still true?)

I don't think this is an issue, but haven't tested.

---

Here are the possible motivations I can think of:

1. Can make a mess of graphs where large numbers of ICP tasks are referenced in subsequent recurrences and dummy tasks are not used to collate these dependencies (accepting your point from above).
   • Dummy tasks are an awkward workaround.
   • Graph configuration might be another possible workaround.
2. ICP dependence has to be added everywhere, in large workflows it might be possible to miss it in one recurrence section by accident.
   • No workaround at present, just "get it right".
   • Task/graph-level configuration might be an alternative workaround for this (i.e, some form of auto-dependency configuration along the lines of "sequential tasks".
3. Dependencies on ICP tasks are easy (e.g, build[^] => run), however, dependencies for FCP tasks are tricky (e.g, run => clean[$] is not legal).
   • No real workaround at present, this just has to be brute-forced.
   • Supporting inter-cycle offsets on the RHS might be an alternative solution to this.

---

Here's a couple of use cases which are awkward at the moment.

1. Spinup graphs.

   A patten we see in a couple of workflows, something along the lines of this:

    [scheduling]
        [[graph]]
            R1 = """
                spinup<x, y> => spinup<x, y+1>
                spinup<x, y> => configure
            """
            P1Y = """
                configure[^] => model
                model[-P1Y] => model
            """

   The R1 section (and this is a gross simplification) is more akin to integer cycling, the x parameter can get quite large. Used to cause problems, but since Cylc 8 (SoD specifically), this hasn't caused issues as we're no longer loading the entire cycle's worth of tasks.

2. Result collation

   A more general example of FCP dependence issues. This is a common pattern in climate graphs where there is often no inter-cycle dependence. Cycles may run in a highly arbitrary order as each cycle starts with an archive retrieval task (like a random sleep). The runahead limit is typically cranked up high in order to get as many archive retrieval requests in as possible and make as much progress through the workflow as possible at the rate the retrieval tasks kick out data. The problem is collating the results.

   [scheduling]
      cycling mode = 360  # solution only works with artificial calendars!
       [[graph]]
           P1D = """
               retrieve => model => process
           """
           P1M = """
               process & process[-P1D] & process[-P2D] ... & process[-P30D] => collate_month
           """
           P1Y = """
               collate_month & collate_month[-P1M] & collate_month[-P2M] ... & collate_month[-P11M] => collate_year
           """
          P10Y = """
               collate_month ... => collate_decade
          """
          R1/$ = """
               collate_decade ... => report
          """

   May or may not be a case for this sort of feature?

   Possibly solvable with relative cycle syntax and inter-cycles on the RHS, e.g: process => collate_month[+R1]?

[hjoliver]

It makes an unbelievable mess of graph visualizations for real workflows

A real-life example, one of our many operational workflows that start with a bunch of "deploy" tasks in the first cycle (in fact, a relatively uncomplicated one!):

1. with initial dependencies removed - clear structure

2. with initial dependencies restored

3. zoomed in bit - it IS an unbelievable mess

perhaps you have more ICP dependencies in your workflows than we typically see in our workflows?
Or just more resistance to dummy tasks?

I think both might be true. I think our operational system is more diverse and modular, and all the workflows have initial deployment tasks that result in this problem, and all (or most) research workflows are similar (most are focused on evolving the operational ones).

I agree (and I more or less said so above) that the dummy task solution is preferable to this BUT it's not particularly intuitive - users don't think of it themselves - whereas perpetual dependence on initial tasks is reasonably obvious (IF all you have to play with is one main cycling graph).

NOTE some of those ICP dependencies are probably superfluous, due to triggering from all members of family that actually has internal dependencies - but regardless, that's what users do - they typically don't think too hard about the graph structure so long as it works - and isolated graphs would relieve that cognitive burden...

[hjoliver]

it makes retriggering the startup graph difficult or confusing (will it "flow on" again?)

I'm not sure about this one, if "flow on" behaviour is confusing for R1 tasks, then it is confusing for all tasks. Now that the --flow option isn't needed for re-running graphs, I'm not sure how relevant this is (users are just confused by --flow in general IMO).

I'm not entirely sure about this one anymore. I have had a number of complaints (including from @dwsutherland - do you want to comment David?) that it is difficult, but I have not had a chance to try to replicate.

To the extent that flow-on is difficult to either understand or manage (if it still is) retriggering ICP "deployment" tasks is kind of a worst case scenario because:

• it has to be done reasonably often (whenever code needs to be redeployed into a running system)
  • and by operational operators, rather than the workflow owners/experts
• it is many many cycles back in the past, which is a problem if unwanted flow-on does occur (if)
• all tasks, including otherwise parentless ones, depend on these

It can cause performance issues in the cycling computations, far from the ICP (is this still true?)

I don't think this is an issue, but haven't tested.

It certainly used to be an issue, but it was quite some time ago so maybe we fixed it somehow, not sure.

I wonder if there could be other solutions to this problem, say something like the old [visualisation] section for configuring the graph?

Maybe, but it strikes me that isolated graphs are the obvious solution and a nice fit for the typical mental model: I want to do a bunch of stuff THEN start the main cycling graph (and I'll likely need to retrigger the early stuff again mid-run).

Perpetual ICP dependence makes sense IF all you have to play with is the main cycling graph, but it would be easier (conceptually too) not have to deal with that.

[hjoliver]

Here are the possible motivations I can think of:

Yes I think I entirely agree with your motivations. (Which mostly overlap with or extend mine a bit).

[dwsutherland]

Special startup and shutdown cycle points?

Another problem with this approach is you cannot use the cycle point startup to know what date data to source of deploy like ICP (or some offset from the ICP).

Here are the possible motivations I can think of:

Yes these are all good, and another of couple:

• You can loose your ICP prerequisite satisfaction if you happen to change your ICP
• If you startup your workflow with start tasks off the ICP (because you might only want to run select startup), you need to no-flow run startup tasks so it doesn't spawn the gap in, i.e.

 [scheduling]
     [[graph]]
         R1 = """
             deploy => configure
             spinup<x, y> => spinup<x, y+1>
             spinup<x, y> => configure
         """
         P1Y = """
             configure[^] => model
             model[-P1Y] => model
         """

If the ICP is 1973, but you do cylc play --pause --start-task=1999/model .. Followed by cylc trigger --flow=none wflow//1973/deploy and then cylc trigger --flow=none wflow//1973/deploy
(because perhaps you want to avoid spin up, and don't want to flow into the )
Then dependencies are ignored prior to the start tasks..

I'm not sure about this one, if "flow on" behaviour is confusing

I'm not entirely sure about this one anymore. I have had a number of complaints

However if you restart the workflow the dependency configure[^] will become unsatisfied.. And then if you set/run the corresponding ICP task, model will spawn into the 1973-1998 gap..
In fact if you just forget to --flow=none then you'll have this problem when re-running startup tasks..

Even though this might be a bug in losing the ignore-dependencies-before-start-task feature on restart (and reload?), this whole scenario/complication can just be avoided by having a separate spinup/startup graph section..

dummy_task won't help with this one...

Possibly solvable with relative cycle syntax and inter-cycles on the RHS

Think we talked about extending the ^ syntax to < and > for previous and next instance respectively.

It can cause performance issues in the cycling computations, far from the ICP (is this still true?)

I think it is still and issue, reloads still take an age in some cases (if this is related)

Our modular operation means we have a lot of parentless tasks waiting on xtriggers, and it doesn't always make sense to tie them all to dummy tasks or have some sort of pseudo dependency between cycles just to avoid running before/while startup tasks..

And we haven't even talked about tying FCP/shutdown tasks to the completion of everything that went before..

Separate prioritised graph sections will introduce a massive and welcome simplification, and a feature that can't truly be replicated in any other way...

And on the implementation front something like this, and as mentioned in Element, the validation enforcement of making sure graph sections don't contain the same named task (to avoid inter section deps and ensure sections aren't rerun at the same time) would be a good start..

[oliver-sanders]

You can loose your dependence on ICP tasks if you happen to change your ICP

I don't understand this one, do you have an example?

If ICP deps are behaving differently to regular inter-cycle deps, that's a bug.

If you startup your workflow with start tasks off the ICP (because you might only want to run select startup), you need to no-flow run startup tasks so it doesn't spawn the gap in, i.e.

I'm not sure about this one, is that a valid expectation of --flow=none?

[dwsutherland]

I don't understand this one, do you have an example?

Sorry, fixed, ICP prerequisite satisfaction

[oliver-sanders]

Sorry, fixed, ICP prerequisite satisfaction

Hmm, still don't think it should matter whether an ICP dep is used here. If there's a difference, raise a bug report.

[dwsutherland]

Point is we wouldn't have to care about the ICP dep as much didn't use it in most cases, even if a bug fix is applied, although it might be argued that this is expected behavior

[oliver-sanders]

I don't understand how the presence of the ICP dep is hindering you here?

Note that isolated graphs implementations essentially replace dependency on a task at the ICP with a dependency on the ICP itself, so may or may not actually make a difference here (implementation depending).

[dwsutherland]

I don't understand how the presence of the ICP dep is hindering you here?

Essentially it's a complication that doesn't need to be there, and any resulting behavior/quirks we wouldn't have to deal with.

Note that isolated graphs implementations essentially replace dependency on a task at the ICP with a dependency on the ICP itself

Well it's a dependency on the graph section not being active (which might be ICP or w/e).. It avoids task dependencies, which is the desirable part.