Rerun workflows

[vinjana]

What

Many workflow engines have a rerun feature that allows the user to restart the workflow at a safe stage after it was stopped ungracefully.

Why

Workflows may terminally end in many situations, including unrecoverable error in the infrastructure (e.g. batch processing system, storage). Some of these conditions may be covered by automatic restarts, like some infrastructure problems, others, however, are best be handled by cancellation or automatic termination of workflows, such as workflow bugs, inability to deal with unexpected inputs, etc. One valid situation that we repeatedly observe is a workflow failing because some odd input data requires 10 times the memory or takes 10 times longer than a "normal" sample (e.g. normal cancer sample vs. chromothrypsis; some optimization algorithms with occasionally bad convergence characteristics)

In principle, workflow runs may be started from the beginning, and indeed some workflow engines do even not provide a restart feature. (Restarting from the beginning, i.e. with a new RunRequest, in any case is the safer solution if one does not trust an engine's rerun feature.)

However, restarting workflows at a later stage is still interesting, because it may safe resources (CPU, IO, energy/CO2), and time -- which is important in some application areas, like the timely processing of patient data.

How

The request would just start exactly the same workflow again, i.e., use the same parameters. There may also be use-cases where a parameter may differ for a rerun. For instance, it may be necessary to increase the memory and time resources for an odd sample.

The rerun may be applied to a workflow in any terminal-state, such as CANCELED, EXECUTOR_ERROR, SYSTEM_ERROR, but obviously not in a running state, like RUNNING, QUEUED, etc.

The correct rerunning itself should be left to the workflow engine. We should not care whether the engine or even the workflow allow for this rerunning (that's outside the responsibility of a WES, I think).

This could be implemented as a separate RerunRequest route (e.g. with the run-id in the route). The whole feature should be optional, because it does not make sense to require the implementation of a RerunRequest API endpoint for a workflow engine that cannot do reruns.

[uniqueg]

Very reasonable use case, of course.

But is there anything stopping you from implementing this functionality without there being a specific endpoint to trigger it? Just like, e.g., Snakemake does it? From a WES client's point of view, surely it's trivial to implement re-run functionality, either with identical or updated parameters ("just re-trigger that previous run as is or give me a copy of the form so that I can modify it first"). And a WES can surely (try to) identify re-runs or partial re-runs and reuse cached artifacts, from containers to input data to intermediate data to run checkpoints, regardless of whether the client specifically asks for it or not. And this could be restricted to a particular user's previous runs. Or not. Possibly dependening on the tradeoff between security/privacy considerations and costs/performance. Or we could add a boolean field for do_not_reuse_artifacts for cases where we want to force a fresh run.

A couple of other thoughts:

• OpenAPI 3 does not formally provide for optional paths or methods. On the other hand, the choice to implement any endpoint is entirely optional, of course. Maybe the question is, then, at which point do we start/stop calling an implementation a WES implementation? Is GET /tasks essential? Or how about a read-only WES?
• Actually, any WES has the ability to re-run a workflow, independent of the engine. Only the ability to reuse artifacts of a previous run requires support by the engine (and the system, which needs to be able and willing to preserve such artifacts). So if we decide to go for a specific route for this use case, in principle we do not really need to make it optional.
• The functionality could also be provided through an (optional) parameter reuse_from in the form, which takes the ID of a previous run. The engine could then try to use from that workflow run whatever it can (provided the user has access, else 403).

[vsmalladi]

I like the reuse_from as a field that can provide a previous ID for re-run paramenters.

[uniqueg]

I like the reuse_from as a field that can provide a previous ID for re-run paramenters.

Thinking about it again, I'm not sure how it would work though. And isn't there a danger that it may raise expectations on a given WES to support caching for re-runs? I mean, if it's just about getting the form parameters, a client could also look at Log to get them.

What I do think would be good to have to address this issue:

• A caching parameter in a capabilities section in the service info that a server instance could use to broadcast whether it is able to reuse old runs or not. This could either be boolean, an enum of pre-defined values (e.g., none, user, global to define that an implemention may be able/willing to check for cached artifacts only in the user or in the global space) or even a free enum where implementations would then provide descriptions of what the caching values mean (and which MUST at the very least include none to allow users to disable caching).
• A use_cache parameter in the RunRequest form to allow the user to pick one of the supported caching strategies or explicitly disable the use of caching.
• Possibly another parameter, e.g., cache_run to enable the client to indicate whether it is okay for the server to cache artifacts of the run to be used by, e.g., the same user in the future, or all users of that WES.

With these fields, instances could broadcast hints as to their ability to use caches for re-runs (and their limitations). It would also enable the client to opt out of caching for security reasons or in case re-runs because of broken caches etc.

And I think @vinjana's use case could be solved by extending the documentation for implementers as to how we imagine that caches could be used via the regular POST /runs request when a sufficiently similar request comes in.

[vinjana]

@uniqueg Good points. Sorry, for getting back to this after such a long time, but the issue is still open 😁.

But is there anything stopping you from implementing this functionality without there being a specific endpoint to trigger it? Just like, e.g., Snakemake does it?
...[other post] ... could be solved by extending the documentation for implementers as to how we imagine that caches could be used via the regular POST /runs request when a sufficiently similar request comes in.

I find this approach has some weaknesses

1. There is a lot of logic necessary, to say whether two workflow runs are really the same, or not. Is that something that should be in the WES?
2. Even worse, there may be non-deterministic workflows (yes, s.b. forgot to make the random seed configurable -- not an unrealistic use case, in my experience). For these, multiple parallel runs of the same workflow, with the same parameters may even be useful in some situations.

And isn't there a danger that it may raise expectations on a given WES to support caching for re-runs?

For that reason, I said this should be an "optional" route.

The main reason to use a dedicated route would be that one accesses an existing resource -- namely the existing, possibly incomplete output of a previous workflow run. Of course, this can be done with the same parameters, but may be with slightly changed parameters (as outlined before, e.g., memory).

Caching

I think, caching really is an important feature (dependent on the infrastructure that one thinks uses), but caching is a different feature than resuming.

I assume with caching you mean that a WES also manages pulling/pushing of input/output data from/to some storage system, DRS, or whatever, to/from some fast local storage used for processing, right?

This caching is not even necessary for the workflow resumption: Sure, resuming an incomplete workflow only makes sense, if incomplete output data is still present, and not cached out. But why would the incomplete output data be moved away, if they are incomplete? The WES can reasonably assume the data is incomplete, if the workflow exits ungracefully. Also, although it could be inefficient to re-download input data, but that does not imply that a workflows resumption will not be able to just run with re-downloaded input data (dependent on what criteria the engine uses to trigger resumption, e.g., file-existence vs. file-timestamps).

So I really think resumption in they way I mean, and caching in the way described above -- as mechanism to reduce, e.g., network traffic or workflow execution latency -- are to entirely different mechanisms with different purposes.

And, if you talk about caching for other users, it gets more complicated. That feature, I think, belongs rather in the authorization management domain. E.g., a reasonable assumption may be that all data derived from input data to which a user has access and transformed with workflows which the users is allowed to use should also be accessible to that user. It actually even does not matter, whether it is cached or not.

[uniqueg]

All makes sense to me. The RESTy day for this would surely be a POST /runs/{id}:resume operation for resuming a resumable run (whatever that is should be left up to the engine). Possibly PUT /runs/{id} or PATCH /runs/{id} could also be bastardized.

How would that look like in your opinion. Do you wanna move this forward by creating a PR, @vinjana ? I'm curious what payload you are suggesting.

The optional issue is another story. Good use case for moving forward with extensions.