-
Notifications
You must be signed in to change notification settings - Fork 4
Description
I have inputs arguments that needs to be passed through 3 identical modules. The only difference between the modules it the job iterations/labels. Since the limit of jobs submission on Cranium is 3000 and we want to run 9000 jobs per experiment, each module runs 3000 jobs, with labels 1-3000, 3001-6000 and 6001-9000 for each module. I want the module 1-3000 to pass its input arguments (i.e., specs, R script, dataset to process, workspace directory,...) as inputs arguments for the 3001-6000 module, and so on.
Ultimately, I want to pass these specs across different experiments.
The final workflow will have a unique module for initialization, where we specify: i) the R script to use, ii) the dataset to process, iii) the arguments as specs for each experiment, iv) the workspace directory, v) the jobs labels to run, and vi) flags for the shell script. The only module-specific parameter will be the experiment label.