[FEATURE] Fair and affordable scheduling of pipeline jobs #20
Description
Description
Fair and affordable scheduling of pipeline jobs when they are launched in the XNAT container service
Benefit Hypothesis
Once pipelines are available to the wider AIS/NIF community, if researchers are able to launch as many jobs as they like, then we are likely to hit computational resource limits fairly quickly. The rate at which these limits are hit will depend on the number of users and how the pipelines service is configured at each node. In the best case, when the resource limit is hit, new jobs will just sit in the queue and wait for the running jobs to exit, in the worst case, more resources will be elastically assigned to the pipelines service (depending on cloud configuration) than are budgeted for by the institution.
If new jobs can be scheduled in a fair, affordable yet still timely manner, then it will greatly improve user experience of the service and accelerate the research being performed.
Acceptance Criteria
- Every AIS node is able to control the scheduling of jobs in a manner that suits their available resources