Skip to content

Change resource allocations #115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 17 commits into from
Closed

Change resource allocations #115

wants to merge 17 commits into from

Conversation

@YukWoon
Copy link

@YukWoon YukWoon commented Mar 26, 2025

PR checklist

  • This comment contains a description of changes (with reason).
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nf-test test main.nf.test -profile test,docker).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).

Changed the resource allocations, including CPU, memory and time

YukWoon added 13 commits March 25, 2025 15:49
@YukWoon
Update base.config
cfd3050 
Changing the resource allocation to be more reasonable
@YukWoon
Update test.config
e0c9ddc
@YukWoon
Update base.config
fa90ca0 
Changes the parameters of single and low
@YukWoon
Update main.nf
95975c2
@YukWoon
Update main.nf
8b70893
@YukWoon
Update main.nf
5391189
@YukWoon
Update base.config
44d1d86
@YukWoon
Update test.config
76d9fb0
@YukWoon
Update base.config
1829a10
@YukWoon
Update main.nf
24132d4 
Reducing resource allocation. Changed module to use process_single.
@YukWoon
Update main.nf
20ee4b1 
process_low
@YukWoon
Update test.config
0f08884
@YukWoon
Update base.config
2ac0008 
Even number of CPUs for compatibility
@FranBonath
Copy link
Member

FranBonath commented Mar 26, 2025

@maxulysse the "process_single" etc. labels are not standardized in nf-core, right?

Aratz
Aratz reviewed Mar 26, 2025
View reviewed changes
Copy link
Collaborator

@Aratz Aratz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this! I have two comments for you to look into

modules/nf-core/bowtie2/build/main.nf
@@ -1,6 +1,6 @@
process BOWTIE2_BUILD {
tag "$fasta"
label 'process_high'
Copy link
Collaborator

@Aratz Aratz Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The files in modules/nf-core are automatically fetched from https://github.com/nf-core/modules/ and should not be modified manually.

Instead, you should specify the new label in base.config. See this example from Sarek: https://github.com/nf-core/sarek/blob/master/conf/base.config#L74-L77

conf/base.config
Comment on lines 29 to 33
withLabel:process_single {
cpus = { 1 }
memory = { 6.GB * task.attempt }
time = { 4.h * task.attempt }
cpus = { 2 }
memory = { 2.GB * task.attempt }
time = { 10.m * task.attempt }
}
Copy link
Collaborator

@Aratz Aratz Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not exactly sure about which modules use process_single, but the name suggest these modules should run on only one core no? If you modify this label, there is a risk that future modules will be overallocated.

Wouldn't it be better to set the label to process_low for bowtie, fastqc and fastq_screen?

Copy link
Author

@YukWoon YukWoon Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall change that now along with reverting the main.nf files to original

@github-actions
Copy link

github-actions bot commented Mar 26, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit f9bedc6

+| ✅ 193 tests passed       |+
#| ❔   2 tests were ignored |#
!| ❗  16 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in nextflow.config: Specify your pipeline's command line flags
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in README.md: TODO nf-core:
  • pipeline_todos - TODO string in README.md: Include a figure that guides the user through the major workflow steps. Many nf-core
  • pipeline_todos - TODO string in README.md: Fill in short bullet-pointed list of the default steps in the pipeline
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in main.nf: Remove this line if you don't need a FASTA file
  • pipeline_todos - TODO string in ro-crate-metadata.json: "description": "

    \n \n <source media="(prefers-color-scheme: dark)" srcset="docs/images/nf-core-seqinspector_logo_dark.png">\n <img alt="nf-core/seqinspector" src="docs/images/nf-core-seqinspector_logo_light.png">\n \n

    \n\nGitHub Actions CI Status\nGitHub Actions Linting StatusAWS CICite with Zenodo\nnf-test\n\nNextflow\nrun with conda\nrun with docker\nrun with singularity\nLaunch on Seqera Platform\n\nGet help on SlackFollow on TwitterFollow on MastodonWatch on YouTube\n\n## Introduction\n\nnf-core/seqinspector is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)\n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run nf-core/seqinspector \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\nFor more details and further functionality, please refer to the usage documentation and the parameter documentation.\n\n## Pipeline output\n\nTo see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page.\nFor more details about the output files and reports, please refer to the\noutput documentation.\n\n## Credits\n\nnf-core/seqinspector was originally written by Adrien Coulier.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\nFor further information or help, don't hesitate to get in touch on the Slack #seqinspector channel (you can join with this invite).\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use nf-core/seqinspector for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX \n\n Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nYou can cite the nf-core publication as follows:\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
  • pipeline_todos - TODO string in test.config: Specify the paths to your test data on nf-core/test-datasets
  • pipeline_todos - TODO string in test.config: Give any required params for the test so that command line flags are not needed
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
  • pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.2.0
  • Run at 2025-03-26 16:32:49

@FranBonath
Copy link
Member

FranBonath commented Nov 6, 2025

The settings should be reviewed again before merging. Seems a bit low on resource allocation.

@FranBonath FranBonath added this to the 1.0.0 milestone Nov 6, 2025
@FranBonath FranBonath mentioned this pull request Nov 7, 2025
Open
@MatthiasZepper
Copy link
Member

MatthiasZepper commented Nov 24, 2025

Thank you for your contribution and looking into this.

We have discussed your suggestions and while they seem reasonable for the small test dataset, we fear that they will be unhelpful for a full dataset. I also prefer to leave the base configs for process_large etc. intact and rather add some with_name selectors for those tools where those values are widely off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Aratz Aratz Aratz left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

1.0.0

Development

Successfully merging this pull request may close these issues.

5 participants

@YukWoon @FranBonath @MatthiasZepper @Aratz