Skip to content

Commit 5bd9300

Browse files
adamrtalbotFriederikeHanssenmaxulyssevdauweramribeirodantas
authored
Intermediate training draft (#644)
* Add working with files side quest Adds a side quest for working with files, vaguely based on the Metadata propagation section of the advanced training. Story: - Create `file` object from string - Look at `file` object attributes - Extract sample metadata from filename - Use `Channel.fromPath` to create a channel of files - Extract sample metadata from filename within map operator - Use `Channel.fromFilePairs` to create a channel of file pairs - Use `publishDir` to save results Should help introduce the concept of handling files better. Problems: - Doesn't ram home the "always use files as inputs!!!" message. We could do that in the final section on processes? * Metadata Training (#611) * rough plan for metadata training * solution for training * update solution * first half of the text * add intro and second half of training * add to side bar * linting fixes * linting fixes * add sumary and key concepts * linting. again. * fix line highlights * fix line highlights, formatting * fix indents * flesh out some of the explanations * fix text * Add working with files side quest (#601) * Add working with files side quest Adds a side quest for working with files, vaguely based on the Metadata propagation section of the advanced training. Story: - Create `file` object from string - Look at `file` object attributes - Extract sample metadata from filename - Use `Channel.fromPath` to create a channel of files - Extract sample metadata from filename within map operator - Use `Channel.fromFilePairs` to create a channel of file pairs - Use `publishDir` to save results Should help introduce the concept of handling files better. Problems: - Doesn't ram home the "always use files as inputs!!!" message. We could do that in the final section on processes? * fixups and make paths relative to codespaces * Add real FASTQ data * Reduce code blocks for clarity * Typo in splitting and grouping path * Remove splitting and grouping (wrong branch) * Clarify the difference between strings and files in Nextflow * Clarify files in Nextflow * Correct summary sentence to reflect code, it was copy+pasted wrong * Update docs/side_quests/working_with_files.md Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> * Clarify why we want to flatten tuples * Simplify the map operation part * Remove reference to Groovy methods * Fix indentation * Conver data to metamap to introduce metamaps early * Refine summary points * Use markdown numbering * Add link to file properties documentation * Explain multiple assignment better * Update docs/side_quests/working_with_files.md Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io> * Add Channel.fromFilePairs docs as link to fromFilePairs section * Added glob explanation as a note * Added glob explanation as a note * Use bullet points instead of numbers for key concepts of working with files * working with files use before-after syntax correctly * working with files add highlighting * working with files highlighting fixup * Fixups: Respond to review comments --------- Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io> * Intermediate training Reorganisation (Meta maps, working with files, splitting & grouping) (#636) * Bump NXF_VER to latest stable (25.04.3) (#618) * Bump NXF_VER to latest stable (25.04.3) * Set devcontainer image from base:ubuntu to base:dev-ubuntu-24.04 --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Hello Nextflow: Transcripts (#615) * Add transcripts for Hello Nextflow videos * Check headings: ignore transcript files --------- Co-authored-by: Geraldine Van der Auwera <geraldinevdauwera@gmail.com> * Small fixes to Hello Nextflow outputs (#620) * Update outputs * Update script to match training content --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Minor link fixes (#621) * Minor link fixes * Delete nf-develop mention * Update French translation for the Home and Help pages (#622) * Fix buildx error when image contains attestation manifest (#623) Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Address #491 - Selecting a Nextflow version and Environment variables (#627) * Address #491 - Selecting a Nextflow version and Environment variables * removing Environment section * Prettify orientation file --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io> * expand intro to go from samplespecific data to the term meta data * rephrase intro * fix wording to romance * remove ternary operator to focus on meta maps, instead highlight merging of maps * remove ternary operator from learnings * move map explanation up * First pass of the Nextflow Run course (#626) Abridged version of Hello Nextflow focused on running rather than developing. - Run basic operations (Hello World level) - Run pipelines (channels for multiple inputs, multi-step example, modules, containers) - Configuration * remove filtering from the metamap training. * remove join, filter, fix language, line numbers * fix numbering * add samplesheet parsing into starter file * update solution with simplified training * abbreviate title * add trailing dot to make the linter happy * very minor refactoring, simplifying the quests to how to get to a meta data map * fix linting * closing message * ad solution * use meta maps * Update docs/side_quests/working_with_files.md Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> * remove duplication from sidebar * add working with files to side quest overview * remove more duplications * remove section 5.2/5.3, add solution * linting * prettier linting * simplify if/else more * formatting * linting * add meta data training as prereq * Adding posthog (#640) * Adding GTM * Add custom GTM integration for MkDocs - Create custom GTM analytics partial to avoid posthog-js dependency issues - Maintain GTM integration while using Material for MkDocs supported approach - Remove empty package-lock.json that was causing build confusion * Updating just posthog * Updating just posthog * Fix linting error --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Add a proper intro to IDE features (#632) * Add ide intro * Correct heading number * Fix numbering * Add a bunch of images * Prettier * Add to index * Crop some images for renamed file * Minor fixups including: - Clarifying ctrl/cmd key usage - Fixing some language - Removing IDE from title * Some more fixups and adjustments based on a run through * fixup numbering * Couple more feedback tweaks * Resove extensions panel opening via icon * typo * Move Cmd admonition, add files icon, tidy up Cntrl/ Cmd * Update syntax showcase image * Fix operators autocomplete image * Fix task autocomplete * Fix config autocomplete * correct link following * Better resolve linking / process inspecting * Misc fixes * Fix script * Text improvement * More minor text fixes * Add source control screenshot * Strip AI coverage in IDE section due to plugin bug * Clarify terminal shortcut * Add note on structure * Fix numbering * Update docs/side_quests/ide_features.md --------- Co-authored-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> --------- Co-authored-by: Geraldine Van der Auwera <geraldinevdauwera@gmail.com> Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io> Co-authored-by: Kristina Gagalova <kristina.gagalova@curtin.edu.au> Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io> Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Co-authored-by: mavi-sqr <marta.vidal@seqera.io> Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io> * docs(side_quests): add remote file usage section to working with files guide - Add comprehensive documentation on using remote files with URIs - Include examples for HTTP, S3, Azure, and Google Cloud Storage protocols - Demonstrate seamless switching between local and remote data - Update summary and key concepts sections with remote file examples - Show how Nextflow automatically handles file staging and caching * refactor(docs): clean up side quests navigation and remove duplicate files - Remove duplicate navigation entries for nf-test.md and nf-core.md - Delete outdated splitting-and-grouping.md file - Reorder side quests navigation for better organization - Fix navigation structure to eliminate redundancy * Apply suggestions from code review Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io> * replace sample with file or data in the main text, use consistent spelling for datasheet and metadata * fix indentation & format solution. Add fix for changed if clause to adhere to language server * add in problem statement * adding @pinin4fjords suggestions on emphasis * add map example * replace attributes/properties with metadata * add publishDir right away * add in reference to splitting and grouping suggested by @pinin4fjords * tweak wording suggested by @pinin4fjords * Update docs/side_quests/metadata.md Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io> * add a follow on sentence on the balance between meta maps and explicit inputs * fix linting * sample -> patient * Promote remote files coverage * Add object type coverage * Do better with strings vs Path objects * link out to path docs * Add example to illustrate file handling in a process * Formatting fix * Update new example * Formatting fixes * Formatting fixes * update title * Correct highlights * Remove duplicate content * Explain debug * fix path * More formatting fixes * More formatting fixes * Cover string class * main.nf -> file_operations.nf * main.nf -> file_operations.nf * Fix link * Continuity fixes * Apply minor suggestions from code review * Link to 'working with metadata' * Fix title * Fix takeaways * Remove lingering 'sample' references * Update learning outcomes * First batch of edits from final review * Minor fixes, and we need pipefail * fix nextflow versions * Remove messy bolds * Declare files inside workflow block * attempt indent fixes * More fixes * Files fixes including summary * Prettier * Initial cleanup of splitting/ grouping * syntax fixes * Convert to file object and link out * Fix csv name and nextflow versions * Intro tweaks * misc fixes * Add map explanation * More SnG tweaks * More SnG tweaks * More SnG tweaks * More SnG tweaks * Fix line nums * Fix line nums * tumour -> tumor * Fix line nums * More SnG tweaks * Misc fixes * Misc fixes * More fixes * Fix up takeaways * AI-assisted prose cleanup * Fix up code block titles * Tweak summary * Smooth transitions * Update solution * Prettier --------- Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io> Co-authored-by: Geraldine Van der Auwera <geraldinevdauwera@gmail.com> Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io> Co-authored-by: Kristina Gagalova <kristina.gagalova@curtin.edu.au> Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io> Co-authored-by: mavi-sqr <marta.vidal@seqera.io> Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>
1 parent 8d87019 commit 5bd9300

40 files changed

+2968
-717
lines changed

‎docs/side_quests/index.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,10 @@ Otherwise, select a side quest from the menu below.
2929
## Menu of Side Quests
3030

3131
- [Introduction to nf-core](./nf-core.md)
32+
- [Sample-specific data](./metadata.md)
33+
- [Splitting and Grouping](./splitting_and_grouping.md)
3234
- [Testing with nf-test](./nf-test.md)
3335
- [Workflows of workflows](./workflows_of_workflows.md)
36+
- [Working with files](./working_with_files.md)
3437

3538
Let us know what other domains and use cases you'd like to see covered here by posting in the [Training section](https://community.seqera.io/c/training/) of the community forum.

‎docs/side_quests/metadata.md

Lines changed: 793 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/side_quests/orientation.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,11 @@ If you run this inside `side-quests`, you should see the following output:
2424

2525
```console title="Directory contents"
2626
.
27+
├── metadata
2728
├── nf-core
2829
├── nf-test
2930
├── solutions
31+
├── splitting_and_grouping
3032
└── workflows_of_workflows
3133
```
3234

‎docs/side_quests/splitting_and_grouping.md

Lines changed: 628 additions & 708 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)