Intermediate training draft by adamrtalbot · Pull Request #644 · nextflow-io/training

adamrtalbot · 2025-08-05T09:09:58Z

The content is based on existing training material, that are reordered in this PR + addressing reviews from the original PRs by @adamrtalbot (#601) and me (#611)

Metadata training:

Removed filtering to focus on meta maps. filtering operator is covered in working with files / splitting & grouping
Removed joining channels, instead files are passed through processes
Fixed typos and language suggestions from original review
Removed ternary operator, replaced with more verbose if/else statement.
Moved language on what splitCSV does and what maps are from splitting & grouping
TODO: How do we want to name this one?

Working with Files

Shorten the journey to using a meta map
Remove readNum when using FilePairs, since this is not defined anymore

Splitting & Grouping

Remove tumor/normal single map early in the training
Removing section 5.2 & 5.3. We are already using meta maps from the beginning, so this part of the training does not fit into the story anymore.
TODO: reconsider where we can fit content on collect() operator and groovy collect in.
TODO: is the ending to abrupt now?

Adds a side quest for working with files, vaguely based on the Metadata propagation section of the advanced training. Story: - Create `file` object from string - Look at `file` object attributes - Extract sample metadata from filename - Use `Channel.fromPath` to create a channel of files - Extract sample metadata from filename within map operator - Use `Channel.fromFilePairs` to create a channel of file pairs - Use `publishDir` to save results Should help introduce the concept of handling files better. Problems: - Doesn't ram home the "always use files as inputs!!!" message. We could do that in the final section on processes?

* rough plan for metadata training * solution for training * update solution * first half of the text * add intro and second half of training * add to side bar * linting fixes * linting fixes * add sumary and key concepts * linting. again. * fix line highlights * fix line highlights, formatting * fix indents * flesh out some of the explanations * fix text

* Add working with files side quest Adds a side quest for working with files, vaguely based on the Metadata propagation section of the advanced training. Story: - Create `file` object from string - Look at `file` object attributes - Extract sample metadata from filename - Use `Channel.fromPath` to create a channel of files - Extract sample metadata from filename within map operator - Use `Channel.fromFilePairs` to create a channel of file pairs - Use `publishDir` to save results Should help introduce the concept of handling files better. Problems: - Doesn't ram home the "always use files as inputs!!!" message. We could do that in the final section on processes? * fixups and make paths relative to codespaces * Add real FASTQ data * Reduce code blocks for clarity * Typo in splitting and grouping path * Remove splitting and grouping (wrong branch) * Clarify the difference between strings and files in Nextflow * Clarify files in Nextflow * Correct summary sentence to reflect code, it was copy+pasted wrong * Update docs/side_quests/working_with_files.md Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> * Clarify why we want to flatten tuples * Simplify the map operation part * Remove reference to Groovy methods * Fix indentation * Conver data to metamap to introduce metamaps early * Refine summary points * Use markdown numbering * Add link to file properties documentation * Explain multiple assignment better * Update docs/side_quests/working_with_files.md Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io> * Add Channel.fromFilePairs docs as link to fromFilePairs section * Added glob explanation as a note * Added glob explanation as a note * Use bullet points instead of numbers for key concepts of working with files * working with files use before-after syntax correctly * working with files add highlighting * working with files highlighting fixup * Fixups: Respond to review comments --------- Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> Co-authored-by: Maxime U Garcia <maxime.garcia@seqera.io>

…splitting & grouping) (#636) * Bump NXF_VER to latest stable (25.04.3) (#618) * Bump NXF_VER to latest stable (25.04.3) * Set devcontainer image from base:ubuntu to base:dev-ubuntu-24.04 --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Hello Nextflow: Transcripts (#615) * Add transcripts for Hello Nextflow videos * Check headings: ignore transcript files --------- Co-authored-by: Geraldine Van der Auwera <geraldinevdauwera@gmail.com> * Small fixes to Hello Nextflow outputs (#620) * Update outputs * Update script to match training content --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Minor link fixes (#621) * Minor link fixes * Delete nf-develop mention * Update French translation for the Home and Help pages (#622) * Fix buildx error when image contains attestation manifest (#623) Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Address #491 - Selecting a Nextflow version and Environment variables (#627) * Address #491 - Selecting a Nextflow version and Environment variables * removing Environment section * Prettify orientation file --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io> * expand intro to go from samplespecific data to the term meta data * rephrase intro * fix wording to romance * remove ternary operator to focus on meta maps, instead highlight merging of maps * remove ternary operator from learnings * move map explanation up * First pass of the Nextflow Run course (#626) Abridged version of Hello Nextflow focused on running rather than developing. - Run basic operations (Hello World level) - Run pipelines (channels for multiple inputs, multi-step example, modules, containers) - Configuration * remove filtering from the metamap training. * remove join, filter, fix language, line numbers * fix numbering * add samplesheet parsing into starter file * update solution with simplified training * abbreviate title * add trailing dot to make the linter happy * very minor refactoring, simplifying the quests to how to get to a meta data map * fix linting * closing message * ad solution * use meta maps * Update docs/side_quests/working_with_files.md Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> * remove duplication from sidebar * add working with files to side quest overview * remove more duplications * remove section 5.2/5.3, add solution * linting * prettier linting * simplify if/else more * formatting * linting * add meta data training as prereq * Adding posthog (#640) * Adding GTM * Add custom GTM integration for MkDocs - Create custom GTM analytics partial to avoid posthog-js dependency issues - Maintain GTM integration while using Material for MkDocs supported approach - Remove empty package-lock.json that was causing build confusion * Updating just posthog * Updating just posthog * Fix linting error --------- Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> * Add a proper intro to IDE features (#632) * Add ide intro * Correct heading number * Fix numbering * Add a bunch of images * Prettier * Add to index * Crop some images for renamed file * Minor fixups including: - Clarifying ctrl/cmd key usage - Fixing some language - Removing IDE from title * Some more fixups and adjustments based on a run through * fixup numbering * Couple more feedback tweaks * Resove extensions panel opening via icon * typo * Move Cmd admonition, add files icon, tidy up Cntrl/ Cmd * Update syntax showcase image * Fix operators autocomplete image * Fix task autocomplete * Fix config autocomplete * correct link following * Better resolve linking / process inspecting * Misc fixes * Fix script * Text improvement * More minor text fixes * Add source control screenshot * Strip AI coverage in IDE section due to plugin bug * Clarify terminal shortcut * Add note on structure * Fix numbering * Update docs/side_quests/ide_features.md --------- Co-authored-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> Co-authored-by: Friederike Hanssen <friederike.hanssen@seqera.io> --------- Co-authored-by: Geraldine Van der Auwera <geraldinevdauwera@gmail.com> Co-authored-by: Marcel Ribeiro-Dantas <marcel@seqera.io> Co-authored-by: Phil Ewels <phil.ewels@seqera.io> Co-authored-by: Marcel Ribeiro-Dantas <mribeirodantas@seqera.io> Co-authored-by: Kristina Gagalova <kristina.gagalova@curtin.edu.au> Co-authored-by: Marcel Ribeiro-Dantas <marcel.ribeirodantas@seqera.io> Co-authored-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Co-authored-by: mavi-sqr <marta.vidal@seqera.io> Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>

netlify · 2025-08-05T09:10:04Z

✅ Deploy Preview for nextflow-training ready!

Name	Link
🔨 Latest commit	`3a9aed2`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-training/deploys/689c78e8308608000880ac8f
😎 Deploy Preview	https://deploy-preview-644--nextflow-training.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

…s guide - Add comprehensive documentation on using remote files with URIs - Include examples for HTTP, S3, Azure, and Google Cloud Storage protocols - Demonstrate seamless switching between local and remote data - Update summary and key concepts sections with remote file examples - Show how Nextflow automatically handles file staging and caching

…files - Remove duplicate navigation entries for nf-test.md and nf-core.md - Delete outdated splitting-and-grouping.md file - Reorder side quests navigation for better organization - Fix navigation structure to eliminate redundancy

pinin4fjords

Ok, here's my review of the metadata one. Main points:

I'd like to excise 'sample' as a thing here
Need to tidy up indentation, they're a bit mad right now
Could we benefit from a section on combining channels using metas?
Let's try not to encourage bad process designe?

docs/side_quests/metadata.md

FriederikeHanssen · 2025-08-06T13:41:21Z

Could we benefit from a section on combining channels using metas?

we had this and then I removed it 🫠 https://github.com/FriederikeHanssen/training/blob/54c9d4c347d3c1db2171a0b55dd3bccbe3e7ccb0/docs/side_quests/metadata.md?plain=1#L323 reason being, that combining and grouping is covered in the splitting_grouping training and I was worried this one would be overloaded with concepts.

pinin4fjords · 2025-08-06T14:04:54Z

Could we benefit from a section on combining channels using metas?

we had this and then I removed it 🫠 https://github.com/FriederikeHanssen/training/blob/54c9d4c347d3c1db2171a0b55dd3bccbe3e7ccb0/docs/side_quests/metadata.md?plain=1#L323 reason being, that combining and grouping is covered in the splitting_grouping training and I was worried this one would be overloaded with concepts.

OK, fair concern, but I think we should at least cover it in outline, and maybe link to that other material. It's not obvious when you start out that joining on maps is even possible, and for me it's one of the main reasons to add meta maps to channels.

pinin4fjords

This is my review for the file handling module. Bigger points:

Remote files up-front
Clarify Path objects, use .class to demonstrate
Don't over-egg things that aren't the main teaching points here (string parsing, metadata handling)

docs/side_quests/working_with_files.md

pinin4fjords

OK, review of splitting/ grouping.

Mostly fine, which makes sense (it was already there), couple of suggestions/ bug fixes

docs/side_quests/splitting_and_grouping.md

Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>

…lling for datasheet and metadata

…dhere to language server

docs/side_quests/metadata.md

Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>

…t inputs

adamrtalbot and others added 6 commits April 11, 2025 16:23

Merge branch 'master' into intermediate_training

a6924d8

Merge branch 'master' into origin_intermediate_training

bb187e3

adamrtalbot added 2 commits August 5, 2025 11:06

pinin4fjords reviewed Aug 6, 2025

View reviewed changes

adamrtalbot marked this pull request as ready for review August 7, 2025 08:40

pinin4fjords reviewed Aug 7, 2025

View reviewed changes

FriederikeHanssen and others added 8 commits August 8, 2025 11:53

Apply suggestions from code review

7d25cc2

Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>

replace sample with file or data in the main text, use consistent spe…

9e27b10

…lling for datasheet and metadata

fix indentation & format solution. Add fix for changed if clause to a…

2032129

…dhere to language server

add in problem statement

045575c

adding @pinin4fjords suggestions on emphasis

8030efb

add map example

7ad45c1

replace attributes/properties with metadata

3359be9

add publishDir right away

51d9373

FriederikeHanssen reviewed Aug 8, 2025

View reviewed changes

docs/side_quests/metadata.md Outdated Show resolved Hide resolved

FriederikeHanssen and others added 6 commits August 8, 2025 16:54

add in reference to splitting and grouping suggested by @pinin4fjords

c7240d3

tweak wording suggested by @pinin4fjords

361df8e

Update docs/side_quests/metadata.md

7362e9b

Co-authored-by: Jonathan Manning <jonathan.manning@seqera.io>

add a follow on sentence on the balance between meta maps and explici…

e65e453

…t inputs

Merge origin/master

716bb96

fix linting

ca82228

pinin4fjords added 26 commits August 12, 2025 18:54

Initial cleanup of splitting/ grouping

50e6a4c

syntax fixes

90a4f78

Convert to file object and link out

9e8ae72

Fix csv name and nextflow versions

934a250

Intro tweaks

0cde0b8

misc fixes

27e830f

Add map explanation

5892d77

More SnG tweaks

8479457

More SnG tweaks

b8fee4d

More SnG tweaks

ec4107f

More SnG tweaks

aad628c

Fix line nums

a29c6d2

Fix line nums

5acdb84

tumour -> tumor

1295f96

Fix line nums

1d53c97

More SnG tweaks

286fcdd

Misc fixes

e73fb44

Misc fixes

40caf82

More fixes

42c1297

Fix up takeaways

7c5c22a

AI-assisted prose cleanup

b6d5464

Fix up code block titles

a20d154

Tweak summary

587efe6

Smooth transitions

c685be2

Update solution

0e16349

Prettier

3a9aed2

pinin4fjords approved these changes Aug 13, 2025

View reviewed changes

FriederikeHanssen approved these changes Aug 13, 2025

View reviewed changes

pinin4fjords merged commit 5bd9300 into master Aug 13, 2025
7 checks passed

pinin4fjords deleted the intermediate_training branch August 13, 2025 11:46

Conversation

adamrtalbot commented Aug 5, 2025 • edited by FriederikeHanssen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-training ready!

Uh oh!

pinin4fjords left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FriederikeHanssen commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pinin4fjords commented Aug 6, 2025

Uh oh!

pinin4fjords left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pinin4fjords left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

adamrtalbot commented Aug 5, 2025 •

edited by FriederikeHanssen

Loading

netlify bot commented Aug 5, 2025 •

edited

Loading

FriederikeHanssen commented Aug 6, 2025 •

edited

Loading