Add working with files side quest

[adamrtalbot]

Adds a side quest for working with files, vaguely based on the Metadata propagation section of the advanced training.

Story:

• Create file object from string
• Look at file object attributes
• Extract sample metadata from filename
• Use Channel.fromPath to create a channel of files
• Extract sample metadata from filename within map operator
• Use Channel.fromFilePairs to create a channel of file pairs
• Use publishDir to save results

Should help introduce the concept of handling files better.

Problems:

• Doesn't ram home the "always use files as inputs!!!" message. We could do that in the final section on processes?

Notes

FASTQ files generated with:

while read name; do
    # Generate a random number between 1 and 10
    random_num=$((1 + RANDOM % 10))
    fq generate -n ${random_num} --read-length 10 data/${name}_R1_001.fastq.gz data/${name}_R2_001.fastq.gz
done < scripts/samples.txt

Where samples.txt:

sampleA_rep1_normal
sampleA_rep1_tumor
sampleA_rep2_normal
sampleA_rep2_tumor
sampleB_rep1_normal
sampleB_rep1_tumor
sampleC_rep1_normal
sampleC_rep1_tumor

[netlify]

✅ Deploy Preview for nextflow-training ready!

Name	Link
🔨 Latest commit	de22980
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-training/deploys/6819fd3073f79600084979da
😎 Deploy Preview	https://deploy-preview-601--nextflow-training.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[FriederikeHanssen]

• needs to be added to the "Side Quests" Side bar & "Menu of side quests"
• IIRC in the other trainings we started to indent the code blocks with the amount that is needed in the code so we don't have to do this when copying the code blocks.

reviewed until approximately the end of section 3.

[FriederikeHanssen]

I am mostly just thinking out loud here:
I am wondering how this will mix with any metadata module. I think at one point we discussed pulling out the section out of the nf-core module and have it separate. It feels too small to be its own side quest and I think it could fit into here quite well. But I also wouldn't want to convolute this module with extra things about hashmaps and keys.

If the flattening of the meta data is not hugely important, we could replace that with a map and collapse the content?

[FriederikeHanssen]

I suppose this is to use something from the meta info as example on how it can be used in the workflow. Should we do something similar as in the nf-core module or change the nf-core module (we used branch to change the execution path) ? Just to revisit the same concepts again

[adamrtalbot]

That's not a bad idea. We could use filter (from splitting and grouping) instead of branch and use a closure with a filename to unique-ify the published file in config.

[FriederikeHanssen]

The indents of the learnings render weirdly.

[FriederikeHanssen]

Copying discussion from slack here, so it won't get lost:

How about we rename the "Working with Files" chapter to "Working with Input Data" and cover both files and metadata explicitly? To bring things close to nf-core & bridge the gap to grouping/splitting, we could add a section in 3.5 on how to use a map instead of a list.
That would also remove a little complexity in the grouping chapters and focus more on just the data shuffling.

[adamrtalbot]

I am mostly just thinking out loud here: I am wondering how this will mix with any metadata module. I think at one point we discussed pulling out the section out of the nf-core module and have it separate. It feels too small to be its own side quest and I think it could fit into here quite well. But I also wouldn't want to convolute this module with extra things about hashmaps and keys.

If the flattening of the meta data is not hugely important, we could replace that with a map and collapse the content?

I've added a metamap here: 6998ddf

but I'm not sure about it, we might be introducing too much too early?

[adamrtalbot]

@FriederikeHanssen I've addressed most of your comments now - take a second look.

[FriederikeHanssen]

Love it!

• Schedule wise, we should probably run this before nf4-science, since we use the file() objects there already.

[FriederikeHanssen]

I like this addition. The only thing that could potentially be confusing I think is, that in 3.2 (Extracting Metadata from Filenames) we already had a map and a file, but then flatten it

Therefore, it's easier if the input channel is flat instead of the nested structure we have here.

Maybe we need some justification, why the file is not part of the map

[adamrtalbot]

In 3.2 it's a list (array) and a file, do you think that's different enough to be obvious?

[[sampleA, rep1, normal, R1, 001], /workspaces/training/side-quests/working_with_files/data/sampleA_rep1_normal_R1_001.fastq.gz]
[[sampleA, rep1, normal, R2, 001], /workspaces/training/side-quests/working_with_files/data/sampleA_rep1_normal_R2_001.fastq.gz]

[adamrtalbot]

As for why the file isn't in the map - good point. I don't have a great answer for that other than 'because Nextflow says so'...hmmm not sure what to do.

[FriederikeHanssen]

I get the same results, but do we want readNum:R ? I would have expected 1 or 2.

[adamrtalbot]

Ahhhhhhhhhhh fromFilePairs strips the contents of the {}...hmm we might need to rethink this.

[FriederikeHanssen]

I am not sure what we want to cover beyond this tbh. The next step would maybe be adding fields via nf-schema, but I think that would be a training focused on input validation