Skip to content

Commit 7ef3682

Browse files
kenibrewerkenibrewer
committed
🚧 wip(hello_seqera): initial outline of content
1 parent 6fc59eb commit 7ef3682

7 files changed

+150
-30
lines changed

‎docs/hello_nextflow/10_hello_seqera.md

Lines changed: 11 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: Hello Seqera
2+
title: "Part 9: Hello Seqera"
33
description: Get started with Seqera Platform
44
---
55

@@ -16,9 +16,17 @@ You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform
1616

1717
You can use Seqera Platform via either the **CLI**, through the **online GUI** or through the **API**.
1818

19-
## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI
19+
--8<-- "docs/hello_nextflow/seqera/01_run_with_cli.md"
2020

21-
--8<-- "./seqera/01_run_with_cli.md"
21+
--8<-- "docs/hello_nextflow/seqera/02_run_with_launchpad.md"
22+
23+
--8<-- "docs/hello_nextflow/seqera/03_add_compute_environment.md"
24+
25+
--8<-- "docs/hello_nextflow/seqera/04_setup_launchpad.md"
26+
27+
!!! Warning WORK-IN-PROGRESS
28+
29+
Everything below this note is content that was copied from the original guide and needs to be moved or rewritten into the above subpages.
2230

2331
## Online GUI
2432

@@ -28,27 +36,6 @@ To run using the GUI, there are three main steps:
2836
2. Create and configure a new [compute environment](https://docs.seqera.io/platform/24.1/compute-envs/overview).
2937
3. Start [launching pipelines](https://docs.seqera.io/platform/24.1/launch/launchpad#launchpad).
3038

31-
### Configuring your compute environment
32-
33-
Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run.
34-
35-
It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures.
36-
37-
![Compute environments](img/compute_env_platforms.png)
38-
39-
Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below.
40-
41-
!!! tip "The following guides describe how to configure each of these compute environments."
42-
43-
* [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch)
44-
* [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch)
45-
* [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch)
46-
* [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences)
47-
* [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc)
48-
* [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks)
49-
* [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke)
50-
* [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s)
51-
5239
### Selecting a default compute environment
5340

5441
If you have more than one **Compute Environment**, you can select which one will be used by default when launching a pipeline.

‎docs/hello_nextflow/seqera/01_run_with_cli.md

Lines changed: 97 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,28 @@
1+
## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI
2+
13
We'll start by using the Nextflow CLI to launch a pipeline and monitor it in Seqera Platform.
24
Start by logging into the [Seqera Platform](https://cloud.seqera.io/).
35

46
!!! info "Nextflow Tower"
5-
Seqera Platform was previously known as Nextflow Tower.
6-
You'll still see references to the previous name in environment variable and cli option names.
7+
8+
Seqera Platform was previously known as Nextflow Tower.
9+
You'll still see references to the previous name in environment variable and cli option names.
710

811
### 1.1. Set up your Seqera Platform token by exporting it to your environment
912

1013
Follow these steps to set up your token:
1114

1215
1. Create a new token by clicking on the **Settings** drop-down menu:
1316

14-
![Create a token](img/usage_create_token.png)
17+
![Create a token](seqera/img/usage_create_token.png)
1518

1619
2. Name your token:
1720

18-
![Name your token](img/usage_name_token.png)
21+
![Name your token](seqera/img/usage_name_token.png)
1922

2023
3. Save your token safely:
2124

22-
![Save token](img/usage_token.png)
25+
![Save token](seqera/img/usage_token.png)
2326

2427
!!! note
2528

@@ -66,6 +69,94 @@ Hello world!
6669
Use ++ctrl+click++ or ++cmd+click++ on the link to open it in your browser.
6770
You'll see the Seqera Platform interface with the job finished and the logs captured.
6871
69-
![Seqera Platform](img/run_with_tower.png)
72+
![Seqera Platform](seqera/img/run_with_tower.png)
7073
7174
You will see and be able to monitor your **Nextflow jobs** in Seqera Platform.
75+
76+
### 1.3. Set up Seqera Platform logging by default
77+
78+
To set up Seqera Platform logging by default, we first want to store our token more securely as a [Nextflow secret](https://www.nextflow.io/docs/latest/secrets.html).
79+
80+
```bash
81+
nextflow secrets set tower_access_token "eyxxxxxxxxxxxxxxxQ1ZTE="
82+
```
83+
84+
You can set up Seqera Platform logging by default by adding the following line to your `nextflow.config` file.
85+
86+
```groovy title="nextflow.config"
87+
tower {
88+
enabled = true
89+
endpoint = "https://api.cloud.seqera.io"
90+
accessToken = secrets.tower_access_token
91+
}
92+
```
93+
94+
Instead of putting this in the `nextflow.config` file of a specific project, you can also put it in the global file located at `$HOME/.nextflow/config`.
95+
96+
```bash
97+
cat <<EOF >> $HOME/.nextflow/config
98+
tower {
99+
enabled = true
100+
endpoint = "https://api.cloud.seqera.io"
101+
accessToken = secrets.tower_access_token
102+
}
103+
EOF
104+
```
105+
106+
Run your Nextflow workflows as usual:
107+
108+
```bash
109+
nextflow run nextflow-io/hello
110+
```
111+
112+
You will see the following output:
113+
114+
```console title="Output"
115+
N E X T F L O W ~ version 24.04.4
116+
117+
Launching `https://github.com/nextflow-io/hello` [fabulous_euclid] DSL2 - revision: afff16a9b4 [master]
118+
119+
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/KYjRktIlOuxrh
120+
executor > local (4)
121+
[71/eaa915] process > sayHello (3) [100%] 4 of 4 ✔
122+
Ciao world!
123+
124+
Bonjour world!
125+
126+
Hola world!
127+
128+
Hello world!
129+
```
130+
131+
Note that we are logging to Seqera Platform even though we did not use the `-with-tower` command!
132+
133+
### 1.4. Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline
134+
135+
Click on the link provided in the output to open the Seqera Platform for your run, then click on the `Configuration` tab.
136+
If you ran your pipeline from the `hello_nextflow` directory, you'll see something like this:
137+
138+
![Seqera Platform Configuration](seqera/img/resolved_configuration.png)
139+
140+
Notice that configuration for our pipeline run is being run pulled from three separate files:
141+
142+
- `/home/gitpod/.nextflow/config` - This is the global configuration file we just added.
143+
- `/home/gitpod/.nextflow/assets/nextflow-io/hello/nextflow.config` - This is the `nextflow.config` file from the `nextflow-io/hello` repository.
144+
- `/workspace/gitpod/nf-training/hello-nextflow/nextflow.config` - This is the `nextflow.config` file from our current working directory.
145+
146+
Nextflow resolves these configurations at runtime with a [specific order of precedence](https://www.nextflow.io/docs/latest/config.html#configuration-file).
147+
The general rule, however, is that more specific configurations override less specific ones, and config/params specified on the CLI will override defaults in the config files.
148+
149+
Helpfully, Seqera Platform shows us the final output of this configuration resolution process which can be very useful for debugging!
150+
151+
### Takeaway
152+
153+
You have learned how to:
154+
155+
- Set up your Seqera Platform token by exporting it to your environment.
156+
- Run Nextflow CLI with Seqera Platform visualizing and capturing logs.
157+
- Set up Seqera Platform logging by default.
158+
- Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline.
159+
160+
### What's next?
161+
162+
Learn how to launch Nextflow pipelines from Seqera Platform using the Launchpad feature.

‎docs/hello_nextflow/seqera/02_run_with_launchpad.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
## 2 Using Seqera Platform Launchpad to run Nextflow workflows
2+
3+
### 2.1. Navigate to the community/showcase workspace and launch a pipeline
4+
5+
### 2.2. Monitor the execution of a test rna-seq pipeline

‎docs/hello_nextflow/seqera/03_add_compute_environment.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
## 3. Add a compute environment
2+
3+
In this section, we will simulate setting up a new compute environment to run our pipeline in Seqera Platform Launchpad.
4+
Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run.
5+
It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures.
6+
7+
![Compute environments](seqera/img/compute_env_platforms.png)
8+
9+
Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below.
10+
11+
!!! tip "The following guides describe how to configure each of these compute environments."
12+
13+
* [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch)
14+
* [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch)
15+
* [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch)
16+
* [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences)
17+
* [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc)
18+
* [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks)
19+
* [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke)
20+
* [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s)
21+
22+
To practice this process, we will simulate setting up a new slurm compute environment on our gitpod using Tower Agent.
23+
24+
### 3.1. Setup Tower Agent
25+
26+
### 3.2. Add a new Slurm compute environment

‎docs/hello_nextflow/seqera/04_setup_launchpad.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## 4. Setup a new pipeline to run in Seqera Platform Launchpad
2+
3+
In this section, we will set up a new pipeline to run in Seqera Platform Launchpad.
4+
5+
### 4.1. Add your Github credentials to your personal workspace in Seqera Platform
6+
7+
### 4.2. Add the pipeline seqeralabs/nf-hello-gatk to your launchpad
8+
9+
### 4.3. Change the branch of the pipeline to nextflow_schema
10+
11+
### 4.4. _Unsuccessfully_ launch a pipeline on our Tower Agent

‎docs/hello_nextflow/seqera/img/compute_env_platforms.png

333 KB
Loading

‎docs/hello_nextflow/seqera/img/resolved_configuration.png

91.2 KB
Loading

0 commit comments

Comments
 (0)