Skip to content

Commit 89f4c7a

Browse files
kenibrewerkenibrewer
committed
✨ feat:(hello_seqera): first pass of run with Nextflow section
📝 docs(seqera/run_with_cli): incorporate feedback from review
1 parent 6d39e6b commit 89f4c7a

11 files changed

+380
-3
lines changed

‎docs/hello_nextflow/10_hello_seqera.md‎

Lines changed: 171 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,175 @@
1+
---
2+
title: "Part 9: Hello Seqera"
3+
description: Get started with Seqera Platform
4+
---
5+
16
# Part 9: Hello Seqera
27

3-
TODO
8+
So far we've been running Nextflow workflows on our local machine using the command line interface.
9+
In this section, we'll introduce you to Seqera Platform, a powerful cloud-based platform for running, monitoring, and sharing Nextflow workflows.
410

5-
---
11+
!!! tip
12+
13+
[Sign up](https://cloud.seqera.io/) to try Seqera for free or request a [demo](https://seqera.io/demo/) for deployments in your own on-premise or cloud environment.
14+
15+
You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform GUI, and the API.
16+
17+
You can use Seqera Platform via either the **CLI**, through the **online GUI** or through the **API**.
18+
19+
--8<-- "docs/hello_nextflow/seqera/01_run_with_cli.md"
20+
21+
--8<-- "docs/hello_nextflow/seqera/02_run_with_launchpad.md"
22+
23+
--8<-- "docs/hello_nextflow/seqera/03_add_compute_environment.md"
24+
25+
--8<-- "docs/hello_nextflow/seqera/04_setup_launchpad.md"
26+
27+
!!! Warning WORK-IN-PROGRESS
28+
29+
Everything below this note is content that was copied from the original guide and needs to be moved or rewritten into the above subpages.
30+
31+
## Online GUI
32+
33+
To run using the GUI, there are three main steps:
34+
35+
1. Create an account and login into Seqera Platform, available free of charge, at [cloud.seqera.io](https://cloud.seqera.io).
36+
2. Create and configure a new [compute environment](https://docs.seqera.io/platform/24.1/compute-envs/overview).
37+
3. Start [launching pipelines](https://docs.seqera.io/platform/24.1/launch/launchpad#launchpad).
38+
39+
### Selecting a default compute environment
40+
41+
If you have more than one **Compute Environment**, you can select which one will be used by default when launching a pipeline.
42+
43+
1. Navigate to your [compute environments](https://docs.seqera.io/platform/24.1/compute-envs/overview).
44+
2. Choose your default environment by selecting the **Make primary** button.
45+
46+
**Congratulations!**
47+
48+
You are now ready to launch workflows with your primary compute environment.
49+
50+
### Launchpad
51+
52+
Launchpad makes it easy for any workspace user to launch a pre-configured pipeline.
53+
54+
![Launchpad](img/overview_launch.png)
55+
56+
A pipeline is a repository containing a Nextflow workflow, a compute environment and workflow parameters.
57+
58+
### Pipeline Parameters Form
59+
60+
Launchpad automatically detects the presence of a `nextflow_schema.json` in the root of the repository and dynamically creates a form where users can easily update the parameters.
61+
62+
!!! info
63+
64+
The parameter forms view will appear if the pipeline has a Nextflow schema file for the parameters. Please refer to the [Nextflow Schema guide](https://docs.seqera.io/platform/24.1/pipeline-schema/overview) to learn more about the schema file use-cases and how to create them.
65+
66+
This makes it trivial for users without any expertise in Nextflow to enter their workflow parameters and launch.
67+
68+
![Pipeline parameters](img/launch_rnaseq_nextflow_schema.png)
69+
70+
### Adding a new pipeline
71+
72+
Adding a pipeline to the pre-saved workspace launchpad is detailed in full on the [Seqera webpage docs](https://docs.seqera.io/platform/24.1/launch/launchpad#add-new-pipeline).
73+
74+
In brief, these are the steps you need to follow to set up a pipeline.
75+
76+
1. Select the Launchpad button in the navigation bar. This will open the **Launch Form**.
77+
2. Select a [compute environment](https://docs.seqera.io/platform/24.1/compute-envs/overview).
78+
3. Enter the repository of the workflow you want to launch. e.g. <https://github.com/nf-core/rnaseq.git>
79+
4. Select a workflow **Revision number**. The Git default branch (main/master) or `manifest.defaultBranch` in the Nextflow configuration will be used by default.
80+
5. Set the **Work directory** location of the Nextflow work directory. The location associated with the compute environment will be selected by default.
81+
6. Enter the name(s) of each of the Nextflow **Config profiles** followed by the `Enter` key. See the Nextflow [Config profiles](https://www.nextflow.io/docs/latest/config.html#config-profiles) documentation for more details.
82+
7. Enter any workflow parameters in YAML or JSON format. YAML example:
83+
84+
```yaml
85+
reads: "s3://nf-bucket/exome-data/ERR013140_{1,2}.fastq.bz2"
86+
paired_end: true
87+
```
88+
89+
8. Select Launch to begin the pipeline execution.
90+
91+
!!! info
92+
93+
Nextflow workflows are simply Git repositories and can be changed to any public or private Git-hosting platform. See [Git Integration](https://docs.seqera.io/platform/24.1/git/overview) in the Seqera Platform docs and [Pipeline Sharing](https://www.nextflow.io/docs/latest/sharing.html) in the Nextflow docs for more details.
94+
95+
!!! note
96+
97+
The credentials associated with the compute environment must be able to access the work directory.
98+
99+
!!! info
100+
101+
In the configuration, the full path to a bucket must be specified with single quotes around strings and no quotes around booleans or numbers.
102+
103+
!!! tip
104+
105+
To create your own customized Nextflow Schema for your workflow, see the examples from the `nf-core` workflows that have adopted this approach. For example, [eager](https://github.com/nf-core/eager/blob/2.3.3/nextflow_schema.json) and [rnaseq](https://github.com/nf-core/rnaseq/blob/3.0/nextflow_schema.json).
106+
107+
For advanced settings options check out this [page](https://docs.seqera.io/platform/24.1/launch/launchpad#advanced-settings).
108+
109+
There is also community support available if you get into trouble, join the Nextflow Slack by following this [link](https://www.nextflow.io/slack-invite.html).
110+
111+
!!! cboard-list-2 "Summary"
112+
113+
In this step you have learned:
114+
115+
1. How to create an account and login into Seqera Platform
116+
2. How to configure your compute environment.
117+
3. How to add, customize, and launch a pipeline with Seqera Platform.
118+
119+
## API
120+
121+
To learn more about using the Seqera Platform API, visit the [API section](https://docs.seqera.io/platform/24.1/api/overview) in the documentation.
122+
123+
## Workspaces and Organizations
124+
125+
Seqera Platform simplifies the development and execution of pipeline by providing a centralized interface for users and organizations.
126+
127+
Each user has a unique **workspace** where they can interact and manage all resources such as workflows, compute environments and credentials. Details of this can be found [here](https://docs.seqera.io/platform/24.1/getting-started/workspace-setup).
128+
129+
Organisations can have multiple workspaces with customized access for specific organisation **members** and **collaborators**.
130+
131+
### Organization resources
132+
133+
You can create your own organization and participant workspace by following the docs at [Seqera](https://docs.seqera.io/platform/24.1/orgs-and-teams/workspace-management).
134+
135+
Seqera Platform allows the creation of multiple organizations, each of which can contain multiple workspaces with shared users and resources. This allows any organization to customize and organize the usage of resources while maintaining an access control layer for users associated with a workspace.
136+
137+
### Organization users
138+
139+
Any user can be added or removed from a particular organization or a workspace and can be allocated a specific access role within that workspace.
140+
141+
The Teams feature provides a way for organizations to group various users and participants together into teams. For example, `workflow-developers` or `analysts`, and apply access control to all the users within this team collectively.
142+
143+
For further information, please refer to the [User Management](https://docs.seqera.io/platform/24.1/orgs-and-teams/organizations) section.
144+
145+
### Setting up a new organization
146+
147+
Organizations are the top-level structure and contain Workspaces, Members, Teams and Collaborators.
148+
149+
To create a new Organization:
150+
151+
1. Click on the dropdown next to your name and select New organization to open the creation dialog.
152+
2. On the dialog, fill in the fields as per your organization. The Name and Full name fields are compulsory.
153+
154+
!!! note
155+
156+
A valid name for the organization must follow a specific pattern. Please refer to the UI for further instructions.
157+
158+
3. The rest of the fields such as Description, Location, Website URL and Logo Url are optional.
159+
4. Once the details are filled in, you can access the newly created organization using the organization’s page, which lists all of your organizations.
160+
161+
!!! note
162+
163+
It is possible to change the values of the optional fields either using the Edit option on the organization’s page or by using the Settings tab within the organization page, provided that you are the Owner of the organization.
164+
165+
!!! tip
166+
167+
A list of all the included Members, Teams and Collaborators can be found on the organization page.
168+
169+
!!! cboard-list-2 "Summary"
170+
171+
In this step you have learned:
6172

7-
TODO
173+
1. How to create a new organization
174+
2. How to access the newly created organization
175+
3. How to change organization settings

‎docs/hello_nextflow/seqera/01_run_with_cli.md‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
## 1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI
2+
3+
We'll start by using the Nextflow CLI to launch a pipeline and monitor it in Seqera Platform.
4+
Start by logging into the [Seqera Platform](https://cloud.seqera.io/).
5+
6+
!!! info "Nextflow Tower"
7+
8+
Seqera Platform was previously known as Nextflow Tower.
9+
You'll still see references to the previous name in environment variable and cli option names.
10+
11+
### 1.1. Set up your Seqera Platform token by exporting it to your environment
12+
13+
Follow these steps to set up your token:
14+
15+
1. Create a new token by clicking on the **Settings** drop-down menu:
16+
17+
![Create a token](seqera/img/usage_create_token.png)
18+
19+
2. Name your token:
20+
21+
![Name your token](seqera/img/usage_name_token.png)
22+
23+
3. Save your token safely:
24+
25+
![Save token](seqera/img/usage_token.png)
26+
27+
!!! note
28+
29+
Leave this browser tab open as we will need the token once more to store it as a Nextflow secret.
30+
31+
4. To make your token available to the Nextflow CLI, export it on the command line:
32+
33+
Open a terminal and type:
34+
35+
```bash
36+
export TOWER_ACCESS_TOKEN=eyxxxxxxxxxxxxxxxQ1ZTE=
37+
```
38+
39+
Where `eyxxxxxxxxxxxxxxxQ1ZTE=` is the token you have just created.
40+
41+
### 1.2. Run Nextflow cli with Seqera Platform visualizing and capturing logs
42+
43+
Run your Nextflow workflows as usual with the addition of the `-with-tower` command:
44+
45+
```bash
46+
nextflow run nextflow-io/hello -with-tower
47+
```
48+
49+
You will see output similar to the following:
50+
51+
```console title="Output"
52+
N E X T F L O W ~ version 24.04.4
53+
54+
Launching `https://github.com/nextflow-io/hello` [evil_engelbart] DSL2 - revision: afff16a9b4 [master]
55+
56+
Downloading plugin nf-tower@1.9.1
57+
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/5Gs0qqV9Y9rguE
58+
executor > local (4)
59+
[80/810411] process > sayHello (1) [100%] 4 of 4 ✔
60+
Ciao world!
61+
62+
Bonjour world!
63+
64+
Hola world!
65+
66+
Hello world!
67+
```
68+
69+
Use ++ctrl+click++ or ++cmd+click++ on the link to open it in your browser.
70+
You'll see the Seqera Platform interface with the job finished and the logs captured.
71+
72+
![Seqera Platform](seqera/img/run_with_tower.png)
73+
74+
You will see and be able to monitor your **Nextflow jobs** in Seqera Platform.
75+
76+
### 1.3. Set up Seqera Platform in Nextflow configuration
77+
78+
Doing that token setup regularly can get bit tedious, but the same setup can be applied in configuration applied to Nexflow configuration so that it does not need to be set each time.
79+
This can be the `nextflow.config` file of a specific project, or the global file located at `$HOME/.nextflow/config`, which will apply to all your runs.
80+
81+
Before we set the configuration, we need to permanently store the token in Nextflow using a [Nextflow secret](https://www.nextflow.io/docs/latest/secrets.html):
82+
83+
```bash
84+
nextflow secrets set tower_access_token "eyxxxxxxxxxxxxxxxQ1ZTE="
85+
```
86+
87+
The following block of configuration will enable Seqera Platform logging by default:
88+
89+
```groovy title="nextflow.config"
90+
tower {
91+
enabled = true
92+
endpoint = "https://api.cloud.seqera.io"
93+
accessToken = secrets.tower_access_token
94+
}
95+
```
96+
97+
However, instead of enabling Seqera Platform for an individual pipeline, we want to enable it for ourselves globally.
98+
99+
Run the following command to put the config block in your user configuration file located at `$HOME/.nextflow/config`.
100+
101+
```bash
102+
cat <<EOF >> $HOME/.nextflow/config
103+
tower {
104+
enabled = true
105+
endpoint = "https://api.cloud.seqera.io"
106+
accessToken = secrets.tower_access_token
107+
}
108+
EOF
109+
```
110+
111+
Run your Nextflow workflows as usual:
112+
113+
```bash
114+
nextflow run nextflow-io/hello
115+
```
116+
117+
You will see the following output:
118+
119+
```console title="Output"
120+
N E X T F L O W ~ version 24.04.4
121+
122+
Launching `https://github.com/nextflow-io/hello` [fabulous_euclid] DSL2 - revision: afff16a9b4 [master]
123+
124+
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/KYjRktIlOuxrh
125+
executor > local (4)
126+
[71/eaa915] process > sayHello (3) [100%] 4 of 4 ✔
127+
Ciao world!
128+
129+
Bonjour world!
130+
131+
Hola world!
132+
133+
Hello world!
134+
```
135+
136+
Note that we are logging to Seqera Platform even though we did not use the `-with-tower` command!
137+
138+
### 1.4. Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline
139+
140+
Click on the link provided in the output to open the Seqera Platform for your run, then click on the `Configuration` tab.
141+
If you ran your pipeline from the `hello_nextflow` directory, you'll see something like this:
142+
143+
![Seqera Platform Configuration](seqera/img/resolved_configuration.png)
144+
145+
Notice that configuration for our pipeline run is being run pulled from three separate files:
146+
147+
- `/home/gitpod/.nextflow/config` - This is the global configuration file we just added.
148+
- `/home/gitpod/.nextflow/assets/nextflow-io/hello/nextflow.config` - This is the `nextflow.config` file from the `nextflow-io/hello` repository.
149+
- `/workspace/gitpod/nf-training/hello-nextflow/nextflow.config` - This is the `nextflow.config` file from our current working directory.
150+
151+
Nextflow resolves these configurations at runtime with a [specific order of precedence](https://www.nextflow.io/docs/latest/config.html#configuration-file).
152+
The general rule, however, is that more specific configurations override less specific ones, and config/params specified on the CLI will override defaults in the config files.
153+
154+
Helpfully, Seqera Platform shows us the final output of this configuration resolution process which can be very useful for debugging!
155+
156+
### Takeaway
157+
158+
You have learned how to:
159+
160+
- Set up your Seqera Platform token by exporting it to your environment.
161+
- Run Nextflow CLI with Seqera Platform visualizing and capturing logs.
162+
- Set up Seqera Platform logging by default.
163+
- Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline.
164+
165+
### What's next?
166+
167+
Learn how to launch Nextflow pipelines from Seqera Platform using the Launchpad feature.

‎docs/hello_nextflow/seqera/02_run_with_launchpad.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
## 2 Using Seqera Platform Launchpad to run Nextflow workflows
2+
3+
### 2.1. Navigate to the community/showcase workspace and launch a pipeline
4+
5+
### 2.2. Monitor the execution of a test rna-seq pipeline

‎docs/hello_nextflow/seqera/03_add_compute_environment.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
## 3. Add a compute environment
2+
3+
In this section, we will simulate setting up a new compute environment to run our pipeline in Seqera Platform Launchpad.
4+
Seqera Platform uses the concept of **Compute Environments** to define the execution platform where a workflow will run.
5+
It supports the launching of workflows into a growing number of **cloud** and **on-premise** infrastructures.
6+
7+
![Compute environments](seqera/img/compute_env_platforms.png)
8+
9+
Each compute environment must be pre-configured to enable Seqera Platform to submit tasks. You can read more on how to set up each environment using the links below.
10+
11+
!!! tip "The following guides describe how to configure each of these compute environments."
12+
13+
* [AWS Batch](https://docs.seqera.io/platform/24.1/compute-envs/aws-batch)
14+
* [Azure Batch](https://docs.seqera.io/platform/24.1/compute-envs/azure-batch)
15+
* [Google Batch](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-batch)
16+
* [Google Life Sciences](https://docs.seqera.io/platform/24.1/compute-envs/google-cloud-lifesciences)
17+
* [HPC (LSF, Slurm, Grid Engine, Altair PBS Pro)](https://docs.seqera.io/platform/24.1/compute-envs/hpc)
18+
* [Amazon Kubernetes (EKS)](https://docs.seqera.io/platform/24.1/compute-envs/eks)
19+
* [Google Kubernetes (GKE)](https://docs.seqera.io/platform/24.1/compute-envs/gke)
20+
* [Hosted Kubernetes](https://docs.seqera.io/platform/24.1/compute-envs/k8s)
21+
22+
To practice this process, we will simulate setting up a new slurm compute environment on our gitpod using Tower Agent.
23+
24+
### 3.1. Setup Tower Agent
25+
26+
### 3.2. Add a new Slurm compute environment

‎docs/hello_nextflow/seqera/04_setup_launchpad.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## 4. Setup a new pipeline to run in Seqera Platform Launchpad
2+
3+
In this section, we will set up a new pipeline to run in Seqera Platform Launchpad.
4+
5+
### 4.1. Add your Github credentials to your personal workspace in Seqera Platform
6+
7+
### 4.2. Add the pipeline seqeralabs/nf-hello-gatk to your launchpad
8+
9+
### 4.3. Change the branch of the pipeline to nextflow_schema
10+
11+
### 4.4. _Unsuccessfully_ launch a pipeline on our Tower Agent

‎docs/hello_nextflow/seqera/img/compute_env_platforms.png‎

333 KB
Loading

‎docs/hello_nextflow/seqera/img/resolved_configuration.png‎

91.2 KB
Loading

‎docs/hello_nextflow/seqera/img/run_with_tower.png‎

88.9 KB
Loading

‎docs/hello_nextflow/seqera/img/usage_create_token.png‎

211 KB
Loading

‎docs/hello_nextflow/seqera/img/usage_name_token.png‎

193 KB
Loading

0 commit comments

Comments
 (0)