Merge pull request #169 from assemblerflow/dev

Version 1.4.0 release

Author: ODiogoSilva

changelog.md

@@ -2,6 +2,54 @@
 
 ## Changes in upcoming release (`dev` branch) 
 
+## 1.4.0
+
+### New features
+
+- Added new `recipe` system to flowcraft along with 6 starting recipes.
+Recipes are pre-made and curated pipelines that address specific questions.
+To create a recipe, the `-r <recipe_name>` can be used. To list available
+recipes, the `--recipe-list` and `--recipe-list-short` options were added. 
+- Added `-ft` or `--fetch-tags` which allows to retrieve all DockerHub 
+container tags.
+- Added function to collect all the components from the components classes,
+replacing the current process_map dictionary implementation. Now, it will be
+generated from the engine rather than hardcoded into the dict.
+
+### Components changes
+
+- Added new `disableRR` param in the `spades` component that disables repeat
+resolution
+- The `abyss` and `spades` components emit GFA in a secondary channel.
+- The new `bandage` component can accept either FASTA from a primary channel
+  or GFA from a secondary channel.
+- Updated skesa to version 2.3.0.
+- Updated mash based components for the latest version - 1.6.0-1.
+
+### New components
+
+- Added component `abyss`.
+- Added component `bandage`.
+- Added component `unicycler`.
+- Added component `prokka`.
+- Added component `bcalm`.
+- Added component `diamond`.
+
+### Minor/Other changes
+
+- Added removal of duplicate IDs from `reads_download` component input.
+- Added seed parameter to `downsample_fastq` component.
+- Added bacmet database to `abricate` component.
+- Added default docker option to avoid docker permission errors.
+- Changed the default URL generated by inspect and report commands.
+- Added directives to `-L` parameter of build module.
+
+### Bug fixes
+
+- Fixed forks with same source process name.
+- Fixed `inspect` issue when tasks took more than a day in duration.
+- Added hardware address to `inpsect` and `report` hash.
+
 ## 1.3.1
 
 ### Features
@@ -16,6 +64,7 @@ which is particularly useful in very large workflows.
 `mapping_patlas`.
 
 ### New components
+
 - Added component `fast_ani`.
 
 ### Minor/Other changes

docs/dev/create_process.rst

@@ -14,10 +14,6 @@ The addition of a new process to FlowCraft requires three main steps:
    information about the process (e.g., expected input/output, secondary inputs,
    etc.).
 
-#. `Add to available processes`_: Add the :class:`~flowcraft.generator.process` class to the
-   dictionary of available process in
-   :attr:`flowcraft.generator.engine.process_map`.
-
 .. _create-process:
 
 Create process template
@@ -211,20 +207,20 @@ Depending on the process, other attributes may be required:
     - `Directives`_: Default information for RAM/CPU/Container directives
       and more.
 
-Add to available processes
+Add to available components
 ::::::::::::::::::::::::::
 
-The final step is to add your new process to the list of available processes.
-This list is defined in :attr:`flowcraft.generator.engine.process_map`
-module, which is a dictionary
-mapping the process template name to the corresponding template class::
-
-    process_map = {
-    <other_process>
-    "my_process_template": process.MyProcess
-    }
+Contrary to previous implementation (version <= 1.3.1), the available components
+are now retrieved automatically by FlowCraft and there is no need to add the
+process to any dictionary (previous ``process_map``). In order for the component
+to be accessible to ``flowcraft build`` the process template name in
+``snake_case`` must match the process class in ``CamelCase``. For instance,
+if the process template is named ``my_process.nf``, the process class must
+be ``MyProcess``, then the FlowCraft will be able to automatically add it to the
+list of available components.
 
-Note that the template string does not include the ``.nf`` extension.
+.. note::
+    Note that the template string does not include the ``.nf`` extension.
 
 Process attributes
 ------------------

docs/dev/create_recipe.rst

@@ -0,0 +1,116 @@
+Recipe creation guidelines
+===========================
+
+Recipes are pre-made pipeline strings that may be associated with specific
+parameters and directives and are used to rapidly build a certain type of
+pipeline.
+
+Instead of building a pipeline like::
+
+    -t "integrity_coverage fastqc_trimmomatic fastqc spades pilon"
+
+The user simply can specific a recipe with that pipeline::
+
+    -r assembly
+
+Recipe creation
+---------------
+
+The creation of new recipes is a very simple and straightforward process.
+You need to create a new file in the ``flowcraft/generator/recipes`` folder
+with any name and create a basic class with three attributes::
+
+    try:
+        from generator.recipe import Recipe
+    except ImportError:
+        from flowcraft.generator.recipe import Recipe
+
+
+    class Innuca(Recipe):
+
+        def __init__(self):
+            super().__init__()
+
+            # Recipe name
+            self.name = "innuca"
+
+            # Recipe pipeline
+            self.pipeline_str = <pipeline string>
+
+            # Recipe parameters and directives
+            self.directives = { <directives> }
+
+And that's it! Now there is a new recipe available with the ``innuca`` name and
+we can build this pipeline using the option ``-r innuca``.
+
+Name
+^^^^
+
+This is the name of the recipe, which is used to make a match with the recipe
+name provided by the user via the ``-r`` option.
+
+Pipeline_str
+^^^^^^^^^^^^
+
+The pipeline string as if provided via the ``-t`` option.
+
+Directives
+^^^^^^^^^^
+
+A dictionary containing the parameters and directives for each process in the
+pipeline string. **Setting this attribute is optional and components
+that are not specified here will assume their default values**. In general, each
+element in this dictionary should have the following format::
+
+    self.directives = {
+        "component_name": {
+            "params": {
+                "paramA": "value"
+            },
+            "directives": {
+                "directiveA": "value"
+            }
+        }
+    }
+
+This will set the provided parameters and directives to the component, but it is
+possible to provide only one.
+
+A more concrete example of a real component and directives follows::
+
+    self.pipeline_str = "integrity_coverage fastqc"
+
+    # Set parameters and directives only for integrity_coverage
+    # and leave fastqc with the defaults
+    self.directives = {
+        "integrity_coverage": {
+            "params": {
+                "minCoverage": 20
+            },
+            "directives": {
+                "memory": "1GB"
+            }
+        }
+    }
+
+Duplicate components
+~~~~~~~~~~~~~~~~~~~~
+
+In some cases, the same component may be present multiple times in the pipeline
+string of a recipe. In these cases, directives can be assigned to each individual
+component by adding a ``#<id>`` suffix to the component::
+
+    self.pipeline_str = "integrity_coverage ( trimmomatic spades#1 | spades#2)"
+
+    self.directives = {
+        "spades#1": {
+            "directives": {
+                "memory": "10GB"
+            }
+        },
+        "spades#2": {
+            "directives": {
+                "version": "3.7.0"
+            }
+        }
+    }

docs/index.rst

@@ -44,6 +44,7 @@ A NextFlow pipeline assembler for genomics.
    dev/general_orientation
    dev/create_process
    dev/create_template
+   dev/create_recipe
    dev/containers
    dev/process_dotfiles
    dev/pipeline_reporting

docs/resources/reports/mash_dist_table.png

@@ -393,7 +393,7 @@ Local visualization
 :::::::::::::::::::
 
 The FlowCraft report JSON file can also be visualized locally by drag and dropping
-it into the FlowCraft web application page, currently hosted at http://192.92.149.169/reports
+it into the FlowCraft web application page, currently hosted at http://www.flowcraft.live/reports
 
 Offline visualization
 :::::::::::::::::::::

docs/resources/reports/sliding_window_mash_dist.png

@@ -0,0 +1,55 @@
+diamond
+=======
+
+Purpose
+-------
+
+This component performs ``blastx`` or ``blastp`` with diamond. The database
+used by diamond can be provided from the local disk or generated in the process.
+This component uses the same output type as abricate with the same blast output
+fields.
+
+.. note::
+    Software page: https://github.com/bbuchfink/diamond
+
+
+Input/Output type
+-----------------
+
+- Input type: ``Fasta``
+- Output type: None
+
+.. note::
+    The default input parameter for fasta data is ``--fasta``.
+
+Parameters
+----------
+
+- ``pathToDb``: Provide full path for the diamond database. If none is provided
+  then will try to fetch from the previous process. Default: None
+
+- ``fastaToDb``: Provide the full path for the fasta to construct a diamond
+  database. Default: None
+
+- ``blastType``: Defines the type of blast that diamond will do. Can wither be
+  blastx or blastp. Default: blastx
+
+Published results
+-----------------
+
+- ``results/annotation/diamond*``: Stores the results of the abricate screening
+  for each sample and for each specified database.
+
+Published reports
+-----------------
+
+None.
+
+Default directives
+------------------
+
+- ``diamond``:
+    - ``container``: flowcraft/diamond
+    - ``version``: 0.9.22-1
+    - ``memory``: { 4.GB * task.attempt }
+    - ``cpus``: 2

docs/user/basic_usage.rst

@@ -23,6 +23,7 @@ Parameters
 - ``genomeSize``: Genome size estimate for the samples. It is used to
   estimate the coverage.
 - ``depth``: The target depth to which the reads should be subsampled.
+- ``seed``: The seed number for seqtk. By default it is 100.
 
 Published results
 -----------------

docs/user/components/diamond.rst

@@ -44,7 +44,10 @@ Parameters
 
 - ``refFile``: Specifies the reference file to be provided to mash. It can either
   be a fasta or a .msh reference sketch generated by mash.
-  Default: '/ngstools/data/patlas.msh'.
+  Default: '/ngstools/data/patlas.msh'. If the component ``mash_sketch_fasta``
+  is executed before this component, this parameter will be ignored and instead
+  the secondary link between the two processes will be used to feed this
+  component with the reference sketch.
 
 
 Published results

docs/user/components/downsample_fastq.rst

@@ -1,11 +1,11 @@
 mash_screen
-==============
+===========
 
 Purpose
 -------
 
 This component performes mash screen to find plasmids
-contained in high throughoput sequencing data, using as inputs read files
+contained in high throughout sequencing data, using as inputs read files
 (FastQ files). Then, the resulting file can
 be imported into `pATLAS <http://www.patlas.site/>`_.
 This component searches for containment of a given sequence in read sequencing
@@ -38,8 +38,11 @@ Parameters
   reference sequence. Default: 0.9.
 
 - ``refFile``: "Specifies the reference file to be provided to mash. It can
-  either be a fasta or a .msh reference sketch generated by mash.
-  Default: '/ngstools/data/patlas.msh'.
+  either be a fastq or a .msh reference sketch generated by mash.
+  Default: '/ngstools/data/patlas.msh'. If the component ``mash_sketch_fastq``
+  is executed before this component, this parameter will be ignored and instead
+  the secondary link between the two processes will be used to feed this
+  component with the reference sketch.
 
 
 Published results

docs/user/components/mash_dist.rst

@@ -44,4 +44,4 @@ Default directives
 
 - ``mashSketchFasta``:
     - ``container``: flowcraft/mash-patlas
-    - ``version``: 1.4.1
+    - ``version``: 1.6.0-1

docs/user/components/mash_screen.rst

@@ -4,7 +4,9 @@ mash_sketch_fastq
 Purpose
 -------
 
-This component performs mash sketch for fastq input files.
+This component performs mash sketch for fastq input files. These sketches can
+be used by ``mash_dist`` and ``mash_screen`` components to fetch the
+reference file for mash.
 
 .. note::
     - MASH documentation can be found `here <https://mash.readthedocs.io/en/latest/>`_.
@@ -52,4 +54,4 @@ Default directives
 
 - ``mashSketchFastq``:
     - ``container``: flowcraft/mash-patlas
-    - ``version``: 1.4.1
+    - ``version``: 1.6.0-1