Nextflow training

Nextflow can be used on any POSIX compatible system (Linux, OS X, etc). It requires Bash and Java 8 (or later) to be installed.

Optional requirements:

Once you launched the environment, download the training materials running the command below on the terminal window that you will find at the botton of the page:

We recommend to follow this tutorial using the AWS Cloud 9 environment. However, if you want to use your machine you can alsodownload all the materials using curl:

Or alternatively if you have wget instead of curl in your machine:

To complete the environment setup you just need to run the script below, which installs Java an other requirements for this tutorial.

Install the latest version of Nextflow copy & pasting the following snippet in a terminal window:

Check the correct installation running the following command:

Nextflow is workflow orchestration engine and a programming domain specific language (DSL) that eases the writing of data-intensive computational pipelines.

It is designed around the idea that the Linux platform is the lingua franca of data science. Linux provides many simple but powerful command-line and scripting tools that, when chained together, facilitate complex data manipulations.

Nextflow extends this approach, adding the ability to define complex program interactions and a high-level parallel computational environment based on the dataflow programming model. Nextflow core features are:

Copy the following example into your favourite text editor and save it to a file named hello.nf :

This script defines two processes. The first splits a string into files containing chunks of 6 characters. The second receives these files and transforms their contents to uppercase letters. The resulting strings are emitted on the result channel and the final output is printed by the view operator.

Execute the script by entering the following command in your terminal:

It will output something similar to the text shown below:

You can see that the first process is executed once, and the second twice. Finally the result string is printed.

It’s worth noting that the process convertToUpper is executed in parallel, so there’s no guarantee that the instance processing the first split (the chunk Hello) will be executed before the one processing the second split (the chunk world!).

Thus, it is perfectly possible that you will get the final result printed out in a different order:

Nextflow keeps track of all the processes executed in your pipeline. If you modify some parts of your script, only the processes that are actually changed will be re-executed. The execution of the processes that are not changed will be skipped and the cached result used instead.

This helps a lot when testing or modifying part of your pipeline without having to re-execute it from scratch.

For the sake of this tutorial, modify the convertToUpper process in the previous example, replacing the process script with the string rev $y, so that the process looks like this:

Then save the file with the same name, and execute it by adding the -resume option to the command line:

It will print output similar to this:

You will see that the execution of the process splitLetters is actually skipped (the process ID is the same), and its results are retrieved from the cache. The second process is executed as expected, printing the reversed strings.

Pipeline parameters are simply declared by prepending to a variable name the prefix params, separated by dot character. Their value can be specified on the command line by prefixing the parameter name with a double dash character, i.e. --paramName

For the sake of this tutorial, you can try to execute the previous example specifying a different input string parameter, as shown below:

The string specified on the command line will override the default value of the parameter. The output will look like this:

Nextflow distinguish two different kinds of channels: queue channels and value channels.

The script block is a string statement that defines the command that is executed by the process to carry out its task.

A process contains one and only one script block, and it must be the last statement when the process contains input and output declarations.

The script block can be a simple string or multi-line string. The latter simplifies the writing of non trivial scripts composed by multiple commands spanning over multiple lines. For example::

By default the process command is interpreted as a Bash script. However any other scripting language can be used just simply starting the script with the corresponding Shebang declaration. For example:

Nextflow processes are isolated from each other but can communicate between themselves sending values through channels.

Inputs implicitly determine the dependency and the parallel execution of the process. The process execution is fired each time a new data is ready to be consumed from the input channel:

The input block defines which channels the process is expecting to receive inputs data from. You can only define one input block at a time and it must contain one or more inputs declarations.

The input block follows the syntax shown below:

The output declaration block allows to define the channels used by the process to send out the results produced.

There can be defined at most one output block and it can contain one or more outputs declarations. The output block follows the syntax shown below:

The when declaration allows you to define a condition that must be verified in order to execute the process. This can be any expression that evaluates a boolean value.

It is useful to enable/disable the process execution depending the state of various inputs and parameters. For example:

Directive declarations allow the definition of optional settings that affect the execution of the current process without affecting the semantic of the task itself.

They must be entered at the top of the process body, before any other declaration blocks (i.e. input, output, etc).

Directives are commonly used to define the amount of computing resources to be used or other meta directives like that allows the definition of extra information for configuration or logging purpose. For example:

The complete list of directives is available at this link.

Operators can be chained to implement custom behaviors:

Operators can be separated in to five groups:

Check the operators documentation on Nextflow web site.

To print something is as easy as using one of the print or println methods.

The only difference between the two is that the println method implicitly appends a new line character to the printed string.

Comments use the same syntax as in the C-family programming languages:

To define a variable, simply assign a value to it:

Local variables are defined using the def keyword:

It should be always used when defining variables local to a function or a closure.

A List object can be defined by placing the list items in square brackets:

You can access a given item in the list with square-bracket notation (indexes start at 0) or using the get method:

In order to get the length of the list use the size method:

Lists can also be indexed with negative indexes and reversed ranges.

List objects implements all methods provided by the Java java.util.List interface plus the extension methods provided by Groovy API.

Maps are like lists that have an arbitrary type of key instead of integer. Therefore, the syntax is very much aligned.

Maps can be accessed in a conventional square-bracket syntax or as if the key was a property of the map.

To add data or to modify a map, the syntax is similar to adding values to list:

Maps objects implements all methods provided by the Java java.util.Map interface plus the extension methods provided by Groovy API.

String literals can be defined enclosing them either with single-quoted or double-quotes characters.

Double-quoted strings can contain the value of an arbitrary variable by prefixing its name with the $ character, or the value of any expression by using the ${expression} syntax, similar to Bash/shell scripts:

This code prints:

Finally string literals can also be defined using the / character as delimiter. They are known as slashy strings and are useful for defining regular expressions and patterns, as there is no need to escape backslashes. As with double quote strings they allow to interpolate variables prefixed with a $ character.

Try the following to see the difference:

it prints:

A block of text that span multiple lines can be defined by delimiting it with triple single or double quotes:

Finally multi-line strings can also be defined with slashy string. For example:

Like before, multi-line strings inside double quotes and slash characters support variable interpolation, while single-quoted multi-line strings do not.

The if statement uses the same syntax common other programming lang such Java, C, JavaScript, etc.

The else branch is optional. Also curly brackets are optional when the branch define just a single statement.

Therefore a statement like:

Can be written as:

See the Groovy-Truth for details.

The previous statement can be further simplified using the Elvis operator as shown below:

The classical for loop syntax is supported as shown here:

Iteration over list objects is also possible using the syntax below:

It is possible to define a custom function into a script, as shown here:

A function can take multiple arguments separating them with a comma. The return keyword can be omitted and the function implicitly returns the value of the last evaluated expression. Also explicit types can be omitted (thought not recommended):

Closures are the swiss army knife of Nextflow/Groovy programming. In a nutshell a closure is is a block of code that can be passed as an argument to a function, it could also be defined an anonymous function.

More formally, a closure allows the definition of functions as first class objects.

The curly brackets around the expression it * it tells the script interpreter to treat this expression as code. The it identifier is an implicit variable that represents the value that is passed to the function when it is invoked.

Once compiled the function object is assigned to the variable square as any other variable assignments shown previously. To invoke the closure execution use the special method call or just use the round parentheses to specify the closure parameter(s). For example:

This is not very interesting until we find that we can pass the function square as an argument to other functions or methods. Some built-in functions take a function like this as an argument. One example is the collect method on lists:

It prints:

By default, closures take a single parameter called it, to give it a different name use the → syntax. For example:

It’s also possible to define closures with multiple, custom-named parameters.

For example, the method each() when applied to a map can take a closure with two arguments, to which it passes the key-value pair for each entry in the map object. For example:

It prints:

A closure has two other important features. First, it can access and modify variables in the scope where it is defined.

Second, a closure can be defined in an anonymous manner, meaning that it is not given a name, and is defined in the place where it needs to be used.

As an example showing both these features, see the following code fragment:

Learn more about closures in the Groovy documentation.

The complete Groovy language documentation is available at this link.

A great resource to master Apache Groovy syntax is Groovy in Action.

The script script1.nf defines the pipeline input parameters.

Run it by using the following command:

Try to specify a different input parameter, for example:

Nextflow allows the execution of any command or user script by using a process definition.

A process is defined by providing three main declarations: the process inputs, the process outputs and finally the command script.

The second example adds the index process.

It takes the transcriptome file as input and creates the transcriptome index by using the salmon tool.

Note how the input declaration defines a transcriptome variable in the process context that it is used in the command script to reference that file in the Salmon command line.

Try to run it by using the command:

The execution will fail because Salmon is not installed in your environment.

Add the command line option -with-docker to launch the execution through a Docker container as shown below:

This time it works because it uses the Docker container nextflow/rnaseq-nf defined in the nextflow.config file.

In order to avoid to add the option -with-docker add the following line in the nextflow.config file:

This step shows how to match read files into pairs, so they can be mapped by Salmon.

Edit the script script3.nf and add the following statement as the last line:

Save it and execute it with the following command:

It will print an output similar to the one shown below:

The above example shows how the read_pairs_ch channel emits tuples composed by two elements, where the first is the read pair prefix and the second is a list representing the actual files.

Try it again specifying different read files by using a glob pattern:

The script script4.nf adds the quantification process.

In this script note as the index_ch channel, declared as output in the index process, is now used as a channel in the input section.

Also note as the second input is declared as a tuple composed by two elements: the pair_id and the reads in order to match the structure of the items emitted by the read_pairs_ch channel.

Execute it by using the following command:

You will see the execution of the quantification process.

The -resume option cause the execution of any step that has been already processed to be skipped.

Try to execute it with more read files as shown below:

You will notice that the quantification process is executed more than one time.

Nextflow parallelizes the execution of your pipeline simply by providing multiple input data to your script.

This step implements a quality control of your input reads. The inputs are the same read pairs which are provided to the quantification steps

You can run it by using the following command:

The script will report the following error message:

This step collect the outputs from the quantification and fastqc steps to create a final report by using the MultiQC tool.

Execute the script with the following command:

It creates the final report in the results folder in the current work directory.

In this script note the use of the mix and collect operators chained together to get all the outputs of the quantification and fastqc process as a single input.

This step shows how to execute an action when the pipeline completes the execution.

Note that Nextflow processes define the execution of asynchronous tasks i.e. they are not executed one after another as they are written in the pipeline script as it would happen in a common imperative programming language.

The script uses the workflow.onComplete event handler to print a confirmation message when the script completes.

Try to run it by using the following command:

Real world pipelines use a lot of custom user scripts (BASH, R, Python, etc). Nextflow allows you to use and manage all these scripts in consistent manner. Simply put them in a directory named bin in the pipeline project root. They will be automatically added to the pipeline execution PATH.

For example, create a file named fastqc.sh with the following content:

Save it, give execute permission and move it in the bin directory as shown below:

Then, open the script7.nf file and replace the fastqc process' script with the following code:

Run it as before:

Nextflow is able to produce multiple reports and charts providing several runtime metrics and execution information.

Run the rnaseq-nf pipeline previously introduced as shown below:

The -with-report option enables the creation of the workflow execution report. Open the file report.html with a browser to see the report created with the above command.

The -with-trace option enables the create of a tab separated file containing runtime information for each executed task. Check the content of the file trace.txt for an example.

The -with-timeline option enables the creation of the workflow timeline report showing how processes where executed along time. This may be useful to identify most time consuming tasks and bottlenecks. See an example at this link.

Finally the -with-dag option enables to rendering of the workflow execution direct acyclic graph representation. Note: this feature requires the installation of Graphviz in your computer. See here for details.

Note: runtime metrics may be incomplete for run short running tasks as in the case of this tutorial.

Send a notification email when the workflow execution complete using the -N <email address> command line option. Execute again the previous example specifying your email address:

Alternatively you can provide the settings of the STMP server needed to send the mail notification in the Nextflow config file. See mail documentation for details.

Nextflow allows the execution of a pipeline project directly from a GitHub repository (or similar services eg. BitBucket and GitLab).

This simplifies the sharing and the deployment of complex projects and tracking changes in a consistent manner.

The following GitHub repository hosts a complete version of the workflow introduced in this tutorial:

github.com/nextflow-io/rnaseq-nf

You can run it by specifying the project name as shown below:

It automatically downloads it and store in the $HOME/.nextflow folder. Use the command info to show the project information, e.g.:

Nextflow allows the execution of a specific revision of your project by using the -r command line option. For Example:

Revision are defined by using Git tags or branches defined in the project repository.

This allows a precise control of the changes in your project files and dependencies over time.

Get practice with basic Docker commands to pull, run and build your own containers.

A container is a ready-to-run Linux environment which can be executed in an isolated manner from the hosting system. It has own copy of the file system, processes space, memory management, etc.

Containers are a Linux feature known as Control Groups or Cgroups introduced with kernel 2.6.

Docker adds to this concept an handy management tool to build, run and share container images.

These images can be uploaded and published in a centralised repository know as Docker Hub, or hosted by other parties like for example Quay.

Singularity is container runtime designed to work in HPC data center, where the usage of Docker is generally not allowed due to security constraints.

Singularity implements a container execution model similarly to Docker however it uses a complete different implementation design.

A Singularity container image is archived as a plain file that can be stored in a shared file system and accessed by many computing nodes managed by a batch scheduler.

Configuration properties can be used as variables in the configuration file itself, by using the usual $propertyName or ${expression} syntax.

Configuration files use the same conventions for comments used in the Nextflow script:

Configuration settings can be organized in different scopes by dot prefixing the property names with a scope identifier or grouping the properties in the same scope using the curly brackets notation. This is shown in the following example:

The scope params allows the definition of workflow parameters that overrides the values defined in the main workflow script.

This is useful to consolidate one or more execution parameters in a separate file.

The env scope allows the definition one or more variable that will be exported in the environment where the workflow tasks will be executed.

The process directives allow the specification of specific settings for the task execution such as cpus, memory, container and other resources in the pipeline script.

This is useful specially when prototyping a small workflow script.

However it’s always a good practice to decouple the workflow execution logic from the process configuration settings, i.e. it’s strongly suggested to define the process settings in the workflow configuration file instead of the workflow script.

The process configuration scope allows the setting of any process directives in the Nextflow configuration file. For example:

The above config snippet defines the cpus, memory and container directives for all processes in your workflow script.

The process selector can be used to apply the configuration to a specific process or group of processes (discussed later).

INFO: The syntax for setting process directives in the configuration file requires = ie. assignment operator, instead it should not be used when setting process directives in the workflow script.

This important especially when you want to define a config setting using a dynamic expression using a closure. For example:

Directives that requires more than one value, e.g. pod, in the configuration file need to be expressed as a map object.

Finally directives that allows to be repeated in the process definition, in the configuration files need to be defined as a list object. For example:

The use of a Conda environment can also be provided in the configuration file adding the following setting in the nextflow.config file:

You can either specify the path of an existing Conda environment directory or the path of Conda environment YAML file.

A key Nextflow feature is the ability to decouple the workflow implementation from the actual execution platform implementing an abstraction layer that allows the deployment of the resulting workflow on any executing platform support by the framework.

To run your pipeline with a batch scheduler modify the nextflow.config file specifying the target executor and the required computing resources if needed. For example:

When using a batch scheduler is generally needed to specify the amount of resources i.e. cpus, memory, execution time, etc. required by each task.

This can be done using the following process directives:

Configuration files can contain the definition of one or more profiles. A profile is a set of configuration attributes that can be activated/chosen when launching a pipeline execution by using the -profile command line option.

Configuration profiles are defined by using the special scope profiles which group the attributes that belong to the same profile using a common prefix. For example:

This configuration defines three different profiles: standard, cluster and cloud that set different process configuration strategies depending on the target runtime platform. By convention the standard profile is implicitly used when no other profile is specified by the user.

To enable a specific profile use -profile option followed by the profile name:

AWS Batch is a managed computing service that allows the execution of containerised workloads in the Amazon cloud infrastructure.

Nextflow provides a built-in support for AWS Batch which allows the seamless deployment of a Nextflow pipeline in the cloud offloading the process executions as Batch jobs.

Once the Batch environment is configured specifying the instance types to be used and the max number of cpus to be allocated, you need to created a Nextflow configuration file like the one showed below:

The complete details about AWS Batch deployment are available at this link.

EBS volumes (or other supported storage) can be mounted in the job container using the following configuration snippet:

Multiple volumes can be specified using comma-separated paths. The usual Docker volume mount syntax can be used to define complex volumes for which the container paths is different from the host paths or to specify a read-only option:

IMPORTANT:

Nextflow automatically creates the Batch Job definitions needed to execute your pipeline processes. Therefore it’s not required to define them before run your workflow.

However, you may still need to specify a custom Job Definition to provide fine-grained control of the configuration settings of a specific job e.g. to define custom mount paths or other special settings of a Batch Job.

To use your own job definition in a Nextflow workflow, use it in place of the container image name, prefixing it with the job-definition:// string. For example:

Since Nextflow requires the AWS CLI tool to be accessible in the computing environment a common solution consists of creating a custom AMI and install it in a self-contained manner e.g. using Conda package manager.

The following snippet shows how to install AWS CLI with Miniconda:

Finally specify the aws full path in the Nextflow config file as show below:

An alternative to is to create a custom AMI using a Launch template that installs the AWS CLI tool during the instance boot via a custom user-data.

In the EC2 dashboard create a Launch template specifying in the user data field:

Then in the Batch dashboard create a new compute environment and specify the newly created launch template in the corresponding field.

Nextflow allows the use of multiple executors in the same workflow application. This feature enables the deployment of hybrid workloads in which some jobs are execute in the local computer or local computing cluster and some jobs are offloaded to AWS Batch service.

To enable this feature use one or more process selectors in your Nextflow configuration file to apply the AWS Batch configuration only to a subset of processes in your workflow. For example:

The -resume command line option allow the continuation of a pipeline execution since the last step that was successfully completed:

In practical terms the pipeline is executed from the beginning however before launching the execution of a process. Nextflow uses the task unique ID to check if the work directory already exists and it contains a valid command exit status and the expected output files.

If this condition is satisfied the task execution is skipped and previously computed results are used as the process results.

The first task, for which a new output is computed, invalidates all downstream executions in the remaining DAG.

The task work directories are created in the folder work in the launching path by default. This is supposed to be a scratch storage area that can be cleaned up once the computation is completed.

Workflow final output are supposed to the stored in a different location specified using one or more publishDir directive.

A different location for the execution work directory can be specified using the command line option -w e.g.

The hash code for input files is computed using:

The complete file path

The file size

The last modified timestamp

Therefore just touching a file will invalidated the related task execution.

It’s a good practice to organize each experiment in its own folder. The experiment main input parameters should be specified using a Nextflow config file. This makes simply to track and replicate the experiment over time.

Note that in the same experiment the same pipeline can be executed multiple times, however it should be avoided to launch two (or more) Nextflow instances in the same directory concurrently.

The nextflow log command lists the executions run in the current folder:

You can use either the session ID or the run name to recover a specific execution. For example:

The log command when provided with a run name or session ID can return many useful information about a pipeline execution that can be used to create a provenance report.

By default, it lists the work directories used to compute each task. For example:

Using the option -f (fields) it’s possible to specify which metadata should be printed by the log command. For example:

The complete list of available fields can be retrieved with the command:

The option -F allows the specification of a filtering criteria to print only a subset of tasks. For example:

This can be useful to locate specific tasks work directories.

Finally, the -t option allow the creation of a basic custom provenance report proving a template file, in any format of your choice. For example:

Save the above snippet in a file named template.html. Then run this command:

Finally open the file prov.html file with a browser.

If your workflow execution is not resumed as expected and one or more task are re-executed all the times, these may be the most likely causes:

The problem in this snippet is that the X variable in the closure definition is defined in the global scope. Therefore, since operators are executed in parallel, the X value can be overwritten by the other map invocation.

The correct implementation requires the use of the def keyword to declare the variable local.

In practical term, consider the following snippet:

The inputs declared at line 19,20 can be delivered in any order because the execution order of the process foo and bar is not deterministic due to the parallel executions of them.

Therefore the input of the third process needs to be synchronized using the join operator or a similar approach. The third process should be written as:

When a process execution exit with a non-zero exit status, Nextflow stops the workflow execution and report the failing task:

Review carefully all these data, they can provide valuable information on the cause of the error.

If this is not enough, change in the task work directory. It contains all the files to replicate the issue in a isolated manner.

The task execution directory contains these files:

Verify that the .command.sh file contains the expected command to be executed and all variables are correctly resolved.

Also verify the existence of the file .exitcode. If missing and also the file .command.begin does not exist, the task was never executed by the subsystem (eg. the batch scheduler). If the .command.begin file exist, the job was launched but the it likely was abruptly killed.

You can replicate the failing execution using the command bash .command.run and verify the cause of the error.

There are cases in which a process error may be expected and it should not stop the overall workflow execution.

To handle this use case set the process errorStrategy to ignore:

If you want to ignore any error set the same directive in the config file as default setting:

process.errorStrategy = 'ignore'

In (rare) cases errors may be caused by transient conditions. In this situation an effective strategy consists in trying to re-execute the failing task.

Using the retry error strategy the task is re-executed a second time if it returns a non-zero exit status before stopping the complete workflow execution.

The directive maxRetries can be used to set number of attempts the task can be re-execute before declaring it failed with an error condition.

There are cases in which the required execution resources may be temporary unavailable e.g. network congestion. In these cases simply re-executing the same task will likely result in the identical error. A retry with an exponential backoff delay can better recover these error conditions.

It’s a very common scenario that different instances of the same process may have very different needs in terms of computing resources. In such situations requesting, for example, an amount of memory too low will cause some tasks to fail. Instead, using a higher limit that fits all the tasks in your execution could significantly decrease the execution priority of your jobs.

To handle this use case you can use a retry error strategy and increasing the computing resources allocated by the job at each successive attempt.