Nextflow

In Nextflow, pipelines are constructed from individual processes that correspond to computational tasks. Each process is set up with input requirements and output declarations. Rather than running in a fixed succession, the execution of a process commences when all its input requirements are met. By specifying the output of one process as the input of another step, a logical and sequential connection between processes is created.[12]

This reactive implementation of processes is a characteristic design pattern of Nextflow and also known as functional dataflow model.[13] It allows for the efficient parallelization of independent compute steps.

Processes and whole workflows are programmed in a domain-specific language (DSL) that is provided by Nextflow and based on Apache Groovy.[14] While Nextflow's DSL is used to declare the workflow logic, developers can use their scripting language of choice within a process and mix multiple languages in a workflow. Porting existing scripts and workflows to Nextflow is therefore straightforward. Supported scripting languages include bash, csh, ksh, Python, Ruby, and R. Any scripting language that uses the standard Unix shebang declaration (#!/bin/bash) is supported in Nextflow.

An exemplary workflow consisting of only one process is shown below:

process hello_world {
    input:
    val greeting

    output:
    path "${greeting}.txt"

    script:
    """
    echo "${greeting} World!" > ${greeting}.txt
    """
}

workflow {
    Channel.of("Hello", "Ciao", "Hola", "Bonjour") | hello_world
}

To facilitate straightforward collaboration on workflows, Nextflow has native support for source-code management systems and DevOps-platforms including GitHub, GitLab, and others..[15]

Workflows written in Nextflow's DSL can be deployed and run across diverse computing environments without modifications to the pipeline code.

To enable portability, Nextflow ships with dedicated executors for a variety of platforms[16] including those of major cloud providers. Because Nextflow decouples individual process steps, it can optionally be configured to spread execution across multiple computing platforms. It supports the following environments for pipeline execution:

• Local – the default executor. Nextflow pipelines run on Linux or Mac OS and execution occurs on the computer where the pipeline is launched.
• HPC workload managers – Slurm, SGE, LSF, Moab, PBS Pro, PBS/Torque, HTCondor, NQSII, OAR
• Kubernetes – local or cloud-based Kubernetes implementations (GKE, EKS, or AKS)
• Cloud batch services – AWS Batch,[17] Azure Batch[18]
• Other environments – Apache Ignite, Google Life Sciences[19]

A fundamental concept of Nextflow is its tight integration with software containers. Whole workflows and, in later versions, also single processes can harness containers to allow their execution across various compute environments without tedious installation and configuration routines.[3]

This design choice was strongly influenced by Solomon Hyke's talk at dotScale in 2013,[20] which had a significant impact on Nextflow's principal developer, Paolo Di Tommaso.[21]

Container frameworks supported by Nextflow include Docker, Singularity, Charliecloud, Podman, and Shifter.[22] Those type of containers can be utilized in a workflow and are automatically retrieved from external repositories when the pipeline is executed. At Nextflow Summit 2022, it was unveiled that future versions of Nextflow will support a dedicated container provisioning service for an improved integration of customized containers into workflows.[23]

In addition to the Centre for Genomic Regulation,[28] other sequencing facilities quickly adopted Nextflow as their preferred Scientific workflow system, among them the Quantitative Biology Center in Tübingen, the Francis Crick Institute, A*STAR Genome Institute of Singapore, and the Swedish National Genomics Infrastructure.[21]

Efforts to share, harmonize, and curate the bioinformatic pipelines used by those facilities[29][30][31][32] eventually turned into the nf-core project.[33] Spearheaded by Phil Ewels from the Swedish National Genomics Infrastructure[34][35] the focus of the nf-core project is to ensure that pipelines are reproducible and portable across different hardware, operating systems, and software versions. In July 2020, Nextflow and nf-core were awarded a grant from the Chan Zuckerberg Initiative, recognizing its role as a vital open-source software.[36]

As of 2022, the nf-core organization hosts 73 Nextflow pipelines for the biosciences and more than 700 process modules. Uniting more than 500 developers and scientists, it is the largest collaborative effort and community to develop bioinformatic data analysis pipelines.[37]

Nextflow is preferred in sequencing data processing and genomic data analysis. Over the last five years, numerous pipelines for many different applications and analyses in the field of genomics have been published.

A notable use case in this regard was for pathogen surveillance during the COVID-19 pandemic.[38] Monitoring the emergence of new virus variants and retracing its global spread required the swift and highly automatized, yet accurate, processing of raw data, variant analysis, and the designation of lineages, which was enabled by pipelines written in Nextflow.[39] [40] [41] [42] [43] [44] [45]

Nextflow also plays an important role for the non-profit plasmid repository Addgene, which uses it to corroborate the integrity of all deposited plasmids.[46]

Apart from genomics, Nextflow is gaining popularity in other domains of biomedical data processing, which also require the application of complex workflows on large amounts of primary data: Drug screening,[47] Diffusion magnetic resonance imaging (dMRI) in radiology,[48] and mass spectrometry data processing,[49][50][51] the latter with a particular focus on proteomics[52] [53] [54] [55] [56] [57] [58] [59]