Running a pipeline on an HPC

🚀 Start now: — first launch takes a couple of minutes to build.

⏱ Estimated time: ~40 minutes  •  🟡 Practical · optional

🎯 Who is this for? Anyone who will run nf-core pipelines on a High-Performance Computing (HPC) cluster.

You don’t need a cluster for this lesson. In Step 1 you’ll turn your Codespace into a small working cluster, running the same HPC scheduler software (Slurm). Every command you type here is one you’d type on a real cluster. Step 6 then covers what changes when you move to your institution’s machine.

What you’ll learn

• How a cluster works: login node, scheduler, compute nodes
• How to submit, watch and cancel jobs yourself
• How to get a pipeline onto the cluster
• How to run Nextflow on an HPC
• How to keep a long run alive, and running in the background

Step 0 — How a cluster works

So far everything has run on one machine. A cluster is many machines sharing one filesystem. You log in to a shared login node, and a scheduler hands out time on the compute nodes to everyone’s jobs.

Nextflow can talk to the scheduler for you, so you never write submission scripts by hand. The nextflow run process you start (the driver) does no analysis itself: it sends every task to the scheduler as a separate job with the correct resources/time/container, to compute nodes.

flowchart LR
  you["💻 You"] -- "ssh" --> login["🚪 Login node<br/>Nextflow driver<br/>(small, long-running)"]
  login -- "one job per task<br/>(sbatch · qsub)" --> sched["🗂️ Scheduler<br/>Slurm · SGE"]
  sched --> n1["⚙️ Compute node<br/>FASTQC (sample 1)"]
  sched --> n2["⚙️ Compute node<br/>FASTQC (sample 2)"]
  sched --> n3["⚙️ Compute node<br/>MULTIQC"]
  n1 --- fs[("🗄️ Shared filesystem<br/>work/ · results")]
  n2 --- fs
  n3 --- fs

Step 1 — Turn your Codespace into a cluster

Run this from the eco-flow-training folder:

bash hpc/start_slurm.sh

▶ Your practice Slurm cluster is ready 🎉
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
codespace*    up   infinite      1   idle codespaces-abc123

You now have a real Slurm installation: the same software that runs on thousands of HPC systems, with the same commands. What it doesn’t have is other machines, so nothing here runs any faster.

⚠️ If your Codespace restarts, run bash hpc/start_slurm.sh again. It’s safe to re-run at any time.

Step 2 — Submit a job yourself

Before we dive into Nextflow, lets submit a job the old fashioned way with a submission/job script.

In slurm, this will look like the following:.

▶️ Try it — write the script

Create hello_job.sh (nano hello_job.sh):

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=codespace     # on a real cluster: one of your cluster's partitions
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00
#SBATCH --output=hello_%j.log     # %j is replaced by the job number

echo "Running on $(hostname)"
sleep 30
echo "Finished"

The #SBATCH lines are the resource request: how many CPUs, how much memory, how long, and which partition. The scheduler uses them to decide where and when your job runs.

▶️ Try it — submit it and watch it

sbatch hello_job.sh
squeue

Submitted batch job 1
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 1 codespace    hello   vscode  R       0:01      1 codespaces-abc123

▶️ Try it — cancel a job

Submit it again and stop it while it’s still running:

sbatch hello_job.sh
squeue
scancel <JOBID>      # the number sbatch printed
squeue

The job disappears from the queue, and its log stops where you interrupted it.

You’ve just written a job script by hand, easy. But when you run a pipeline, Nextflow writes one of these per task and submits them all for you. That’s where Nextflow excels.

#!/bin/bash
#$ -N hello                # job name
#$ -q all.q                # your queue
#$ -l h_rt=00:05:00        # wall time
#$ -l mem=1G               # memory per core (some clusters use h_vmem)
#$ -cwd                    # run in the directory you submitted from
#$ -j y                    # merge errors into the output file
#$ -o hello.log

echo "Running on $(hostname)"
sleep 30
echo "Finished"

Step 3 — Get a pipeline on the cluster

We’ll use nf-core/demo: a tiny nf-core pipeline (FastQC → trimming with seqtk → MultiQC) that runs in a couple of minutes. You get it the same way as any nf-core pipeline: Nextflow fetches it for you.

▶️ Try it

nextflow pull nf-core/demo -r 1.2.0
nextflow list
nextflow info nf-core/demo

Checking nf-core/demo:1.2.0 ...
 downloaded from https://github.com/nf-core/demo.git - revision: 32893afef8 [1.2.0]

 project name: nf-core/demo
 repository  : https://github.com/nf-core/demo
 local path  : /home/vscode/.nextflow/assets/.repos/nf-core/demo
 main script : main.nf
 revisions   :
   ...

This is what nextflow run nf-core/demo does automatically the first time: it downloads the pipeline from GitHub into ~/.nextflow/assets/, and -r picks the release. To delete a downloaded copy, use nextflow drop nf-core/demo.

⚠️ Always pin the version (see Part 3). Nextflow itself changes too, and a very old pipeline release may not run on a current Nextflow. If an old release stops with Config parsing failed, move -r to a newer release.

git clone --branch 1.2.0 https://github.com/nf-core/demo.git nf_practical/demo
ls nf_practical/demo

CHANGELOG.md  CITATIONS.md  CODE_OF_CONDUCT.md  LICENSE  README.md  assets  conf  docs
main.nf  modules  modules.json  nextflow.config  nextflow_schema.json  nf-test.config
ro-crate-metadata.json  subworkflows  tests  tower.yml  workflows

nextflow run ./nf_practical/demo/main.nf -profile test,docker --outdir demo_results

nextflow config ./nf_practical/demo/main.nf -profile test,docker
nextflow config ./nf_practical/demo/main.nf -profile test,singularity

rm -rf /workspaces/training/eco-flow-training/nf_practical

Step 4 — Let Nextflow submit the jobs

Now let’s try running a pipeline. Start the way you would on a laptop, with no cluster config at all — the pipeline from Step 3, with the test profile for its tiny example data:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir demo_results

Watch the line near the top of the output:

executor >  local (3)

local means Nextflow is running the tasks itself, right here, and nothing has been sent to the scheduler. Check from a second terminal (the ➕ in the terminal panel):

squeue      # none of your jobs are listed

That one line was the whole point, so stop the run with Ctrl+C.

⚠️ On a real HPC this is the mistake to avoid. You’d be logged in to the login node, so executor > local means every step of your pipeline runs there — on the machine everyone shares — and none of the cluster gets used. It’s the first thing to check whenever a cluster run behaves oddly.

Now tell Nextflow about the scheduler

One small config file changes that. It makes Nextflow write the #SBATCH header you wrote in Step 2 — for every task in the pipeline:

cat hpc/slurm_codespaces.config

process {
    executor = 'slurm'
    queue    = 'codespace'    // the Slurm partition start_slurm.sh creates
}

executor = 'slurm' is the whole trick; queue says which partition to use.

Run exactly the same command again, with the config added:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c hpc/slurm_codespaces.config --outdir demo_results

While it runs, open a second terminal (the ➕ in the terminal panel) and watch jobs come and go:

watch -n 2 squeue      # Ctrl+C to stop watching

✅ What to look for: the line executor > slurm. That means every task is going to the scheduler as its own job. The run takes a few minutes (longer the first time, while containers download) and ends like this:

executor >  slurm (8)
[f4/044c12] NFCORE_DEMO:DEMO:COWPY                   [100%] 1 of 1 ✔
[94/ace98a] NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE)     [100%] 3 of 3 ✔
[68/46c1d9] NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE) [100%] 3 of 3 ✔
[be/7be914] NFCORE_DEMO:DEMO:MULTIQC (demo)          [100%] 1 of 1 ✔
-[nf-core/demo] Pipeline completed successfully-

If it says executor > local, the config wasn’t picked up and the tasks ran outside the scheduler. Check the -c hpc/slurm_codespaces.config path.

grep "^#SBATCH" work/94/ace98a*/.command.run     # use your own FASTQC hash

#SBATCH -J nf-NFCORE_DEMO_DEMO_FASTQC_(SAMPLE2_PE)
#SBATCH -o /workspaces/training/eco-flow-training/work/94/ace98a.../.command.log
#SBATCH --no-requeue
#SBATCH --signal B:USR2@30
#SBATCH -c 2
#SBATCH -t 01:00:00
#SBATCH --mem 4096M
#SBATCH -p codespace

process FASTQC {
    tag "${meta.id}"
    label 'process_low'

withLabel:process_low {
    cpus   = { 2     * task.attempt }
    memory = { 12.GB * task.attempt }
    time   = { 4.h   * task.attempt }
}

process {
    executor = 'sge'
    queue    = 'all.q'        // your queue
    penv     = 'smp'          // the parallel environment name — ask your HPC team
}

Step 5 — Keep the run alive

A real analysis runs for hours or days, and the driver has to keep running the whole time. If you close your laptop or lose your connection, nextflow run stops, and the pipeline stops with it. There are two ways to keep it going, and you’ll mostly use the first.

1. Submit the driver as a job — the right choice for anything long. It works on every cluster, and on the ones that forbid long-running processes on the login node it’s the only option. The driver becomes a small job of its own, which is what hpc/run_demo_slurm.sh does:

cat hpc/run_demo_slurm.sh
sbatch hpc/run_demo_slurm.sh
watch -n 2 squeue      # Ctrl+C to stop watching

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                12 codespace nf-NFCOR   vscode  R       0:07      1 codespaces-abc123
                13 codespace nf-NFCOR   vscode  R       0:07      1 codespaces-abc123
                11 codespace nf_drive   vscode  R       0:24      1 codespaces-abc123

▶️ Try it — stop a run, then resume it

1. While pipeline jobs are running, cancel the driver: scancel <driver JOBID>. Check squeue: its pipeline jobs go too, because Nextflow cancels its own jobs when it stops.
2. Submit it again: sbatch hpc/run_demo_slurm.sh (the script already includes -resume).
3. When it finishes, find the steps that were reused: grep -i cached nf_driver_<new JOBID>.log

⚠️ One driver at a time in a folder. Each sbatch hpc/run_demo_slurm.sh (or -bg run, below) starts another Nextflow run, and two runs launched from the same folder share the same cache, so they get in each other’s way. If squeue shows two nf_driver jobs, scancel the extra one.

2. Run it in the background with -bg — simpler, and fine for shorter runs on clusters that allow them. Add the flag and Nextflow detaches itself: you get your prompt back, and you can close your terminal without stopping the run.

▶️ Try it — once squeue shows no nf_driver job, run Step 4’s command with -bg on the end:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c hpc/slurm_codespaces.config --outdir demo_results -bg > bg_run.log
tail -f bg_run.log      # Ctrl+C stops watching, not the run

Nextflow still prints its progress, so send it to a file (> bg_run.log), or it keeps appearing in your terminal. Check on it with tail -f bg_run.log whenever you log back in. To stop the run early, use kill $(cat .nextflow.pid): Nextflow saves its process ID in .nextflow.pid when you use -bg.

Step 6 — Taking this to your own cluster

Everything so far works the same on a real HPC. These are the differences.

1. Getting on, and checking what’s there

ssh your_username@your.cluster.ac.uk   # log in to the login node
module avail nextflow                  # is Nextflow provided as a module?
module load nextflow                   # load it (names vary between clusters)
nextflow -version
java -version                          # Nextflow needs Java 17 or newer
singularity --version || apptainer --version

• No Nextflow module? You can usually install it in your home directory: see the official install guide.
• Singularity, not Docker. Docker needs a background service running with root privileges, which shared clusters can’t hand out. So where this lesson used -profile docker, you’ll use -profile singularity (or apptainer). Both run the same containers.

2. Is there already a config for your cluster?

This is the most useful thing to check. The nf-core community keeps ready-made configs for around 160 institutions at nf-co.re/configs, each describing that cluster’s scheduler, container engine and limits — so you don’t have to write the executor/queue config from Step 4 yourself. The page lists them by name: cambridge, crick, eddie, imperial, sanger, ucl_myriad and many more.

Each row is one cluster, and the Executor column tells you which scheduler it uses — a quick way to see how common Slurm and SGE are in practice.

If yours is listed, add it as a profile and Nextflow knows how to talk to your cluster:

nextflow run nf-core/demo -r 1.2.0 -profile test,<your_cluster> --outdir demo_results

That’s the -profile ucl_myriad you saw in Part 6. Open your cluster’s page first — most list setup steps, such as which Java module to load.

3. Where your files should be

⚠️ Ask where large data should live — don’t assume. Many clusters have a “scratch” area for this, but not all, and the rules differ: some aren’t backed up, some delete files you haven’t touched for a few weeks. Your HPC team, or your cluster’s page on nf-co.re/configs, will tell you. Two things to remember once you know: copy results somewhere safe, and if work/ is deleted, -resume has nothing left to resume from.

4. Your own driver script

On your cluster, write your own version of Step 5’s hpc/run_demo_slurm.sh: your partition or queue, a module load line if your cluster uses modules, -profile singularity,<your_cluster>, and -w pointing at wherever your cluster keeps large data:

#!/bin/bash
#SBATCH --job-name=nf_driver
#SBATCH --partition=<your_partition>
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=48:00:00
#SBATCH --output=nf_driver_%j.log

module load nextflow                  # if your cluster uses modules

nextflow run nf-core/demo -r 1.2.0 \
  -profile singularity,<your_cluster> \
  --outdir /path/to/results \
  -w /path/with/space/work \
  -ansi-log false \
  -resume

#!/bin/bash -l
#$ -N nf_driver
#$ -l h_rt=48:00:00
#$ -l mem=4G                          # some SGE clusters use h_vmem instead
#$ -cwd
#$ -o nf_driver.log
#$ -j y

module load nextflow                  # if your cluster uses modules

nextflow run nf-core/demo -r 1.2.0 \
  -profile singularity,<your_cluster> \
  --outdir /path/to/results \
  -w /path/with/space/work \
  -ansi-log false \
  -resume

For a shorter run, the same nextflow run line with -bg works too (Step 5).

git clone https://github.com/Eco-Flow/training.git
cd training/eco-flow-training

# The samplesheet from Part 3, with paths for wherever you cloned it
cat > samplesheet.csv <<EOF
sample,fastq_1,fastq_2,strandedness
CONTROL_REP1,$PWD/data/SRR6357070_1.fastq.gz,$PWD/data/SRR6357070_2.fastq.gz,auto
CONTROL_REP2,$PWD/data/SRR6357071_1.fastq.gz,$PWD/data/SRR6357071_2.fastq.gz,auto
CONTROL_REP3,$PWD/data/SRR6357072_1.fastq.gz,$PWD/data/SRR6357072_2.fastq.gz,auto
MANIPULATED_REP1,$PWD/data/SRR6357073_1.fastq.gz,,auto
MANIPULATED_REP2,$PWD/data/SRR6357074_1.fastq.gz,,auto
MANIPULATED_REP3,$PWD/data/SRR6357075_1.fastq.gz,,auto
EOF

# The genome and annotation, as in Part 3 Step 4
wget -O genome.fasta https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/reference/genome.fasta
wget -O genes.gff.gz https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/reference/genes.gff.gz

nextflow run nf-core/rnaseq -r 3.26.0 \
  -profile singularity,<your_cluster> \
  --input $PWD/samplesheet.csv \
  --fasta $PWD/genome.fasta \
  --gff $PWD/genes.gff.gz \
  --outdir rnaseq_results \
  -resume

5. What the practice cluster couldn’t show you

• Queues and limits. Real clusters have several partitions/queues with maximum times and sizes, and jobs asking for more are rejected. Your HPC team’s documentation lists them.
• Waiting. On a busy cluster, jobs sit in PD/qw for a while before they start. That’s normal.
• Real parallelism. Here everything shared one machine; there, tasks run on many machines at once.
• Internet access. Compute nodes often have none. Pull pipelines and containers on the login node first: nextflow pull, or nf-core pipelines download for the containers too.

Step 7 — Common issues list

🔍 Exit codes 130–145 (for example 137 or 140) usually mean the scheduler killed the job for going over its memory or time. nf-core pipelines retry once with double the resources (Step 4). If a step fails again, it needs more than it’s allowed: ask your HPC team, or see how to raise a label’s resources on the ★ advanced page.

Recap

• A cluster is a login node plus a scheduler handing out compute nodes. You send it jobs.
• A job is a script with a resource request in its header (#SBATCH, or #$ on SGE). You wrote one in Step 2.
• Nextflow writes those scripts for you: executor = 'slurm' (or 'sge') is all it takes, and each step’s CPUs and memory come from its label in conf/base.config.
• Smoke-test with the test profile and check the banner says executor > slurm, not local.
• Keep the driver alive, as its own job or with -bg, and use -resume after any interruption.
• On your own cluster: load Nextflow, use Singularity, and check nf-co.re/configs for a ready-made profile before writing anything yourself.

Finish

🎉 You’ve run a pipeline through a real job scheduler, and seen how Nextflow turns each step into a job — exactly as it will on your institution’s cluster.

Next steps:

• Continue to Part 9 · Seqera Platform ▶️ to watch your runs live in the browser.
• Your cluster has no ready-made config? See ★ Advanced: setting up Nextflow for your HPC.
• Stuck? The nf-core Slack is full of people running pipelines on clusters like yours. Or get in touch with us at Eco-Flow: ecoflow . ucl @ gmail . com