Running a pipeline on an HPC

🚀 Start now: — first launch takes a couple of minutes to build.

⏱ Estimated time: ~40 minutes  •  🟡 Practical · optional

🎯 Who is this for? Anyone who will run nf-core pipelines on a High-Performance Computing (HPC) cluster.

You don’t need a cluster for this lesson. In Step 1 you’ll turn your Codespace into a small working cluster, running the same HPC scheduler software (Slurm). Every command you type here is one you’d type on a real cluster. Step 6 then covers what changes when you move to your institution’s machine.

What you’ll learn

• How a cluster works: login node, scheduler, compute nodes
• How to submit, watch and cancel jobs yourself
• How to get a pipeline onto the cluster
• How to run Nextflow on an HPC
• How to keep a long run alive, and running in the background

Step 0 — How a cluster works

So far everything has run on one machine. A cluster is many machines sharing one filesystem. You log in to a shared login node, and a scheduler hands out time on the compute nodes to everyone’s jobs.

Nextflow can talk to the scheduler for you, so you never write submission scripts by hand. The nextflow run process you start (the driver) does no analysis itself: it sends every task to the scheduler as a separate job with the correct resources/time/container, to compute nodes.

flowchart LR
  you["💻 You"] -- "ssh" --> login["🚪 Login node<br/>Nextflow driver<br/>(small, long-running)"]
  login -- "one job per task<br/>(sbatch · qsub)" --> sched["🗂️ Scheduler<br/>Slurm · SGE"]
  sched --> n1["⚙️ Compute node<br/>FASTQC (sample 1)"]
  sched --> n2["⚙️ Compute node<br/>FASTQC (sample 2)"]
  sched --> n3["⚙️ Compute node<br/>MULTIQC"]
  n1 --- fs[("🗄️ Shared filesystem<br/>work/ · results")]
  n2 --- fs
  n3 --- fs

Step 1 — Turn your Codespace into a cluster

Run this from the eco-flow-training folder:

bash hpc/start_slurm.sh

▶ Your practice Slurm cluster is ready 🎉
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
codespace*    up   infinite      1   idle codespaces-abc123

You now have a real Slurm installation: the same software that runs on thousands of HPC systems, with the same commands. What it doesn’t have is other machines, so nothing here runs any faster.

⚠️ If your Codespace restarts, run bash hpc/start_slurm.sh again. It’s safe to re-run at any time.

Step 2 — Submit a job yourself

Before we dive into Nextflow, lets submit a job the old fashioned way with a submission/job script.

In slurm, this will look like the following:.

▶️ Try it — write the script

Create hello_job.sh (nano hello_job.sh):

#!/bin/bash
#SBATCH --job-name=hello
#SBATCH --partition=codespace     # on a real cluster: one of your cluster's partitions
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00
#SBATCH --output=hello_%j.log     # %j is replaced by the job number

echo "Running on $(hostname)"
sleep 30
echo "Finished"

The #SBATCH lines are the resource request: how many CPUs, how much memory, how long, and which partition. The scheduler uses them to decide where and when your job runs.

▶️ Try it — submit it and watch it

sbatch hello_job.sh
squeue

Submitted batch job 1
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 1 codespace    hello   vscode  R       0:01      1 codespaces-abc123

▶️ Try it — cancel a job

Submit it again and stop it while it’s still running:

sbatch hello_job.sh
squeue
scancel <JOBID>      # the number sbatch printed
squeue

The job disappears from the queue, and its log stops where you interrupted it.

You’ve just written a job script by hand, easy. But when you run a pipeline, Nextflow writes one of these per task and submits them all for you. That’s where Nextflow excels.

The commands, on both common schedulers

Your cluster may use SGE instead of Slurm. The ideas are identical, the names differ:

And the job states you’ll see most often:

#!/bin/bash
#$ -N hello                # job name
#$ -q all.q                # your queue
#$ -l h_rt=00:05:00        # wall time
#$ -l mem=1G               # memory per core (some clusters use h_vmem)
#$ -cwd                    # run in the directory you submitted from
#$ -j y                    # merge errors into the output file
#$ -o hello.log

echo "Running on $(hostname)"
sleep 30
echo "Finished"

Step 3 — Get a pipeline on the cluster

There are two ways to get a pipeline.

We’ll use nf-core/demo: a tiny nf-core pipeline (FastQC → trimming with seqtk → MultiQC) that runs in a couple of minutes. This is the same for any nf-core pipeline!

Way A — let Nextflow fetch it

▶️ Try it

nextflow pull nf-core/demo -r 1.2.0
nextflow list
nextflow info nf-core/demo

Checking nf-core/demo:1.2.0 ...
 downloaded from https://github.com/nf-core/demo.git - revision: 32893afef8 [1.2.0]

 project name: nf-core/demo
 repository  : https://github.com/nf-core/demo
 local path  : /home/vscode/.nextflow/assets/.repos/nf-core/demo
 main script : main.nf
 revisions   :
   ...

This is what nextflow run nf-core/demo does automatically the first time: it downloads the pipeline from GitHub into ~/.nextflow/assets/, and -r picks the release. To delete a downloaded copy, use nextflow drop nf-core/demo.

That copy is Nextflow’s to manage — it’s filed away under a long path like ~/.nextflow/assets/.repos/nf-core/demo/clones/<commit>/, which is fine for running but awkward for reading. When you want to look at a pipeline’s code, clone it instead.

Way B — clone it yourself

Do this one too — Step 4 runs this copy.

▶️ Try it

mkdir -p ./nf_practical      # a folder of its own
cd ./nf_practical
git clone https://github.com/nf-core/demo.git
cd demo
git checkout 1.2.0
ls

CHANGELOG.md  CITATIONS.md  CODE_OF_CONDUCT.md  LICENSE  README.md  assets  conf  docs
main.nf  modules  modules.json  nextflow.config  nextflow_schema.json  nf-test.config
ro-crate-metadata.json  subworkflows  tests  tower.yml  workflows

Keep the clone — Step 4 runs it. For now, go back to the course folder, so everything below runs from one place:

cd /workspaces/training/eco-flow-training

With a clone, you point Nextflow at the main.nf rather than the pipeline name (“nf/core/demo”).

⚠️ Always pin the version (see Part 3). Nextflow itself changes too, and a very old pipeline release may not run on a current Nextflow. If an old release stops with Config parsing failed, move -r to a newer release.

Who decides the CPUs and memory?

Every step asks the scheduler for CPUs, memory and time. Those numbers come from the pipeline itself, in two parts.

First, each step carries a label. Open a module in the clone you just made — nf_practical/demo/modules/nf-core/fastqc/main.nf — and the third line says:

process FASTQC {
    tag "${meta.id}"
    label 'process_low'

Second, the pipeline’s conf/base.config turns each label into a resource request:

withLabel:process_low {
    cpus   = { 2     * task.attempt }
    memory = { 12.GB * task.attempt }
    time   = { 4.h   * task.attempt }
}

So FASTQC asks for 2 CPUs, 12 GB and 4 hours, and on a cluster that becomes the job’s request: it waits until a node with 12 GB free is available. You’ll see the request Nextflow actually sends in the next step.

💡 task.attempt is 1 on the first try and 2 on a retry, so a step that gets killed for using too much memory automatically asks for double next time. (nf-core pipelines retry on exit codes 130–145 — the codes schedulers use when they kill a job.)

nextflow config ./nf_practical/demo/main.nf -profile test,docker
nextflow config ./nf_practical/demo/main.nf -profile test,singularity

Step 4 — Let Nextflow submit the jobs

Now let’s try running a pipeline. Start the way you would on a laptop, with no cluster config at all — the same copy you cloned in Step 3, and the test profile for its tiny example data:

nextflow run ./nf_practical/demo/main.nf -profile test,docker --outdir demo_results

Watch the line near the top of the output:

executor >  local (3)

local means Nextflow is running the tasks itself, right here, and nothing has been sent to the scheduler. Check from a second terminal (the ➕ in the terminal panel):

squeue      # none of your jobs are listed

That one line was the whole point, so stop the run with Ctrl+C.

⚠️ On a real HPC this is the mistake to avoid. You’d be logged in to the login node, so executor > local means every step of your pipeline runs there — on the machine everyone shares — and none of the cluster gets used. It’s the first thing to check whenever a cluster run behaves oddly.

Now tell Nextflow about the scheduler

One small config file changes that. It makes Nextflow write the #SBATCH header you wrote in Step 2 — for every task in the pipeline:

cat hpc/slurm_codespaces.config

process {
    executor = 'slurm'
    queue    = 'codespace'    // the Slurm partition start_slurm.sh creates
}

executor = 'slurm' is the whole trick; queue says which partition to use.

Run exactly the same command again, with the config added:

nextflow run ./nf_practical/demo/main.nf -profile test,docker -c hpc/slurm_codespaces.config --outdir demo_results

(No -r here: the clone is already fixed at release 1.2.0 by the git checkout you did. Running the downloaded copy instead — nextflow run nf-core/demo -r 1.2.0 … — does exactly the same thing.)

While it runs, open a second terminal (the ➕ in the terminal panel) and watch jobs come and go:

watch -n 2 squeue      # Ctrl+C to stop watching

✅ What to look for: the line executor > slurm. That means every task is going to the scheduler as its own job. The run takes a few minutes (longer the first time, while containers download) and ends like this:

executor >  slurm (8)
[f4/044c12] NFCORE_DEMO:DEMO:COWPY                   [100%] 1 of 1 ✔
[94/ace98a] NFCORE_DEMO:DEMO:FASTQC (SAMPLE3_SE)     [100%] 3 of 3 ✔
[68/46c1d9] NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE3_SE) [100%] 3 of 3 ✔
[be/7be914] NFCORE_DEMO:DEMO:MULTIQC (demo)          [100%] 1 of 1 ✔
-[nf-core/demo] Pipeline completed successfully-

If it says executor > local, the config wasn’t picked up and the tasks ran outside the scheduler. Check the -c hpc/slurm_codespaces.config path.

▶️ Challenge — what did Nextflow ask Slurm for?

Nextflow writes a job script for every task: the .command.run file you met in Part 3. Take the hash from a FASTQC line of your output and look at the top of its script:

grep "^#SBATCH" work/94/ace98a*/.command.run     # use your own FASTQC hash

In Step 3 you found FASTQC wants 2 CPUs, 12 GB and 4 hours. Does the job ask for that? If not, why not?

#SBATCH -J nf-NFCORE_DEMO_DEMO_FASTQC_(SAMPLE2_PE)
#SBATCH -o /workspaces/training/eco-flow-training/work/94/ace98a.../.command.log
#SBATCH --no-requeue
#SBATCH --signal B:USR2@30
#SBATCH -c 2
#SBATCH -t 01:00:00
#SBATCH --mem 4096M
#SBATCH -p codespace

process {
    executor = 'sge'
    queue    = 'all.q'        // your queue
    penv     = 'smp'          // the parallel environment name — ask your HPC team
}

Step 5 — Keep the run alive

A real analysis runs for hours or days, and the driver has to keep running the whole time. If you close your laptop or lose your connection, nextflow run stops, and the pipeline stops with it.

The usual answer is to submit the driver itself as a small job. That’s what hpc/run_demo_slurm.sh does:

cat hpc/run_demo_slurm.sh
sbatch hpc/run_demo_slurm.sh
watch -n 2 squeue      # Ctrl+C to stop watching

             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                12 codespace nf-NFCOR   vscode  R       0:07      1 codespaces-abc123
                13 codespace nf-NFCOR   vscode  R       0:07      1 codespaces-abc123
                11 codespace nf_drive   vscode  R       0:24      1 codespaces-abc123

▶️ Try it — stop a run, then resume it

1. While pipeline jobs are running, cancel the driver: scancel <driver JOBID>. Check squeue: its pipeline jobs go too, because Nextflow cancels its own jobs when it stops.
2. Submit it again: sbatch hpc/run_demo_slurm.sh (the script already includes -resume).
3. When it finishes, find the steps that were reused: grep -i cached nf_driver_<new JOBID>.log

⚠️ One driver at a time in a folder. Each sbatch hpc/run_demo_slurm.sh starts another Nextflow run, and two runs launched from the same folder share the same work directory and cache, so they get in each other’s way. If squeue shows two nf_driver jobs, scancel the extra one.

Notice the script runs nf-core/demo -r 1.2.0 rather than your clone: a job script should stand on its own, so it names the pipeline and its version instead of depending on a folder that might move. That’s how you’d write it on a real cluster too.

One line in that script is worth a look: -w work_slurm. -w chooses where Nextflow keeps its working files, and this run keeps them separate from Step 4’s rather than mixing the two together. It doesn’t stop -resume reusing work: that cache lives in the folder you launched from, so steps Step 4 already finished can show up here as Cached process > lines. On a cluster, -w matters for a different reason — those files get very large, so you point it at wherever your cluster keeps big data.

Step 6 — Taking this to your own cluster

Everything so far works the same on a real HPC. These are the differences.

1. Getting on, and checking what’s there

ssh your_username@your.cluster.ac.uk   # log in to the login node
module avail nextflow                  # is Nextflow provided as a module?
module load nextflow                   # load it (names vary between clusters)
nextflow -version
java -version                          # Nextflow needs Java 17 or newer
singularity --version || apptainer --version

• No Nextflow module? You can usually install it in your home directory: see the official install guide.
• Singularity, not Docker. Docker needs a background service running with root privileges, which shared clusters can’t hand out. So where this lesson used -profile docker, you’ll use -profile singularity (or apptainer). Both run the same containers.

2. Is there already a config for your cluster?

This is the most useful thing to check. The nf-core community keeps ready-made configs for around 160 institutions at nf-co.re/configs, each describing that cluster’s scheduler, container engine and limits — so you don’t have to write the executor/queue config from Step 4 yourself. The page lists them by name: cambridge, crick, eddie, imperial, sanger, ucl_myriad and many more.

Each row is one cluster, and the Executor column tells you which scheduler it uses — a quick way to see how common Slurm and SGE are in practice.

If yours is listed, add it as a profile and Nextflow knows how to talk to your cluster:

nextflow run nf-core/demo -r 1.2.0 -profile test,<your_cluster> --outdir demo_results

That’s the -profile ucl_myriad you saw in Part 5. Open your cluster’s page first — most list setup steps, such as which Java module to load.

Not listed? In order:

1. Ask around your institution. Someone in a neighbouring group often has a working config already.
2. Ask your HPC team. They may not know Nextflow, but they can tell you the scheduler, queue names and limits, which is most of what a config needs.
3. Ask the community. The nf-core Slack is friendly, and you can open an issue on nf-core/configs describing your cluster.
4. Ask us. Email Eco-Flow at ecoflow . ucl @ gmail . com and we’ll help you put one together.

Writing one yourself is covered in ★ Advanced: setting up Nextflow for your HPC.

3. Where your files should be

⚠️ Ask where large data should live — don’t assume. Many clusters have a “scratch” area for this, but not all, and the rules differ: some aren’t backed up, some delete files you haven’t touched for a few weeks. Your HPC team, or your cluster’s page on nf-co.re/configs, will tell you. Two things to remember once you know: copy results somewhere safe, and if work/ is deleted, -resume has nothing left to resume from.

4. Keeping the driver alive

There are two ways worth knowing, and you’ll mostly use the first.

1. Submit the driver as a job — Step 5’s approach, and the right one for anything long. It works on every cluster, and on the ones that forbid long-running processes on the login node it’s the only option. Write a small run.sh and submit it:

#!/bin/bash
#SBATCH --job-name=nf_driver
#SBATCH --partition=<your_partition>
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=48:00:00
#SBATCH --output=nf_driver_%j.log

module load nextflow                  # if your cluster uses modules

nextflow run nf-core/demo -r 1.2.0 \
  -profile singularity,<your_cluster> \
  --outdir /path/to/results \
  -w /path/with/space/work \
  -ansi-log false \
  -resume

#!/bin/bash -l
#$ -N nf_driver
#$ -l h_rt=48:00:00
#$ -l mem=4G                          # some SGE clusters use h_vmem instead
#$ -cwd
#$ -o nf_driver.log
#$ -j y

module load nextflow                  # if your cluster uses modules

nextflow run nf-core/demo -r 1.2.0 \
  -profile singularity,<your_cluster> \
  --outdir /path/to/results \
  -w /path/with/space/work \
  -ansi-log false \
  -resume

2. Run it in the background with -bg — simpler, and fine for shorter runs on clusters that allow them. Add the flag and Nextflow detaches itself: you get your prompt back, and you can close your terminal without stopping the run.

nextflow run nf-core/demo -r 1.2.0 \
  -profile singularity,<your_cluster> \
  --outdir /path/to/results \
  -w /path/with/space/work \
  -resume -bg

Progress goes to .nextflow.log, so check on it whenever you log back in:

tail -f .nextflow.log

git clone https://github.com/Eco-Flow/training.git
cd training/eco-flow-training

# The samplesheet from Part 3, with paths for wherever you cloned it
cat > samplesheet.csv <<EOF
sample,fastq_1,fastq_2,strandedness
CONTROL_REP1,$PWD/data/SRR6357070_1.fastq.gz,$PWD/data/SRR6357070_2.fastq.gz,auto
CONTROL_REP2,$PWD/data/SRR6357071_1.fastq.gz,$PWD/data/SRR6357071_2.fastq.gz,auto
CONTROL_REP3,$PWD/data/SRR6357072_1.fastq.gz,$PWD/data/SRR6357072_2.fastq.gz,auto
MANIPULATED_REP1,$PWD/data/SRR6357073_1.fastq.gz,,auto
MANIPULATED_REP2,$PWD/data/SRR6357074_1.fastq.gz,,auto
MANIPULATED_REP3,$PWD/data/SRR6357075_1.fastq.gz,,auto
EOF

# The genome and annotation, as in Part 3 Step 4
wget -O genome.fasta https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/reference/genome.fasta
wget -O genes.gff.gz https://raw.githubusercontent.com/nf-core/test-datasets/7f1614baeb0ddf66e60be78c3d9fa55440465ac8/reference/genes.gff.gz

nextflow run nf-core/rnaseq -r 3.26.0 \
  -profile singularity,<your_cluster> \
  --input $PWD/samplesheet.csv \
  --fasta $PWD/genome.fasta \
  --gff $PWD/genes.gff.gz \
  --outdir rnaseq_results \
  -resume

5. What the practice cluster couldn’t show you

• Queues and limits. Real clusters have several partitions/queues with maximum times and sizes, and jobs asking for more are rejected. Your HPC team’s documentation lists them.
• Waiting. On a busy cluster, jobs sit in PD/qw for a while before they start. That’s normal.
• Real parallelism. Here everything shared one machine; there, tasks run on many machines at once.
• Internet access. Compute nodes often have none. Pull pipelines and containers on the login node first: nextflow pull, or nf-core pipelines download for the containers too.

Step 7 — Common issues list

🔍 Exit codes 130–145 (for example 137 or 140) usually mean the scheduler killed the job for going over its memory or time. nf-core pipelines retry once with double the resources (Step 3). If a step fails again, it needs more than it’s allowed: ask your HPC team, or see how to raise a label’s resources on the ★ advanced page.

Recap

• A cluster is a login node plus a scheduler handing out compute nodes. You send it jobs.
• A job is a script with a resource request in its header (#SBATCH, or #$ on SGE). You wrote one in Step 2.
• Nextflow writes those scripts for you: executor = 'slurm' (or 'sge') is all it takes, and each step’s CPUs and memory come from its label in conf/base.config.
• Smoke-test with the test profile and check the banner says executor > slurm, not local.
• Keep the driver alive as its own job, and use -resume after any interruption.
• On your own cluster: load Nextflow, use Singularity, and check nf-co.re/configs for a ready-made profile before writing anything yourself.

Finish

🎉 You’ve run a pipeline through a real job scheduler, and seen how Nextflow turns each step into a job — exactly as it will on your institution’s cluster.

🧹 Tidying up. The practice clone from Step 3 is no longer needed once you’ve finished the lesson. Check the path before pressing enter — rm -rf deletes without asking:

rm -rf /workspaces/training/eco-flow-training/nf_practical

Next steps:

• Continue to Part 8 · Seqera Platform ▶️ to watch your runs live in the browser.
• Your cluster has no ready-made config? See ★ Advanced: setting up Nextflow for your HPC.
• Stuck? The nf-core Slack is full of people running pipelines on clusters like yours. Or get in touch with us at Eco-Flow: ecoflow . ucl @ gmail . com