Advanced: setting up Nextflow for your HPC

🚀 Start now: — first launch takes a couple of minutes to build.

⏱ Estimated time: ~20 minutes reading  •  🟣 Advanced / optional  •  Reference, not a hands-on run

👋 Just want to run a pipeline on your cluster? You probably want Part 7 · Running a pipeline on an HPC instead. This page is only needed when your cluster doesn’t have a working Nextflow config yet.

🎯 Who is this for? The person setting up Nextflow on a High-Performance Computing (HPC) cluster that has no ready-made config: a researcher working with their HPC team, or the admin themselves. This is a conceptual guide with pointers, not a step-by-step you can run here (every cluster is different).

What you’ll learn

• Why running on an HPC needs a config
• Things to find out from your HPC team before you start
• Where to find existing configs you can reuse or adapt
• How to sketch a minimal config (with optional SGE and Slurm examples)
• How to test a config on a tiny pipeline before a real run
• What to do once it works: share it, and run pipelines as in Part 7

Background

Remember from Part 2 that Nextflow separates what the pipeline does from where it runs. That’s the whole trick here: to move a pipeline onto an HPC you do not change the pipeline at all. You just give Nextflow a configuration describing your cluster — which job scheduler it uses, what resources to request, and which container engine to use.

You’ve actually done this already! In Part 3 you ran with -c .../codespaces.config to cap memory. An HPC config is the exact same idea, just describing a cluster instead of a Codespace.

The two things that change on an HPC are:

Both are set in a config file. The rest of this page is about writing (or borrowing) that file.

Step 1 — Talk to your HPC admin 🧑‍💼

This is the most important step, and it’s mostly non-technical. Every cluster is configured differently, and your admin/support team already knows the answers. Ask them:

• Which scheduler does the cluster use? (Slurm, SGE/UGE, PBS/Torque, LSF…) — this sets the Nextflow executor.
• Which container engine is available? (Almost always Singularity or Apptainer; Docker is usually forbidden on shared clusters.)
• How do I load software — is there a module system? (e.g. module load nextflow, module load singularity), or can I download my own software.
• Which queues/partitions should I submit to, and what are the limits (max CPUs, memory, wall-time per job)?
• Where should work/scratch data go? Clusters usually have a fast scratch filesystem for the (large) Nextflow work/ directory.
• Do compute nodes have internet access? If not, you’ll need to pre-download pipelines and containers on a login node first.
• Is there an account/project code you must charge jobs to?

💡 Write these answers down — every one maps directly to a line in your config below.

In most cases your HPC administrator should be able to help you set all these settings and help make a config for you. But if this is not the case, please shout out to us as Eco-flow, and we will try our best to help get you set up. Also take advantage of the nf-core config repo (https://github.com/nf-core/configs). Here you can even write up an issue and other members of the community should be able to help you build the config.

Step 2 — Check for an existing config (don’t reinvent the wheel) ♻️

Before writing anything, check whether someone has already made a config for your cluster. The nf-core community maintains a big collection of institutional configs:

• 🌐 nf-core/configs: https://nf-co.re/configs
• 📦 The repo: https://github.com/nf-core/configs

If your institution is listed, you may be able to run with nothing more than a profile name:

nextflow run nf-core/rnaseq -profile singularity,<your_institution> ...

That single -profile pulls in a ready-made, tested config for your cluster. Always check here first — it can save you hours.

Step 3 — Find a similar config to adapt 🔎

No config for your exact cluster? Find the closest match and adapt it. In the nf-core/configs conf/ folder there are dozens of real institutional configs. Open a few that use the same scheduler as yours (e.g. search the files for executor = 'slurm') and use them as a template — they show realistic queue names, resource limits and container settings.

📎 Copy the structure, then swap in the specifics your admin gave you in Step 1.

Step 4 — Sketch a minimal config ✍️

A custom config is just a text file (e.g. mycluster.config). At its simplest it sets the executor and enables your container engine:

// mycluster.config
process {
    executor = 'slurm'        // <-- your scheduler: 'slurm', 'sge', 'pbs', ...
    queue    = 'compute'      // <-- your queue/partition name / Or maybe you don't have a specific queue name
}

// Be a good cluster citizen: don't flood the scheduler with submissions
executor {
    queueSize       = 50          // max jobs queued at once
    submitRateLimit = '10/1min'   // max submissions per minute
}

// Use Singularity/Apptainer instead of Docker
singularity {
    enabled    = true
    autoMounts = true
}

You then also point Nextflow at a cache directory for containers (so they’re downloaded once), usually via an environment variable set before you run:

export NXF_SINGULARITY_CACHEDIR=/path/to/shared/singularity_cache

process {
    executor = 'slurm'
    queue    = 'compute'                 // your partition
    // Extra scheduler flags go here, e.g. an account/project to charge:
    clusterOptions = '--account=my_project'
}

executor {
    queueSize       = 100
    submitRateLimit = '10/1min'
}

singularity {
    enabled    = true
    autoMounts = true
}

process {
    executor = 'sge'
    queue    = 'all.q'                   // your queue
    penv     = 'smp'                     // parallel environment name — ASK YOUR ADMIN
    // SGE-specific resource requests often go through clusterOptions:
    clusterOptions = '-l h_vmem=8G'
}

executor {
    queueSize       = 100
    submitRateLimit = '10/1min'
}

singularity {
    enabled    = true
    autoMounts = true
}

🔧 Tune resources with process labels. nf-core pipelines tag processes with labels like process_low, process_medium, process_high. You can override the resources for each label in your config (e.g. give process_high more memory) — see the nf-core tuning docs.

Step 5 — Get Nextflow and containers ready on the cluster 📦

On many HPCs, there is a way of downloaded approved software, usually something like :

module load nextflow      # or install per https://www.nextflow.io/docs/latest/install.html
module load singularity   # or apptainer — whatever your cluster provides
java -version             # Nextflow needs Java 17+

Or you may be able to install nextflow yourself, following the official guidelines (https://docs.seqera.io/nextflow/install). Then you should have nextflow in your PATH, accessible from any directory you are in, by moving the executable “nextflow” to an executable path, e.g. $HOME/.local/bin/. This requires internet.

If the compute nodes have no internet, pre-fetch the pipeline and its containers on the login node first (nf-core provides nf-core download for exactly this).

Step 6 — Test your config on a tiny pipeline ✅

Do this before any real analysis. A quick smoke-test tells you whether your config is actually submitting jobs to the scheduler — much better to find out on a 2-minute test than 12 hours into a real run.

Every nf-core pipeline ships with a built-in test profile that supplies tiny example data, so you don’t need your own samplesheet or genome yet. Run a deliberately small pipeline with your config added:

nextflow run nf-core/demo -r 1.2.0 -profile test,singularity -c mycluster.config --outdir test_results

(nf-core/demo is a minimal example pipeline; you could equally test with -profile test on nf-core/rnaseq.)

✅ What to look for

1. The executor line — this is the key check. Near the top of the output, Nextflow prints which executor it’s using:

executor >  sge (3)

• If it says sge (or slurm, pbs, …) 🎉 your config is working — tasks are being submitted to the scheduler.
• If it says executor > local, your config was not applied and everything is running on the login node (which admins hate!). Re-check the -c mycluster.config path and the executor line inside it.

2. Jobs actually in the queue. Open a second terminal and check the scheduler — you should see Nextflow’s jobs appear and clear:

3. A clean finish. The run should end with something like:

-[nf-core/demo] Pipeline completed successfully-

and leave a test_results/ folder behind.

🔍 If a task fails, Nextflow prints the failing command and a work directory path (e.g. work/a1/b2c3…). cd into it and read .command.log, .command.err and .command.sh — these show exactly what the scheduler ran and why it failed. On an HPC this is your best debugging friend.

If the test fails

• Jobs rejected / wrong resources → your queue, clusterOptions or penv don’t match the cluster. Back to Step 1 with your admin.
• singularity: command not found on compute nodes → the container engine isn’t loaded inside jobs; ask how to make it available (often a module load in the config’s beforeScript).
• Containers fail to download on compute nodes → no internet on nodes; pre-download on a login node (Step 5).
• Add -resume so fixes don’t re-run everything from scratch.

Once this tiny test passes, you can trust the config for real runs.

Next — running it for real

With a working config, you run pipelines like any other user of your cluster: swap -profile docker for -profile singularity -c mycluster.config, and keep the Nextflow driver alive while the pipeline runs. How to do that (a driver job script for Slurm or SGE, tmux, or -bg) is covered in Part 7 · Running a pipeline on an HPC, Step 5.

🤝 Share your config. Once it works, consider adding it to nf-core/configs, so everyone at your institution can simply use -profile <your_cluster>. That’s a pull request, just like in Part 6.

Recap

• You don’t rewrite the pipeline for an HPC — you describe the cluster in a config.
• Ask your admin the scheduler, container engine, queues and limits (Step 1). And if they can help you build and test the config.
• Reuse an nf-core/configs profile if one exists, or adapt a similar one.
• Test it first with -profile test,singularity on a tiny pipeline and confirm the executor > slurm/sge line.
• Then run with -profile singularity -c mycluster.config, keeping the Nextflow driver alive as in Part 7 (a driver job, tmux, or -bg).