Running on HPC clusters#

The preprocessing stages can be parallelized over components, regions and splits using the --component, --region and --split flags described in Run. On a Slurm cluster, each of these units of work can run as its own batch job inside the container image. UPP ships a small set of submission scripts in scripts/slurm/ that automate this.

Prerequisites#

• A cluster with Slurm and apptainer.
• The UPP container image (see Container image). By default the scripts use docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest. To avoid the repeated docker:// to SIF conversion in every job, pull the image once and point UPP_IMAGE at the local file:

apptainer pull upp_latest.sif docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest
export UPP_IMAGE=/path/to/upp_latest.sif

Interactive use#

For quick tests, run UPP inside the container on an interactive allocation:

salloc --ntasks 1 --cpus-per-task 4 --time 2:00:00
srun apptainer exec --contain --pwd "$PWD" -B /home -B /tmp \
    "$UPP_IMAGE" preprocess --config <path/to/config.yaml> --prep

Batch submission scripts#

The scripts/slurm/ directory contains three scripts:

• submit.sh runs on the login node. It reads the components from your preprocessing config and submits one Slurm job per unit of work via sbatch.
• batch.sh is the sbatch payload. It carries the #SBATCH resource header and starts the container on the compute node.
• run_stage.sh runs inside the container and maps the submitted mode onto the preprocess command line flags.

To use them, create a run directory, copy the scripts, and adapt the #SBATCH header in batch.sh to your cluster (partition, account, time and memory limits):

mkdir my_preprocessing && cd my_preprocessing
cp -r <path/to/umami-preprocessing>/scripts/slurm .
$EDITOR slurm/batch.sh

Then submit the stages in order, waiting for all jobs of one stage to finish before submitting the next:

./slurm/submit.sh --config <path/to/config.yaml> --dry-run prepare  # preview only
./slurm/submit.sh --config <path/to/config.yaml> prepare
./slurm/submit.sh --config <path/to/config.yaml> resampling
./slurm/submit.sh --config <path/to/config.yaml> merge
./slurm/submit.sh --config <path/to/config.yaml> normalise
./slurm/submit.sh --config <path/to/config.yaml> plotting

Job logs are written to logs/ in the current directory. Running submit.sh without a mode enters an interactive prompt for the mode and filters.

The available modes and the jobs they submit:

Config-driven job lists#

submit.sh never hardcodes which components exist. It calls the list_components script (part of UPP) to enumerate the components defined in the components: block of your config:

list_components --config <path/to/config.yaml>

lowpt   ttbar   bjets   lowpt_ttbar_bjets
highpt  zprime  bjets   highpt_zprime_bjets
...

Only combinations actually defined in the config are submitted. The selection can be narrowed with filter flags, each taking a comma- or space-separated list:

./slurm/submit.sh --config <path/to/config.yaml> --regions lowpt --splits train prepare
./slurm/submit.sh --config <path/to/config.yaml> --samples ttbar --flavs "bjets,cjets" fine_resampling

Note that enumerating the components fully validates the config, so a broken config fails directly on the login node instead of inside the batch jobs.

Environment variables#

Make sure UPP_BINDS covers your input ntuples and output directory if they live outside /home (e.g. on a scratch filesystem), and export UPP_IMAGE/UPP_BINDS in your shell so they are also picked up by the batch jobs.