Skip to content

Setup

This guide will walk you through the process of setting up the Umami-Preprocessing Python package on your system.

Environment Setup#

Install UPP in a virtual environment to avoid conflicts with other software libraries. UPP currently supports Python 3.11, 3.12, 3.13 and 3.14.

uv is a fast Python package and project manager and is the recommended way to develop UPP. Install it following the official instructions, then let uv create and manage the environment for you:

uv venv
source .venv/bin/activate

When working from a clone of the repository you can skip the manual environment setup entirely and use uv sync (see below).

Set up a fresh conda or mamba environment:

conda create -n upp python=3.13
conda activate upp

venv is a lightweight solution for creating virtual python environments, however it is not as fully featured as a fully fledged package manager such as conda. Create a fresh virtual environment and activate it using

python3 -m venv upp
source upp/bin/activate

If you don't want to use conda environments on lxplus, you can setup python via

setupATLAS
lsetup "python 3.11.9-x86_64-el9"

You can also set up conda on lxplus

PyPi installation#

If you don't plan on editing the source code, the simplest way to install UPP directly from the Python Package Index (PyPI):

python -m pip install umami-preprocessing

On lxplus you may have to use python3 instead of just python

Download source code#

The following instructions are only relevant for those that wish to modify UPP source code. Start by cloning the Umami-Preprocessing repository. If you want to contribute to the development of UPP, you should fork the repository and make sure you do all your edits in a development branch.

git clone https://github.com/umami-hep/umami-preprocessing.git

Install package from code#

Navigate to the newly downloaded repository and install the package in editable mode.

uv sync creates a virtual environment in .venv and installs UPP together with the development dependency group in editable mode:

cd umami-preprocessing
uv sync

Prefix subsequent commands with uv run (e.g. uv run preprocess ...) or activate the environment with source .venv/bin/activate.

Install UPP in editable mode, then add the development dependency group (requires pip >= 25.1, which understands PEP 735 dependency groups):

cd umami-preprocessing
python -m pip install -e .
python -m pip install --group dev

If you don't plan on editing the code you can do a regular install instead

python -m pip install .

Note for running on lxplus

Again, you may have to use python3 here.

You may also see a warning like this:

WARNING: The script preprocess is installed in '/afs/cern.ch/user/X/Y/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

if you do, you can add the directory to your path using

export PATH=$PATH:/afs/cern.ch/user/X/Y/.local/bin

Alternatively, you can just run the scripts by pointing to the main.py

python3 upp/main.py

Container image#

If you don't want to set up a Python environment at all, you can use the UPP container image. The CI builds gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest on every merge to main and a tagged image upp:<tag> (e.g. upp:v0.3.1) for every release. The image comes with UPP and its command line scripts (preprocess, check_input_samples, list_components) pre-installed.

On clusters (lxplus, HPC sites), apptainer can run the image directly from the registry:

apptainer exec docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
    preprocess --config <path/to/config.yaml>

The first docker:// invocation converts the image to apptainer's SIF format, which takes a while. If you run UPP repeatedly, pull the image once and use the local file instead:

apptainer pull upp_latest.sif docker://gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest
apptainer exec upp_latest.sif preprocess --config <path/to/config.yaml>

By default apptainer shares your home directory and working directory with the container. For a cleaner environment use --contain and bind only the paths you need (your input ntuples and output directory) with -B, keeping the working directory with --pwd:

apptainer exec --contain --pwd "$PWD" -B /home -B /tmp -B <path/to/data> \
    upp_latest.sif preprocess --config <path/to/config.yaml>

With docker, mount your working directory and data paths into the container:

docker run --rm -it -v $PWD:$PWD -w $PWD \
    gitlab-registry.cern.ch/aft/training-images/upp-images/upp:latest \
    preprocess --config <path/to/config.yaml>

Pin a release tag for production

upp:latest follows the main branch and changes over time. For reproducible production preprocessing, use a tagged release image like upp:v0.3.1 instead.

For running UPP as batch jobs on Slurm clusters with the container image, see Running on HPC.

Run the tests (Optional)#

To ensure that the package is working correctly, you can run the tests using the pytest framework.

Use pytest to run the tests to make sure the package works

pytest tests

If you want to measure test coverage you can use the commands: 

coverage run --source upp -m pytest tests --show-capture=stdout
coverage report