Configuration#

The configuration of the preprocessing is done with a .yaml file which steers the whole preprocessing. Available example config files for UPP can be found in upp/configs.

Each aspect of the configuration is described in detail below.

The config is built from these top-level keys:

Sample, region and class blocks (ttbar, lowpt, ...) are defined at the top level as well and pulled into components with yaml anchors, as shown below.

Input H5 Samples#

Here we define the input h5 samples which are to be preprocessed. Each sample is defined using one or more DSIDs, which generally come from the training-dataset-dumper. If a list of DSIDs is provided, the objects from each DSID will be merged according to the equal_global_objects flag (see below). The samples are used to define components later on in configs and so one should define them with anchors.

Below is an example and a table explaining each setting.

ttbar: &ttbar
  name: ttbar
  pattern: name1.*.410470.*/*.h5

ttbar: &ttbar
  name: ttbar
  equal_global_objects: False
  pattern:
    - name1.*.410470.*/*.h5
    - name2.*.410470.*/*.h5

The virtual dataset files created from wildcard patterns are by default stored alongside the input ntuples. If you have no write access to the input ntuples directory and would like to collect all VDS files in an accessible directory instead, set vds_dir in the global config (see Global Config). Each pattern gets its own VDS file named after its DSID directory (e.g. vds_dir/user.wlai.601589.e8547_..._output_vds.h5).

Global Cuts#

The selections that should be applied to all the data should be listed under common:. For example these could be outlier removal cuts, or a global kinematic selection. To do this one first provides the variable name (str), then the comparison operator (str) and a number to compare to (int, float or list). Possible operators are:

• "==", "!=", "<=", ">=", ">", "<" which work the same as in python.
• "in" and "notin" to check if the value is in the list.
• "%{i}==", "%{i}!=", "%{i}<=", "%{i}>=" operators to compare the modulo w.r.t. i of an integer, e.g. "%10<=". i can be any integer from 2 to 100.

Along with the common selection cuts, you should also specify the cuts that separate train, val and test splits using modulo of eventNumber. For example:

global_cuts:
  common:
    - [JetFitterSecondaryVertex_mass, "<", 25000]
    - [JetFitter_deltaR, "<", 0.6]
  train:
    - [eventNumber, "%10<=", 7]
  val:
    - [eventNumber, "%10==", 8]
  test:
    - [eventNumber, "%10==", 9]

Resampling Regions#

Next we define any kinematic regions which need to be resampled separately, again using anchors as these will also be used in the definition of our components. For each region you need to provide a name and a list of cuts (see above). Here is an example:

lowpt: &lowpt
  name: lowpt
  cuts:
    - [pt_btagJes, ">", 20_000]
    - [pt_btagJes, "<", 250_000]
highpt: &highpt
  name: highpt
  cuts:
    - [pt_btagJes, ">", 250_000]
    - [pt_btagJes, "<", 6_000_000]

Again, aliasing these just helps to reduce duplication of information when defining the components as can be seen below.

Components#

The components section is where all the configuration comes together. A component is a combination of a region, a sample and a flavour. They allow for full flexibility when defining different preprocessing pipelines (e.g. single-b versus Xbb).

An example components block is provided below.

components:
- region:
    <<: *lowpt
    sample:
    <<: *ttbar
    classes: [bjets, cjets, ujets]
    num_global_objects: 10_000_000

- region:
    <<: *highpt
    sample:
    <<: *zprime
    classes: [bjets, cjets, ujets]
    num_global_objects: 5_000_000

Notice that we use <<* insertion tool to insert already defined regions and samples.

Automatic object counts#

Instead of writing the counts yourself, set num_global_objects: auto in every component and let UPP solve them from a measurement of how many objects are actually available. The measurement is done by estimate_object_counts, which has to be run once before the preprocessing.

components:
  - region:
      <<: *lowpt
    sample:
      <<: *ttbar
    classes: [bjets, cjets, ujets]
    num_global_objects: auto

Each split gets its own count, so num_global_objects_val and num_global_objects_test are not needed. How many objects a region gets relative to the others is set with sample_weight on the sample (see Input H5 Samples).

The optional auto_counts block tunes what is solved for:

auto_counts:
  class_ratios:
    bjets: 1
    cjets: 1
    ujets: 2

Custom classes#

By default the class definitions come from the flavour labels bundled with atlas-ftag-tools, selected with class_category (standard or extended). These are jet flavours, but the framework itself is object-agnostic: to classify any other object type, point class_config at your own classes yaml. It is a list of class definitions, each with a name, plotting label, selection cuts, a colour, a category, and an optional _px probability name:

- name: heavy
  label: Heavy objects
  cuts: ["HadronConeExclTruthLabelID == 5"]
  colour: tab:red
  category: custom
- name: light
  label: Light objects
  cuts: ["HadronConeExclTruthLabelID == 0"]
  colour: tab:blue
  category: custom

Reference it from the global config; a relative path is resolved against base_dir, and class_config takes precedence over class_category:

global:
  global_name: objects
  class_config: custom_flavours.yaml

The classes listed for each component then refer to the name entries in this file (e.g. classes: [heavy, light]).

Variables#

The next thing you need is to provide the variables that are taken from the TDD files and written in the resampled dataset. Selecting only a subset of variables keeps the output files lightweight, and ensures the dataloading does not become a bottleneck during training.

One can simply define them under variables: like:

variables:
  jets:
    inputs:
      - pt_btagJes
      - absEta_btagJes
    labels:
      - HadronConeExclTruthLabelID
      - pt
      - eta

  tracks:
    inputs:
      - dphi
      - deta
      - qOverP
      - IP3D_signed_d0_significance
      - IP3D_signed_z0_significance
    labels:
      - ftagTruthOriginLabel
      - ftagTruthVertexIndex

Alternatively include the variables from your custom variable config by providing the full path to the file after an include statement. The file you provide should have the same structure as shown above but without variable: level. For example:

variables: !include xbb-variables.yaml

One can also import variable configs already provided in this package under upp/configs/ by using the yaml file name, e.g.:

variables: !include variables.yaml

Track selections#

You can apply on the fly selections to tracks in the preprocessing stage (specifically the merging step).

To do this, include a selection key in the variable config block under the tracks, for example:

  tracks:
    inputs:
      - d0
    labels:
      - ftagTruthOriginLabel
    selection:
      - [d0, ">", 0.1]

Variable transformations#

The optional transform: block renames variables and remaps values while the input files are read. This is useful when the naming in your ntuples does not match what the training expects. It is handed to the Transform class of atlas-ftag-tools, which supports three maps:

transform:
  variable_map:
    tracks:
      truthOriginLabel: ftagTruthOriginLabel
      truthVertexIndex: ftagTruthVertexIndex
  ints_map:
    tracks:
      ftagTruthOriginLabel:
        0: 1
        -2: 0
  floats_map:
    jets:
      pt: log

Resampling#

There are currently two resampling methods implemented in the package pdf and countup and they share most of setting. Below is the example of setting up the pdf resampling method and a table describing all the parameters.

In order to run UPP without any kinematic resampling, just set method: none. Note you will still need to run the resampling stage of the preprocessing pipeline.

resampling:
  target: cjets
  method: pdf
  upscale_pdf: 2
  sampling_fraction: auto
  variables:
    pt_btagJes:
      bins: [[20_000, 250_000, 50], [250_000, 1_000_000, 50], [1_000_000, 6_000_000, 50]]
    absEta_btagJes:
      bins: [[0, 2.5, 20]]

Plotting#

Plot labels and styles can be configured under the optional plotting: key. Any omitted setting uses the default defined by the PlottingConfig dataclass (documented below). Variable-label keys are matched case-insensitively against the plotted variable name, so a single pt entry applies to variables such as pt_btagJes.

plotting:
  num_global_objects_plotting: 10_000_000
  variable_labels:
    pt: "Jet $p_\\mathrm{T}$ [GeV]"
    eta: "Jet $|\\eta|$"
    mass: "Jet Mass [GeV]"
  sample_labels:
    ttbar: "$t\\bar{t}"
    zprime: "$Z'$"
  atlas_first_tag: Simulation Internal
  atlas_second_tag: "$\\sqrt{s} = 13/13.6\\,\\mathrm{TeV}$"
  output_formats: [pdf, png]
  bins: 50
  y_scale: 1.5
  figsize: [6, 4]
  logy: true
  linestyles: ["-", "--", "-.", ":"]
  legend_location: upper right
  linestyle_legend_location: upper center
  linestyle_legend_anchor: [0.55, 1]
  output_directory: plots

The ylabel setting supports a {global_name} placeholder. Histogram normalisation and overflow handling can be controlled with norm and underoverflow.

Global Config#

global:
    global_name: objects
    batch_size: 1_000_000
    num_global_objects_estimate: 5_000_000
    base_dir: /my/stuff/
    ntuple_dir: h5-inputs # resolved path: /my/stuff/h5-inputs/