ftag.hdf5.h5reader#

Classes#

H5SingleReader

H5 single reader for a single file.

H5Reader

Reads data from multiple HDF5 files.

Module Contents#

class ftag.hdf5.h5reader.H5SingleReader#

H5 single reader for a single file.

fname#

Path to the file

Type:

Path | str

batch_size#

Batch size to read, by default 100_000

Type:

int, optional

global_objects_name#

Name of the global objects dataset, by default “jets”

Type:

str, optional

precision#

Precision that is to be used, by default None

Type:

str | None, optional

shuffle#

If random global objects are loaded from the file, by default True

Type:

bool, optional

do_remove_inf#

Remove infs from the global objects, by default False

Type:

bool, optional

transform#

Transformation that should be applied, by default None

Type:

Transform | None, optional

groups#

List of the groups that hold metadata, by default None

Type:

list[str] | None, optional

dsets#

List of the datsets, that hold the per global-object/track data, by default None

Type:

list[str] | None, optional

vds_dir#

Directory where virtual datasets will be stored if wildcard is used, by default None. If None, the virtual files will be created in the same directory as the input files.

Type:

Path | str | None, optional

fname: pathlib.Path | str#
batch_size: int = 100000#
global_objects_name: str = 'jets'#
precision: str | None = None#
shuffle: bool = True#
do_remove_inf: bool = False#
transform: ftag.transform.Transform | None = None#
groups: list[str] | None = None#
dsets: list[str] | None = None#
vds_dir: pathlib.Path | str | None = None#
__post_init__() → None#
property num_global_objects: int#
get_attr(name, group=None)#
empty(ds: h5py.Dataset | h5py.Group, variables: list[str]) → numpy.ndarray#
read_chunk(ds: h5py.Dataset, array: numpy.ndarray, low: int) → numpy.ndarray#
remove_inf(data: dict) → dict#
_process_batch(data: dict, cuts: ftag.cuts.Cuts | None = None) → dict#

Apply cuts and transformations to the batch.

Parameters:

  • data (dict) – Dictionary of arrays for each group.

  • cuts (Cuts | None, optional) – Selection cuts to apply, by default None

Returns:

Processed data dictionary with arrays for each group. After applying cuts, (optional) removal of infs, and (optional) transformation.

Return type:

dict

stream(variables: dict | None = None, num_global_objects: int | None = None, cuts: ftag.cuts.Cuts | None = None, start: int = 0, skip_batches: int = 0) → collections.abc.Generator#
get_batch_reader(variables: dict | None = None, cuts: ftag.cuts.Cuts | None = None) → collections.abc.Callable#

Get a function to read batches of selected global objects.

Parameters:

  • variables (dict | None, optional) – Dictionary of variables to for each group, by default use all global object variables.

  • cuts (Cuts | None, optional) – Selection cuts to apply, by default None

Returns:

Function that takes an index and returns a batch of selected global objects.

Return type:

Callable

class ftag.hdf5.h5reader.H5Reader#

Reads data from multiple HDF5 files.

fname#

Path to the HDF5 file or list of paths

Type:

Path | str | list[Path | str]

batch_size#

Number of global objects to read at a time, by default 100_000

Type:

int, optional

global_objects_name#

Name of the global objects dataset, by default “jets”

Type:

str, optional

precision#

Cast floats to given precision, by default None

Type:

str | None, optional

shuffle#

Read batches in a shuffled order, by default True

Type:

bool, optional

weights#

Weights for different input datasets, by default None

Type:

list[float] | None, optional

do_remove_inf#

Remove global objects with inf values, by default False

Type:

bool, optional

transform#

Transform to apply to data, by default None

Type:

Transform | None, optional

equal_global_objects#

Take the same number of global objects (weighted) from each sample, by default True. This is useful when you specify a list of DSIDs for the sample and they are qualitatively different, and you want to ensure that you always return batches with global objects from all DSIDs. This is used for example in the QCD resampling for Xbb. If False, use all global objects in each sample, allowing for the full available statistics to be used. Useful for example if you have multiple ttbar samples and you want to use all available global objects from each sample.

Type:

bool, optional

vds_dir#

Directory where virtual datasets will be stored if wildcard is used, by default None. If None, the virtual files will be created in the same directory as the input files.

Type:

Path | str | None, optional

fname: pathlib.Path | str | list[pathlib.Path | str]#
batch_size: int = 100000#
global_objects_name: str = 'jets'#
precision: str | None = None#
shuffle: bool = True#
weights: list[float] | None = None#
do_remove_inf: bool = False#
transform: ftag.transform.Transform | None = None#
equal_global_objects: bool = False#
vds_dir: pathlib.Path | str | None = None#
__post_init__() → None#
property num_global_objects: int#
property files: list[pathlib.Path]#
dtypes(variables: dict[str, list[str]] | None = None) → dict[str, numpy.dtype]#
shapes(num_global_objects: int, groups: list[str] | None = None) → dict[str, tuple[int, Ellipsis]]#
stream(variables: dict | None = None, num_global_objects: int | None = None, cuts: ftag.cuts.Cuts | None = None, start: int = 0, skip_batches: int = 0) → collections.abc.Generator#

Generate batches of selected global objects.

Parameters:

  • variables (dict | None, optional) – Dictionary of variables to for each group, by default use all global object variables.

  • num_global_objects (int | None, optional) – Total number of selected global objects to generate, by default all.

  • cuts (Cuts | None, optional) – Selection cuts to apply, by default None

  • start (int, optional) – Starting index of the first global object to read, by default 0

  • skip_batches (int, optional) – Number of batches to skip, by default 0

Yields:

Generator – Generator of batches of selected global objects.

get_batch_reader(variables: dict | None = None, cuts: ftag.cuts.Cuts | None = None, shuffle: bool = True) → collections.abc.Callable#

Get a function to read batches of selected global objects.

Parameters:

  • variables (dict | None, optional) – Dictionary of variables to for each group, by default use all global object variables.

  • cuts (Cuts | None, optional) – Selection cuts to apply, by default None

  • shuffle (bool, optional) – Read batches in a shuffled order, by default True

Returns:

Function that takes an index and returns a batch of selected global objects.

Return type:

Callable

load(variables: dict | None = None, num_global_objects: int | None = None, cuts: ftag.cuts.Cuts | None = None) → dict#

Load multiple batches of selected global objects into memory.

Parameters:

  • variables (dict | None, optional) – Dictionary of variables to for each group, by default use all global object variables.

  • num_global_objects (int | None, optional) – Total number of selected global objects to load, by default all.

  • cuts (Cuts | None, optional) – Selection cuts to apply, by default None

Returns:

Dictionary of arrays for each group.

Return type:

dict

estimate_available_global_objects(cuts: ftag.cuts.Cuts, num: int = 1000000) → int#

Estimate the number of global objects available after selection cuts.

Parameters:

  • cuts (Cuts) – Selection cuts to apply.

  • num (int, optional) – Number of global objects to use for the estimation, by default 1_000_000.

Returns:

Estimated number of global objects available after selection cuts, rounded down.

Return type:

int