dcnum.common

Attributes

h5py	Lazily loaded h5py module

Classes

LazyLoader

Lazily load a module

Functions

cpu_count(→ int)	Get the number of processes available
join_worker(worker[, timeout, retries, logger, name])	Patiently join a worker (Thread or Process)
start_workers_threaded(worker_list, logger, name)
setup_h5py(h5py)	Hook for LazyLoader that imports hdf5plugin

Module Contents

class dcnum.common.LazyLoader(modname: str, sibling: str = None, action: Callable = None)

Lazily load a module

Parameters:

• modname (str) – The name of the module (e.g. "scipy.ndimage")

• sibling (str) –

  The __name__ of a sibling of the module. This is useful for performing relative imports. Consider this module structure:

  • module - submod_1 - submod_2

  If submod_1 would like to lazily import submod_2:

  submod_2 = LazyLoader("submod_2", sibling==__name__)
  

• action (Callable) – Method that should be called after the actual import. Must accept the module as an argument. This is useful if any setup steps need to be made after import (e.g. for ensuring reproducibility).

_modname

_mod = None

_action = None

__getattr__(attr)

If the module is accessed, load it and return what was asked for

dcnum.common.cpu_count() → int

Get the number of processes available

multiprocessing.cpu_count() returns the number of logical CPUs available. We are interested in the physical CPUs, because there is no performance advantage to using all logical CPUs and because using more CPUs would imply more workers and thus more RAM usage.

On cluster systems, with jobs possibly having CPU affinity, we want to stick to that limitation as well.

This method handles both cases by:

1. Get the number of physical CPUs using psutil.cpu_count(logical=False)

2. Get the number of CPUs according to process affinity

3. Get the number of CPUs using multiprocessing.cpu_count()

4. Return the minimum of all of the above

dcnum.common.join_worker(worker, timeout=30, retries=10, logger=None, name=None)

Patiently join a worker (Thread or Process)

dcnum.common.start_workers_threaded(worker_list, logger, name)

dcnum.common.setup_h5py(h5py)

Hook for LazyLoader that imports hdf5plugin

dcnum.common.h5py

Lazily loaded h5py module