dcnum.logic.slot_register

Attributes

ndi
mp_spawn

Classes

count_time	Decorator for counting execution time
SlotRegister	A register for `ChunkSlot`s for shared memory access
StateWarden	Context manager for changing the state of a ChunkSlot

Module Contents

dcnum.logic.slot_register.ndi

dcnum.logic.slot_register.mp_spawn = None

class dcnum.logic.slot_register.count_time

Decorator for counting execution time

__call__(func)

class dcnum.logic.slot_register.SlotRegister(job: dcnum.logic.job.DCNumPipelineJob, data: dcnum.read.HDF5Data, event_queue: multiprocessing.Queue | None = None, num_slots: int = 3)

A register for `ChunkSlot`s for shared memory access

The SlotRegister manages all ChunkSlot instances and implements methods to interact with individual `ChunkSlot`s.

job

data

event_queue

chunk_size

num_chunks

_slots = []

timers

counters

_state

num_frames

Total number of frames to process

feat_nevents

Number of events per frame Shared RawArray of length len(data) into which the number of events per frame is written.

__getitem__(idx)

__iter__()

Iterate over slots, sorted by current chunk number

__len__()

property chunks_loaded

A process-safe counter for the number of chunks loaded

This number increments as SlotRegister.task_load_all is called.

property masks_dropped

A process-safe counter for the number of masks dropped

Segmentation may drop invalid masks/events.

property write_queue_size

A process-safe counter for the number of chunks in the writer queue

A large number indicates a slow writer which can be a result of a slow hard disk or a slow CPU (since is used compression). Used for preventing OOM events by stalling data processing when the writer is slow

property slots

A list of all ChunkSlots

property state

State of the SlotRegister, used for communication with workers

• “w”: initialized (workers work)

• “p”: paused (all workers pause)

• “q”: quit (all workers stop)

close()

find_slot(state: str, chunk: int | None = None) → dcnum.logic.chunk_slot.ChunkSlot | None

Return the first ChunkSlot that has the given state

We sort the slots according to the slot chunks so that we always process the slot with the smallest slot chunk number first. Initially, the slot_chunks array is filled with zeros, but we populate it here.

Return None if no matching slot exists

get_counter_lock(name)

get_time(method_name)

Return accumulative time for the given method

reserve_slot_for_task(current_state: str, next_state: str, chunk_slot: dcnum.logic.chunk_slot.ChunkSlot | None = None, batch_size: int | None = None) → StateWarden | None

Return slot with the specified state and lowest chunk index

Parameters:

• current_state – State requried for the task to start

• next_state – State that will be set after the task is done

• chunk_slot – Optional ChunkSlot to operate on; if set to None, search for a matching one, and if none can be found, return None

• batch_size – Number of frames to reserve for performing the task. Defaults to the entire chunk.

Returns:

Context manager that enforces setting the next state or None if no ChunkSlot could be reserved. Usage:

if state_warden is not None:

with state_warden as (chunk_slot, batch_range):

perform_task(chunk_slot,

start_index=batch_range[0], stop_index=batch_range[1] )

The batch_range indices are defined by the batch_size parameter.

This context manager will automatically set the slot state to next_state when the context is exits without exceptions.

Return type:

state_warden

task_load_all(logger: logging.Logger | None = None) → bool

Load chunk data into memory for as many slots as possible

Returns:

did_something – Whether data were loaded into memory

Return type:

bool

task_label_masks(logger: logging.Logger | None = None) → bool

Perform labeling of mask images for a ChunkSlot

This method is process-safe. Multiple processes may call it concurrently, working on the same ChunkSlot.

Returns:

did_something – Whether masks where converted to labels

Return type:

bool

task_process_labels(logger: logging.Logger | None = None) → bool

Perform label processing (e.g. binary closing) for a ChunkSlot

This method is process-safe. Multiple processes may call it concurrently, working on the same ChunkSlot.

Returns:

did_something – Whether labels were processed

Return type:

bool

task_extract_features(logger: logging.Logger | None = None) → bool

Perform feature extraction for a ChunkSlot

This method is process-safe. Multiple processes may call it concurrently, working on the same ChunkSlot.

Returns:

did_something – Whether events were extracted

Return type:

bool

class dcnum.logic.slot_register.StateWarden(chunk_slot: dcnum.logic.chunk_slot.ChunkSlot | dcnum.logic.chunk_slot.ChunkSlotData, current_state: str, next_state: str, batch_size: int | None = None)

Context manager for changing the state of a ChunkSlot

batch_range

batch_size

chunk_slot

current_state

next_state

__enter__()

__exit__(exc_type, exc_val, exc_tb)

__repr__()