dcnum.logic.slot_register ========================= .. py:module:: dcnum.logic.slot_register Attributes ---------- .. autoapisummary:: dcnum.logic.slot_register.ndi dcnum.logic.slot_register.mp_spawn Classes ------- .. autoapisummary:: dcnum.logic.slot_register.count_time dcnum.logic.slot_register.SlotRegister dcnum.logic.slot_register.StateWarden Module Contents --------------- .. py:data:: ndi .. py:data:: mp_spawn :value: None .. py:class:: count_time Decorator for counting execution time .. py:method:: __call__(func) .. py:class:: SlotRegister(job: dcnum.logic.job.DCNumPipelineJob, data: dcnum.read.HDF5Data, event_queue: multiprocessing.Queue | None = None, num_slots: int = 3) A register for `ChunkSlot`s for shared memory access The `SlotRegister` manages all `ChunkSlot` instances and implements methods to interact with individual `ChunkSlot`s. .. py:attribute:: job .. py:attribute:: data .. py:attribute:: event_queue .. py:attribute:: chunk_size .. py:attribute:: num_chunks .. py:attribute:: _slots :value: [] .. py:attribute:: timers .. py:attribute:: counters .. py:attribute:: _state .. py:attribute:: num_frames Total number of frames to process .. py:attribute:: feat_nevents Number of events per frame Shared RawArray of length `len(data)` into which the number of events per frame is written. .. py:method:: __getitem__(idx) .. py:method:: __iter__() Iterate over slots, sorted by current chunk number .. py:method:: __len__() .. py:property:: chunks_loaded A process-safe counter for the number of chunks loaded This number increments as `SlotRegister.task_load_all` is called. .. py:property:: masks_dropped A process-safe counter for the number of masks dropped Segmentation may drop invalid masks/events. .. py:property:: write_queue_size A process-safe counter for the number of chunks in the writer queue A large number indicates a slow writer which can be a result of a slow hard disk or a slow CPU (since is used compression). Used for preventing OOM events by stalling data processing when the writer is slow .. py:property:: slots A list of all `ChunkSlots` .. py:property:: state State of the `SlotRegister`, used for communication with workers - "w": initialized (workers work) - "p": paused (all workers pause) - "q": quit (all workers stop) .. py:method:: close() .. py:method:: find_slot(state: str, chunk: int | None = None) -> dcnum.logic.chunk_slot.ChunkSlot | None Return the first `ChunkSlot` that has the given state We sort the slots according to the slot chunks so that we always process the slot with the smallest slot chunk number first. Initially, the slot_chunks array is filled with zeros, but we populate it here. Return None if no matching slot exists .. py:method:: get_counter_lock(name) .. py:method:: get_time(method_name) Return accumulative time for the given method .. py:method:: reserve_slot_for_task(current_state: str, next_state: str, chunk_slot: dcnum.logic.chunk_slot.ChunkSlot | None = None, batch_size: int | None = None) -> StateWarden | None Return slot with the specified state and lowest chunk index :param current_state: State requried for the task to start :param next_state: State that will be set after the task is done :param chunk_slot: Optional `ChunkSlot` to operate on; if set to None, search for a matching one, and if none can be found, return None :param batch_size: Number of frames to reserve for performing the task. Defaults to the entire chunk. :returns: Context manager that enforces setting the next state or None if no `ChunkSlot` could be reserved. Usage: if state_warden is not None: with state_warden as (chunk_slot, batch_range): perform_task(chunk_slot, start_index=batch_range[0], stop_index=batch_range[1] ) The `batch_range` indices are defined by the `batch_size` parameter. This context manager will automatically set the slot state to `next_state` when the context is exits without exceptions. :rtype: state_warden .. py:method:: task_load_all(logger: logging.Logger | None = None) -> bool Load chunk data into memory for as many slots as possible :returns: **did_something** -- Whether data were loaded into memory :rtype: bool .. py:method:: task_label_masks(logger: logging.Logger | None = None) -> bool Perform labeling of mask images for a `ChunkSlot` This method is process-safe. Multiple processes may call it concurrently, working on the same `ChunkSlot`. :returns: **did_something** -- Whether masks where converted to labels :rtype: bool .. py:method:: task_process_labels(logger: logging.Logger | None = None) -> bool Perform label processing (e.g. binary closing) for a `ChunkSlot` This method is process-safe. Multiple processes may call it concurrently, working on the same `ChunkSlot`. :returns: **did_something** -- Whether labels were processed :rtype: bool .. py:method:: task_extract_features(logger: logging.Logger | None = None) -> bool Perform feature extraction for a `ChunkSlot` This method is process-safe. Multiple processes may call it concurrently, working on the same `ChunkSlot`. :returns: **did_something** -- Whether events were extracted :rtype: bool .. py:class:: StateWarden(chunk_slot: dcnum.logic.chunk_slot.ChunkSlot | dcnum.logic.chunk_slot.ChunkSlotData, current_state: str, next_state: str, batch_size: int | None = None) Context manager for changing the state of a `ChunkSlot` .. py:attribute:: batch_range .. py:attribute:: batch_size .. py:attribute:: chunk_slot .. py:attribute:: current_state .. py:attribute:: next_state .. py:method:: __enter__() .. py:method:: __exit__(exc_type, exc_val, exc_tb) .. py:method:: __repr__()