dcnum.write.queue_writer_base

Attributes

mp_spawn

Classes

QueueWriterBase

Write events from a queue to an .rtdc file

Module Contents

dcnum.write.queue_writer_base.mp_spawn = None

class dcnum.write.queue_writer_base.QueueWriterBase(event_queue: multiprocessing.Queue, write_queue_size: multiprocessing.Value, feat_nevents: multiprocessing.Array, path_out: pathlib.Path, hdf5_dataset_kwargs: dict = None, write_threshold: int = 500, *args, **kwargs)

Write events from a queue to an .rtdc file

Events coming from a queue cannot be guaranteed to be in order. The QueueWriterThread uses a EventStash to sort events into the correct order before sending them to the ChunkWriter for storage.

Parameters:

• event_queue – A queue object to which other processes or threads write events as tuples (frame_index, events_dict).

• write_queue_size – A mp.Value that is populated with the number of event chunks waiting to be written to the output file by the ChunkWriter.

• feat_nevents – This 1D array contains the number of events for each frame in the input data. This serves two purposes: (1) it allows us to determine how many events we are writing when we are writing data from write_threshold frames, and (2) it allows us to keep track how many frames have actually been processed (and thus we can expect entries in event_queue for). If an entry in this array is -1, this means that there is no event in event_queue. See write_threshold below.

• path_out – Output path for writer

• hdf5_dataset_kwargs – Dictionary of keyword arguments (e.g. “compression”) for HDF5 dataset creation.

• write_threshold – This integer defines how many frames should be collected at once and put into writer_dq. For instance, with a value of 500, at least 500 items are taken from the event_queue (they should match the expected frame index, frame indices that do not match are kept in a EventStash). Then, for each frame, we may have multiple or None events, so the output size could be 513 which is computed via np.sum(feat_nevents[idx:idx+write_threshold]).

log_level = 0

logging level (inherited from ‘dcnum’ logger)

event_queue

Event queue from which to collect event data

write_queue_size

Number of event chunks waiting to be written to the output file

feat_nevents

shared array containing the number of events for each frame in data.

path_out

HDF5 output path

hdf5_dataset_kwargs = None

HDF5 dataset creation keyword arguments

write_threshold = 500

Number of frames to send to writer_dq at a time.

written_events

Number of events sent to writer_dq

written_frames

Number of frames from data written to writer_dq

get_logger()

run(buffer_dq=None, writer_dq=None, logger=None)

Start the writing process

This method is intended to be run in a thread or process.

buffer_dq and writer_dq are used for testing.