API Docs

class smart_pipe.SmartPipeWriter(path_prefix, compress=False)[source]

Class which can create smart pipe file

__init__(path_prefix, compress=False)[source]

Constructs smart pipe writer

Parameters:
  • path_prefix – path prefix to be used. Extension for data and index files will be appended to this prefix
  • compress – boolean flag (False by default) indicating compression of data chunks
append(key, value)[source]

Appends given key-value pair to current chunk

Parameters:
  • key – byte string with key
  • value – byte string with value
checkpoint(key)[source]

Flushes current chunk on disk and starts new with the given key

Parameters:key – byte string with binary key
close()[source]

Closes smart pipe writer and flushes data to disk

class smart_pipe.SmartPipeReader(path_prefix)[source]

Reader for smart pipe data

__init__(path_prefix)[source]

Constructs smart pipe reader

Parameters:path_prefix – path prefix for smart pipe data and index files
close()[source]

Closes smart pipe reader

get_next_block_key()[source]

Return index key for the next block

Returns:key of the next block or None if we reached end of pipe
pull_block(index_key=None)[source]

Get list of key,value pairs from next block

Parameters:index_key – optional key of block to seek
Returns:yield block records