zensols.persist package

Submodules

zensols.persist.annotation module

Contains general purpose persistence library classes.

class zensols.persist.annotation.FileTextUtil

Bases: object

Basic file naming utility methods.

static byte_format(num, suffix='B')

Return a human readable string of the number of bytes num.

Parameters:

• num (int) – the number of bytes to format

• suffix (str) – the suffix to append to the resulting string

Attribution:

Fred Cirera <https://stackoverflow.com/questions/1094841/get-human-readable-version-of-file-size>

Return type:

str

classmethod normalize_path(path, replace_char='-', lower=True, regex=None, remove_non_printable=False)

Applies normalize_text() for each component of the path and return it. Directory separators and tilde (~) are not normalized.

See:

normalize_text()

Return type:

Path

classmethod normalize_text(name, replace_char='-', lower=True, regex=None, remove_non_printable=False)

Normalize the name in to a string that is more file system friendly. This removes special characters and replaces them with replace_char.

Parameters:

• name (str) – the name to be normalized

• replace_char (str) – the character used to replace special characters

• lower (bool) – whether to lowercase the text

• regex (Pattern) – the regular expression that matches on text to remove

• remove_non_printable (bool) – whether to keep only ASCII characters

Return type:

str

Returns:

the normalized name

static unique_tracked_name(prefix, include_user=True, include_time=True, extension=None)

Create a unique file name useful for tracking files.

Parameters:

• prefix (str) – the file name that identifier

• include_user (bool) – whether to add the user name in the file

Return type:

str

class zensols.persist.annotation.PersistableContainer

Bases: Deallocatable

Classes can extend this that want to persist PersistedWork instances, which otherwise are not persistable.

This class also manages the deallocation of all PersistedWork attributes of the class, which might be another reason to use it even if there isn’t a persistence use case.

If the class level attribute _PERSITABLE_TRANSIENT_ATTRIBUTES is set, all attributes given in this set will be set to None when pickled.

If the class level attribute _PERSITABLE_REMOVE_ATTRIBUTES is set, all attributes given in this set will be set object deleted when pickled.

If the class level attribute _PERSITABLE_PROPERTIES is set, all properties given will be accessed for force creation before pickling.

If the class level attribute _PERSITABLE_METHODS is set, all method given will be accessed for force creation before pickling.

_clear_persistable_state()

Clear all cached state from all PersistedWork in this instance.

deallocate()

Deallocate all resources for this instance.

class zensols.persist.annotation.PersistableContainerMetadata(container)

Bases: object

Provides metadata about PersistedWork definitions in the class.

__init__(container)

clear()

Clear all PersistedWork instances on this object.

property persisted

Return all PersistedWork instances on this object as a dict.

write(indent=0, include_content=False, recursive=False, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

exception zensols.persist.annotation.PersistableError

Bases: APIError

Thrown for any persistable API error

__annotations__ = {}

__module__ = 'zensols.persist.annotation'

class zensols.persist.annotation.PersistedWork(path, owner, cache_global=False, transient=False, initial_value=None, mkdir=False, deallocate_recursive=False, recover_empty=False)

Bases: Deallocatable

This class caches data in the instance of the contained class and/or global level. In addition, the data is also pickled to disk to avoid any expensive recomputation of the data.

In order, it first looks for the data in owner, then in globals (if cache_global is True), then it looks for the data on the file system. If it can’t find it after all of this it invokes function worker to create the data and then pickles it to the disk.

This class is a callable itself, which is invoked to get or create the work.

There are two ways to implement the data/work creation: pass a worker to the __init__ method or extend this class and override __do_work__.

__init__(path, owner, cache_global=False, transient=False, initial_value=None, mkdir=False, deallocate_recursive=False, recover_empty=False)

Create an instance of the class.

Parameters:

• path (Union[str, Path]) – if type of pathlib.Path then use disk storage to cache of the pickeled data, otherwise a string used to store in the owner

• owner (object) – an owning class to get and retrieve as an attribute

• cache_global (bool) – cache the data globals; this shares data across instances but not classes

• transient (bool) – the data not persisted to disk after invoking the method

• initial_value (Any) – if provided, the method is never called and this value returned for all invocations

• mkdir (bool) – if path is a :class`.Path` object, then recursively create all directories needed to be able to persist the file without missing directory IO errors

• recover_empty (bool) – if True and a path points to a zero size file, treat it as data that has not yet been generated; this is useful when a previous exception was raised leaving a zero byte file

Deallocate_recursive:

the recursive parameter passed to Deallocate._try_deallocate() to try to deallocate the object graph recursively

clear()

Clear the data, and thus, force it to be created on the next fetch. This is done by removing the attribute from owner, deleting it from globals and removing the file from the disk.

clear_global()

Clear only any cached global data.

deallocate()

Deallocate all resources for this instance.

is_set()

Return whether or not the persisted work has been engaged and has data.

Return type:

bool

set(obj)

Set the contents of the object on the owner as if it were persisted from the source. If this is a global cached instance, then add it to global memory.

write(indent=0, include_content=False, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

class zensols.persist.annotation.persisted(name, path=None, cache_global=False, transient=False, allocation_track=True, mkdir=False, deallocate_recursive=False, recover_empty=False)

Bases: object

Class level annotation to further simplify usage with PersistedWork.

See:

PersistedWork

For example:

class SomeClass(object):
    @property
    @persisted('_counter', 'tmp.dat')
    def counter(self):
        return tuple(range(5))

__init__(name, path=None, cache_global=False, transient=False, allocation_track=True, mkdir=False, deallocate_recursive=False, recover_empty=False)

Initialize.

Parameters:

• name (str) – the name of the attribute on the instance to set with the cached result of the method

• cache_global (bool) – if True, globally cache the value at the class definition level

• transient (bool) – if True do not persist only in memory, and not on the file system, which is needed when used with PersistableContainer

• allocation_track (bool) – if False, immediately mark the backing PersistedWork as deallocated

• mkdir (bool) – if path is a :class`.Path` object, then recursively create all directories needed to be able to persist the file without missing directory IO errors

• recover_empty (bool) – if True and a path points to a zero size file, treat it as data that has not yet been generated; this is useful when a previous exception was raised leaving a zero byte file

Param:

path: if set, the path where to store the cached result on the file system

Deallocate_recursive:

the recursive parameter passed to Deallocate._try_deallocate() to try to deallocate the object graph recursively

class zensols.persist.annotation.resource(create_method_name, destroy_method_name)

Bases: object

This annotation uses a template pattern to (de)allocate resources. For example, you can declare class methods to create database connections and then close them. This example looks like this:

For example:

class CrudManager(object):
    def _create_connection(self):
        return sqlite3.connect(':memory:')

    def _dispose_connection(self, conn):
        conn.close()

    @resource('_create_connection', '_dispose_connection')
    def commit_work(self, conn, obj):
        conn.execute(...)

__init__(create_method_name, destroy_method_name)

Create the instance based annotation.

Parameters:

• create_method_name – the name of the method that allocates

• destroy_method_name – the name of the method that deallocates

zensols.persist.composite module

Stash implementations.

class zensols.persist.composite.DirectoryCompositeStash(path, groups, attribute_name, load_keys=None)

Bases: DirectoryStash

A stash distributes the data of each item out over several directories. On dumping, an attribute holding a dict is removed from the item, it’s data is persisted over multiple directories, then the attribute is restored after pickling.

The data is split up amoung groups of keys in the attribute dict of the item. Persistence works similar to the parent DirectoryStash, except the path points a directory that has an instance of each item without the attribute (called the item instance directory), and the split data (called the composite data directory).

The composite data is grouped across keys from the composite attribute. When the data is loaded, if no load_keys are requested from a group, the data is not accessed. In this way, loading data becomes much faster for very large objects (i.e. matrix/tensor) data.

For this reason, it is important to properly group your load keys so the most related data goes together. This is because if only one key is from the data is needed, the entire composite item is loaded.

Note: If order of the data is important, use an instance of collections.OrderedDict as the attribute data.

COMPOSITE_DIRECTORY_NAME = 'comp'

INSTANCE_DIRECTORY_NAME = 'inst'

__init__(path, groups, attribute_name, load_keys=None)

Initialize using the parent class’s default pattern.

Parameters:

• path (Path) – the directory that will have to subdirectories with the files, they are named INSTANCE_DIRECTORY_NAME and COMPOSITE_DIRECTORY_NAME

• groups (Tuple[Set[str], ...]) – the groups of the dict composite attribute, which are sets of keys, each of which are persisted to their respective directory

• attribute_name (str) – the name of the attribute in each item to split across groups/directories; the instance data to persist has the composite attribute of type dict

• load_keys (Set[str]) – the keys used to load the data from the composite stashs in to the attribute dict instance; only these keys will exist in the loaded data, or None for all keys; this can be set after the creation of the instance as well

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

dump(name, inst)

Persist data value inst with key name.

property groups: Tuple[Set[str], ...]

The groups of the dict composite attribute, which are sets of keys, each of which are persisted to their respective directory.

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

exception zensols.persist.composite.MissingDataKeys(keys)

Bases: PersistableError

__annotations__ = {}

__init__(keys)

__module__ = 'zensols.persist.composite'

zensols.persist.dealloc module

Contains general purpose persistence library classes.

class zensols.persist.dealloc.Deallocatable

Bases: ABC

All subclasses have the ability to deallocate any resources. This is useful for cases where there could be reference cycles or deallocation (i.e. CUDA tensors) need happen implicitly and faster.

classmethod _print_undeallocated(include_stack=False, only_counts=False, fail=False)

Print all unallocated objects.

Parameters:

• include_stack (bool) – if True print out the stack traces of all the unallocated references; if only_counts is True, this is ignored

• only_counts (bool) – if True only print the counts of each unallocated class with counts for each

• fail (bool) – if True, raise an exception if there are any unallocated references found

_deallocate_attribute(attrib)

Deallocate attribute attrib if possible, which means it both exists and extends from this class.

Return type:

bool

static _try_deallocate(obj, recursive=False)

If obj is a candidate for deallocation, deallocate it.

Parameters:

obj (Any) – the object instance to deallocate

Return type:

bool

Returns:

True if the object was deallocated, otherwise return False indicating it can not and was not deallocated

ALLOCATION_TRACKING: ClassVar[bool] = False

Enables allocation tracking. When this if False, this functionality is not used and disabled.

PRINT_TRACE: ClassVar[bool] = False

When True, print the stack trace when deallocating with deallocate().

__init__()

classmethod assert_dealloc()

deallocate()

Deallocate all resources for this instance.

class zensols.persist.dealloc.dealloc(inst, track=False, include_stack=False)

Bases: object

Object used with a with scope for deallocating any subclass of Deallocatable. The first argument can also be a function, which is useful when tracking deallocations when track is True.

Example:

with dealloc(lambda: ImportClassFactory('some/path')) as fac:
    return fac.instance('stash')

__init__(inst, track=False, include_stack=False)

Parameters:

• inst (Union[Callable, Deallocatable]) – either an object instance to deallocate or a callable that creates the instance to deallocate

• track (bool) – when True, set Deallocatable.ALLOCATION_TRACKING to True to start tracking allocations

• include_stack (bool) – adds stack traces in the call to Deallocatable._print_undeallocated()

class zensols.persist.dealloc.dealloc_recursive

Bases: object

__init__()

zensols.persist.domain module

Abstracts the concept of a Python dict with additional functionality.

class zensols.persist.domain.CacheFactoryStash(delegate, factory, enable_preemptive=True, dump_factory_nones=False)

Bases: FactoryStash

Like FactoryStash but suitable for ReadOnlyStash factory instances that have a defined key set and only need a backing stash for caching.

__init__(delegate, factory, enable_preemptive=True, dump_factory_nones=False)

dump_factory_nones: bool = False

Whether to pass on None values to the delegate when the factory creates them.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

class zensols.persist.domain.CloseableStash

Bases: Stash

Any stash that has a resource that needs to be closed.

__init__()

abstract close()

Close all resources created by the stash.

class zensols.persist.domain.DelegateDefaults

Bases: object

Defaults set in DelegateStash.

CLASS_CHECK = False

DELEGATE_ATTR = False

class zensols.persist.domain.DelegateStash(delegate)

Bases: CloseableStash

Delegate pattern. It can also be used as a no-op if no delegate is given.

A minimum functioning implementation needs the load() and keys() methods overriden. Inheriting and implementing a Stash such as this is usually used as the factory in a FactoryStash.

This class delegates attribute fetches to the delegate for the unimplemented methods and attributes using a decorator pattern when attribute delegate_attr is set to True.

Note: Delegate attribute fetching can cause strange and unexpected behavior, so use this funcationlity with care. It is advised to leave it off if unexpected AttributeError are raised due to incorrect attribute is access or method dispatching.

See:

delegate_attr

__init__(delegate)

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

close()

Close all resources created by the stash.

delegate: Stash

The stash to delegate method invocations.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist.

Implementation note: sub classes will probably want to override this method given the super method is cavalier about calling exists:() and load(). Based on the implementation, this can be problematic.

Return type:

Any

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

class zensols.persist.domain.FactoryStash(delegate, factory, enable_preemptive=True, dump_factory_nones=True)

Bases: PreemptiveStash

A stash that defers to creation of new items to another factory stash. It does this by calling first getting the data from the delegate stash, then when it does not exist, it uses the the factory to create the data when loading with load(). Similarly, when accessing with get() or indexing, the factory created item is dumped back to the delegate when the delegate does not have it.

ATTR_EXP_META = ('enable_preemptive',)

__init__(delegate, factory, enable_preemptive=True, dump_factory_nones=True)

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

dump_factory_nones: bool = True

Whether to pass on None values to the delegate when the factory creates them.

enable_preemptive: bool = True

If False, do not invoke the super class’s data calculation.

factory: Stash

The stash used to create using load and keys.

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

class zensols.persist.domain.KeyLimitStash(delegate, limit=9223372036854775807)

Bases: DelegateStash

A stash that limits the number of generated keys useful for debugging.

For most stashes, this also limits the iteration output since that is based on key mapping.

ATTR_EXP_META = ('limit',)

__init__(delegate, limit=9223372036854775807)

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

limit: int = 9223372036854775807

The max number of keys provided as a slice of the delegate’s keys.

class zensols.persist.domain.KeySubsetStash(delegate, key_subset, dynamic_subset)

Bases: ReadOnlyDelegateStash

A stash that exposes a subset of the keys available in the delegate.

__init__(delegate, key_subset, dynamic_subset)

dynamic_subset: InitVar

Whether the delegate keys are dynamic, which forces inefficient key checks on the delegate.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist.

Implementation note: sub classes will probably want to override this method given the super method is cavalier about calling exists:() and load(). Based on the implementation, this can be problematic.

Return type:

Any

key_subset: InitVar

A subset of the keys availble. If this is set to a Path, then the keys are read from a newline delimited file.

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

class zensols.persist.domain.NoopStash

Bases: Stash

A stash that does nothing.

__init__()

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

class zensols.persist.domain.NotPickleable

Bases: object

A class marker that raises an error on attempts to pickle the instance.

class zensols.persist.domain.PreemptiveStash(delegate)

Bases: DelegateStash

Provide support for preemptively creating data in a stash. It provides this with has_data and provides a means of keeping track if the data has yet been created.

Implementation note: This stash retrieves data from the delegate without checking to see if it exists first since the data might not have been (preemptively) yet created.

__init__(delegate)

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

get(name, default=None)

See class doc’s implementation note.

Return type:

Any

property has_data: bool

Return whether or not the stash has any data available or not.

class zensols.persist.domain.PrimablePreemptiveStash(delegate)

Bases: PrimeableStash, PreemptiveStash

A stash that’s primable and preemptive.

__init__(delegate)

class zensols.persist.domain.Primeable

Bases: ABC

Any subclass that has the ability (and need) to do preprocessing. For stashes, this means processing before an CRUD method is invoked. For all other classes it usually is some processing that must be done in a single process.

prime()

class zensols.persist.domain.PrimeableStash

Bases: Stash, Primeable

Any subclass that has the ability to do processing before any CRUD method is invoked.

__init__()

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

prime()

class zensols.persist.domain.ProtectiveStash(delegate, log_errors)

Bases: DelegateStash

A stash that guards dump() so that when Exception is raised, the instance of the exception is dumped instead the instance data.

__init__(delegate, log_errors)

dump(name, inst)

Persist data value inst with key name.

log_errors: bool

When True log caught exceptions as warnings.

class zensols.persist.domain.ReadOnlyDelegateStash(delegate)

Bases: DelegateStash, ReadOnlyStash

Makes any stash read only.

__init__(delegate)

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

class zensols.persist.domain.ReadOnlyStash

Bases: Stash

An abstract base class for subclasses that do not support write methods (i.e. dump()). This class is useful to extend for factory type classes that generate data. Paired with container classes such as DictionaryStash provide persistence in a reusable way.

The only method that needs to be implemented is load() and keys(). However, it is recommended to implement exists() to speed things up.

Setting attribute strict to True will raise a PersistableError for any modification attempts. Otherwise, setting it to False (the default) silently ignores and does nothing on dump(), delete() and clear().

Example:

class RangeStash(ReadOnlyStash):
    def __init__(self, n, end: int = None):
        super().__init__()
        self.n = n
        self.end = end

    def load(self, name: str) -> Any:
        if self.exists(name):
            return name

    def keys(self) -> Iterable[str]:
        if self.end is not None:
            return map(str, range(self.n, self.end))
        else:
            return map(str, range(self.n))

    def exists(self, name: str) -> bool:
        n = int(name)
        if self.end is None:
            if (n >= self.n):
                return False
        elif (n < self.n) or (n >= self.end):
            return False
        return True

__init__()

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

class zensols.persist.domain.Stash

Bases: ABC

A dictionary-like pure virtual class for CRUDing data, most of which read and write to/from the file system. One major difference is dictionaries iterate over keys while stashes iterate over items, which calls items().

Note that there are subtle differences a [Stash] and a dict when generating or accessing data. For example, when indexing obtaining the value is sometimes forced by using some mechanism to create the item. When using get it relaxes this creation mechanism for some implementations.

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

abstract delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

abstract dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any

items()

Return an iterable of all stash items.

Return type:

Tuple[str, Any]

key_groups(n)

Return an iterable of groups of keys, each of size at least n.

abstract keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

abstract load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

values()

Return the values in the hash.

Return type:

Iterable[Any]

class zensols.persist.domain.chunks(iterable, size, enum=False)

Bases: object

An iterable that chunks any other iterable in to chunks. Each element returned is a list of elemnets of the given size or smaller. That element that might be smaller is the remainer of the iterable once it is exhausted.

__init__(iterable, size, enum=False)

Initialize the chunker.

Parameters:

• iterable (iter) – any iterable object

• size (int) – the size of each chunk

zensols.persist.shelve module

Uses the shelve OS level API to CRUD binary data.

class zensols.persist.shelve.ShelveStash(path, writeback=True, auto_close=True)

Bases: CloseableStash

Stash that uses Python’s shelve library to store key/value pairs in DBM databases.

__init__(path, writeback=True, auto_close=True)

auto_close: bool = True

If True, close the shelve for each operation.

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

close()

Close the shelve object, which is needed for data consistency.

delete(name=None)

Delete the shelve data file.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

classmethod get_extension()

Return type:

str

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

path: Path

A file to be created to store and/or load for the data storage.

property real_path: Path

The path the shelve API created on this file system. This is provided since path does not take in to account that some (G)DBM implementations add an extension and others do not This differes across libraries compiled against the Python interpreter and platorm.

property shelve

shelve object.

Type:

Return an opened shelve mod

writeback: bool = True

The writeback parameter given to shelve.

class zensols.persist.shelve.shelve(*args, **kwargs)

Bases: object

Object used with a with scope that creates the closes a shelve object. For example, the following opens a file path, sets a temporary variable stash, prints all the data from the shelve, and then closes it:

Example:

with shelve(path) as stash:
    for id, val in stash, 30:
        print(f'{id}: {val}')

__init__(*args, **kwargs)

zensols.persist.stash module

Stash implementations.

class zensols.persist.stash.CacheStash(delegate, cache_stash=<factory>)

Bases: DelegateStash

Provide a dictionary based caching based stash.

__init__(delegate, cache_stash=<factory>)

cache_stash: Stash

A stash used for caching (defaults to DictionaryStash).

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

get(name, default=None)

Load an object or a default if key name doesn’t exist.

Implementation note: sub classes will probably want to override this method given the super method is cavalier about calling exists:() and load(). Based on the implementation, this can be problematic.

Return type:

Any

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

class zensols.persist.stash.DictionaryStash(_data=<factory>)

Bases: Stash

Use a dictionary as a backing store to the stash. If one is not provided in the initializer a new dict is created.

__init__(_data=<factory>)

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

property data

The dictionary that contains the stash data.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any

keys()

Return an iterable of keys in the collection.

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

class zensols.persist.stash.DirectoryStash(path, pattern='{name}.dat')

Bases: Stash

Creates a pickled data file with a file name in a directory with a given pattern across all instances.

ATTR_EXP_META = ('path', 'pattern')

__init__(path, pattern='{name}.dat')

assert_path_dir()

close()

delete(name)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

key_to_path(name)

Return a path to the pickled data with key name.

Return type:

Path

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

path: Path

The directory of where to store the files.

pattern: str = '{name}.dat'

The file name portion with name populating to the key of the data value.

class zensols.persist.stash.IncrementKeyDirectoryStash(path, pattern='{name}.dat', name='data')

Bases: DirectoryStash

A stash that increments integer value keys in a stash and dumps/loads using the last key available in the stash.

__init__(path, pattern='{name}.dat', name='data')

dump(name_or_inst, inst=None)

If only one argument is given, it is used as the data and the key name is derived from get_last_key.

get_last_key(inc=False)

Get the last available (highest number) keys in the stash.

Return type:

str

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name=None)

Just like Stash.load, but if the key is omitted, return the value of the last key in the stash.

Return type:

Any

name: InitVar = 'data'

The name of the pattern to use in the super class.

class zensols.persist.stash.LRUCacheStash(maxsize)

Bases: DictionaryStash

A stash that removes elements on dump() after size elements are kept.

__init__(maxsize)

dump(name, inst)

Persist data value inst with key name.

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

maxsize: InitVar

The highest number of items stored in the stash before deletions.

class zensols.persist.stash.OneShotFactoryStash(delegate)

Bases: PrimablePreemptiveStash

A stash that is populated by a callable or an iterable worker. The data is generated by the worker and dumped to the delegate. This worker is either a callable (i.e. function) or an interable that return tuples or lists of (key, object)

To use this class, this worker must be set as attribute worker or this class extended and worker be a property.

_get_worker_type()

Return the type of worker. This default implementation returns unknown. If not implemented, when _process_work() is called, the API has use callable() to determine if the worker is a method or property. Doing so accessing the property invoking potentially unecessary work.

Return type:

str

Returns:

u for unknown, m for method (callable), or a for attribute or a property

__init__(delegate)

prime()

class zensols.persist.stash.SortedStash(delegate, sort_function=None)

Bases: DelegateStash

Specify an sorting to how keys in a stash are returned. This usually also has an impact on the sort in which values are iterated since a call to get the keys determins it.

ATTR_EXP_META = ('sort_function',)

__init__(delegate, sort_function=None)

items()

Return an iterable of all stash items.

Return type:

Tuple[str, Any]

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

sort_function: Callable = None

values()

Return the values in the hash.

Return type:

Iterable[Any]

class zensols.persist.stash.UnionStash(stashes)

Bases: ReadOnlyStash

A stash joins the data of many other stashes.

__init__(stashes)

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

stashes: Tuple[Stash, ...]

The delegate constituent stashes used for each operation.

values()

Return the values in the hash.

Return type:

Iterable[Any]

zensols.persist.util module

Utility persistence classes.

class zensols.persist.util.FailureFilterStash(delegate, key_path)

Bases: DelegateStash, PrimeableStash

Filter’s instances of Failure. It does this by reading all load all items of the delegate stash and tracking the keys of which are failures.

__init__(delegate, key_path)

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

key_path: Path

The path of the file where valid keys are stored.

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any

zensols.persist.zip module

A stash that accesses a zip file.

class zensols.persist.zip.ZipStash(path, root=None, encoding=None)

Bases: ReadOnlyStash

Acesss a zip file by using the entry file names as keys and the content of the entries as items. The returned items are either byte arrays if created without an encoding, otherwise decode strings are returned.

A root path can be specified so the zip file appears to have been created in a sub-directory.

Implementation note: keys are cached to speed up access and cleared if

the path set on the instance.

__init__(path, root=None, encoding=None)

See class docs.

Parameters:

• path (Path) – the zip file path

• root (str) – the sub-directory in the zip file to base look ups (see class doc)

• encoding (str) – if provided, returned items will be strings decoded with this encoding (such as utf-8)

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Union[bytearray, str]

property path: Path

The zip file path.

Module contents

Submodules provide both in memory and on disk persistance. These include annotations to provide easy, fast and convenient options to cache expensive to create data (structures).