zensols.deeplearn package

Subpackages

Submodules

zensols.deeplearn.cli module

Command line entry point to the application using the application CLI.

class zensols.deeplearn.cli.ClearType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Indicates what type of data to delete (clear).

batch = 2
none = 1
source = 3
class zensols.deeplearn.cli.FacadeApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)[source]

Bases: Deallocatable

Base class for applications that use ModelFacade.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'option_overrides': {'model_path': {'long_name': 'model', 'short_name': None}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)
cache_global_facade: bool = True

Whether to globally cache the facade returned by get_cached_facade().

clear_cached_facade()[source]

Clear any cached facade this application instance.

config: Configurable

The config used to create facade instances.

config_factory_args: Dict[str, Any]

The arguments given to the ImportConfigFactory, which could be useful for reloading all classes while debugingg.

config_overwrites: Configurable = None

A configurable that clobbers any configuration in config for those sections/options set.

create_facade()[source]

Create a new instance of the facade.

Return type:

ModelFacade

deallocate()[source]

Deallocate all resources for this instance.

facade_name: str = 'facade'

The client facade.

get_cached_facade(**kwargs) → ModelFacade

Return a created facade that is cached in this application instance.

Return type:

ModelFacade

model_config_overwrites: Configurable = None

Configuration that is injected into the model loaded by the model.ModelManager.

model_path: Path = None

The path to the model or use the last trained model if not provided.

class zensols.deeplearn.cli.FacadeApplicationFactory(package_resource, app_config_resource='resources/app.conf', children_configs=None, reload_factory=False, reload_pattern=None, error_handler=None)[source]

Bases: ApplicationFactory

This is a utility class that creates instances of FacadeApplication. It’s only needed if you need to create a facade without wanting invoke the command line attached to the applications.

It does this by only invoking the first pass applications so all the correct initialization happens before returning factory artifacts.

There mst be a FacadeApplication.facade_name entry in the configuration tied to an instance of FacadeApplication.

See:

create_facade()

__init__(package_resource, app_config_resource='resources/app.conf', children_configs=None, reload_factory=False, reload_pattern=None, error_handler=None)
create_facade(args=None, app_args=None)[source]

Create the facade tied to the application without invoking the command line.

Parameters:

  • args (List[str]) – the (would be) command line arguments used to create the application

  • app_args (Dict[str, Any]) – the arguments to set on the the facade application after it is created and before it creates the facade

Return type:

ModelFacade

class zensols.deeplearn.cli.FacadeApplicationManager(cli_harness, cli_args_fn=<function FacadeApplicationManager.<lambda>>, reset_torch=True, allocation_tracking=False, logger_name='notebook', default_logging_level='WARNING', progress_bar_cols=120, config_overwrites=<factory>)[source]

Bases: Writable

A very high level client interface making it easy to configure and run models from an interactive environment such as a Python REPL or a Jupyter notebook (see JupyterManager)

__init__(cli_harness, cli_args_fn=<function FacadeApplicationManager.<lambda>>, reset_torch=True, allocation_tracking=False, logger_name='notebook', default_logging_level='WARNING', progress_bar_cols=120, config_overwrites=<factory>)
allocation_tracking: Union[bool, str] = False

Whether or not to track resource/memory leaks. If set to stack, the stack traces of the unallocated objects will be printed. If set to counts only the counts will be printed. If set to True only the unallocated objects without the stack will be printed.

cleanup(include_cuda=True, quiet=False)[source]

Report memory leaks, run the Python garbage collector and optionally empty the CUDA cache.

Parameters:

  • include_cuda (bool) – if True clear the GPU cache

  • quiet (bool) – do not report unallocated objects, regardless of the setting of allocation_tracking

clear()[source]

Clear all post create configuration set with config().

cli_args_fn()

Creates the arguments used to create the facade from the application factory.

cli_harness: CliHarness

The CLI harness used to create the facade application.

config(section, **kwargs)[source]

Add overwriting configuration used when creating the facade.

Parameters:

  • section (str) – the section to be overwritten (or added)

  • kwargs – the key/value pairs used as the section data to overwrite

See:

create_facade()

config_overwrites: Dict[str, Dict[str, str]]

Clobbers any configuration set by config() for those sections/options set.

create_facade(*args, **kwargs)[source]

Create and return a facade. This deallocates and cleans up state from any previous facade creation as a side effect.

Parameters:

args – given to the cli_args_fn function to create arguments passed to the CLI

Return type:

ModelFacade

deallocate()[source]

Deallocate all resources in the CLI factory if it exists.

default_logging_level: str = 'WARNING'

If set, then initialize the logging system using this as the default

logging level. This is the upper case logging name such as WARNING.

property facade: ModelFacade

The current facade for this notebook instance.

Returns:

the existing facade, or that created by create_facade() if it doesn’t already exist

logger_name: str = 'notebook'

The name of the logger to use for logging in the notebook itself.

progress_bar_cols: int = 120

The number of columns to use for the progress bar.

reset_torch: bool = True

Reset random state for consistency for each new created facade.

run(display_results=True)[source]

Train, test and optionally show results.

Parameters:

display_results (bool) – if True, write and plot the results

show_leaks(output='counts', fail=True)[source]

Show all resources/memory leaks in the current facade. First, this deallocates the facade, then prints any lingering objects using Deallocatable.

Important: allocation_tracking must be set to True for this to work.

Parameters:

  • output (str) – one of stack, counts, or tensors

  • fail (bool) – if True, raise an exception if there are any unallocated references found

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_model=False, include_metadata=False, include_settings=False, **kwargs)[source]

Write the contents of this instance to writer using indention depth.

Parameters:

  • depth (int) – the starting indentation depth

  • writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.cli.FacadeBatchApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)[source]

Bases: FacadeApplication

Test, train and validate models.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'mnemonic_overrides': {'batch': {'option_includes': {'clear_type', 'limit', 'split'}}}, 'option_overrides': {'clear': {'short_name': None}, 'clear_type': {'long_name': 'ctype', 'short_name': None}, 'limit': {'short_name': None}, 'model_path': {'long_name': 'model', 'short_name': None}, 'split': {'short_name': None}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)
batch(limit=None, clear_type=ClearType.none, split=False)[source]

Create batches if not already, print statistics on the dataset.

Parameters:

  • clear_type (ClearType) – what to delete to force recreate

  • limit (int) – the number of batches to create

  • split (bool) – also write the stratified splits if available

class zensols.deeplearn.cli.FacadeInfoApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, feature_stash_name='feature_stash')[source]

Bases: FacadeApplication

Contains methods that provide information about the model via the facade.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'mnemonic_overrides': {'print_information': 'info'}, 'option_excludes': {'feature_stash_name'}, 'option_overrides': {'debug_value': {'long_name': 'execlevel', 'short_name': None}, 'info_item': {'long_name': 'item', 'short_name': 'i'}, 'model_path': {'long_name': 'model', 'short_name': None}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, feature_stash_name='feature_stash')
debug(debug_value=None)[source]

Debug the model.

Parameters:

debug_value (int) – the executor debugging level

feature_stash_name: str = 'feature_stash'

The section name of the stash to write for InfoItem.feature.

print_information(info_item=None, model_path=None)[source]

Output facade data set, vectorizer and other configuration information.

Parameters:

  • info_item (InfoItem) – what to print

  • model_path (Path) – the path to the model or use the last trained model if not provided

class zensols.deeplearn.cli.FacadeModelApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, use_progress_bar=False)[source]

Bases: FacadeApplication

Test, train and validate models.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'mnemonic_overrides': {'early_stop': {'name': 'stop', 'option_includes': {}}, 'train_production': 'trainprod'}, 'option_overrides': {'model_path': {'long_name': 'model', 'short_name': None}, 'use_progress_bar': {'long_name': 'progress', 'short_name': 'p'}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, use_progress_bar=False)
create_facade()[source]

Create a new instance of the facade.

Return type:

ModelFacade

early_stop()[source]

Stops the execution of training the model.

test(model_path=None)[source]

Test an existing model the model and dump the results of the test.

Parameters:

model_path (Path) – the path to the model or use the last trained model if not provided

train()[source]

Train the model and dump the results, including a graph of the train/validation loss.

train_production()[source]

Train, test the model on train and test datasets, then dump the results with a graph.

train_test()[source]

Train, test the model, then dump the results with a graph.

use_progress_bar: bool = False

Display the progress bar.

class zensols.deeplearn.cli.FacadePackageApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, packer=None)[source]

Bases: FacadeApplication

Contains methods that package models.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'option_excludes': {'packer'}, 'option_overrides': {'archive_model_dir': {'long_name': 'archdir', 'metavar': 'DIR'}, 'model_path': {'long_name': 'model', 'short_name': None}, 'train_model_dir': {'long_name': 'modeldir', 'metavar': 'DIR'}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None, packer=None)
pack(res_id=None, archive_model_dir=PosixPath('.'))[source]

Package a distribution model.

Parameters:

  • res_id (str) – the result ID or use the last if not given

  • archive_model_dir (Path) – the directory where the packaged model is written

packer: ModelPacker = None

The model packer used to create the model distributions from this app.

update_config(res_id=None, train_model_dir=None)[source]

Update a model’s configuration.

Parameters:

  • res_id (str) – the result ID or use the last if not given

  • train_model_dir (Path) – the trained model directory, which has the weight and state files

class zensols.deeplearn.cli.FacadePredictApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)[source]

Bases: FacadeApplication

An applicaiton that provides prediction funtionality.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'mnemonic_overrides': {'outcomes': 'respreds'}, 'option_overrides': {'model_path': {'long_name': 'model', 'short_name': None}}}

Tell the command line app API to igonore subclass and client specific use case methods.

outcomes(res_id=None, out_file=None)[source]

Write labels and predictions from the test set to a CSV file.

Parameters:

  • res_id (str) – the result ID or use the last if not given

  • out_file (Path) – the output path or - for standard out

class zensols.deeplearn.cli.FacadeResultApplication(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)[source]

Bases: FacadeApplication

Contains methods that dump previous results.

CLI_META = {'mnemonic_excludes': {'clear_cached_facade', 'create_facade', 'deallocate', 'get_cached_facade'}, 'mnemonic_overrides': {'all_runs': 'resall', 'compare_results': 'rescmp', 'latex': 'reslatex', 'majority_label_metrics': 'resmajority', 'metrics': 'reslabel', 'result_ids': 'resids', 'summary': 'ressum'}, 'option_overrides': {'describe': {'short_name': None}, 'include_validation': {'long_name': 'validation', 'short_name': None}, 'model_path': {'long_name': 'model', 'short_name': None}, 'out_file': {'long_name': 'outfile', 'short_name': 'o'}, 'out_format': {'long_name': 'format', 'short_name': 'f'}, 'sort': {'short_name': 's'}}}

Tell the command line app API to igonore subclass and client specific use case methods.

__init__(config, facade_name='facade', model_path=None, config_factory_args=<factory>, config_overwrites=None, cache_global_facade=True, model_config_overwrites=None)
all_runs(out_file=None, include_validation=False)[source]

Create a summary of all archived results.

Parameters:

  • out_file (Path) – the output path or - for standard out

  • include_validation (bool) – whether to include validation results

compare_results(res_id_a, res_id_b)[source]

Compare two previous archived result sets.

Parameters:

  • res_id_a (str) – the first result ID to compare

  • res_id_b (str) – the second result ID to compare

latex(res_id=None, out_file=None)[source]

Create Zensols LaTeX ready results.

Parameters:

  • res_id (str) – the result ID or use the last if not given

  • out_file (Path) – the output path or - for standard out

majority_label_metrics(res_id=None)[source]

Show majority label metrics of the test dataset using a previous result set.

Parameters:

res_id (str) – the result ID or use the last if not given

metrics(sort='wF1', res_id=None, out_file=None, describe=False)[source]

List the performance results by label.

Parameters:

  • sort_col – the column to sort results

  • res_id (str) – the result ID or use the last if not given

  • out_file (Path) – the output path or - for standard out

  • describe (bool) – whether to create Zensols LaTeX results

result_ids()[source]

Show all archived result IDs.

summary(res_id=None, out_file=None, out_format=Format.txt)[source]

List the performance results as a summary.

Parameters:

  • res_id (str) – the result ID or use the last if not given

  • out_file (Path) – the output path or - for standard out

  • out_format (Format) – the output format

class zensols.deeplearn.cli.Format(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

csv = 4
json = 2
txt = 1
yaml = 3
class zensols.deeplearn.cli.InfoItem(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Indicates what information to dump in FacadeInfoApplication.print_information().

batch = 6
config = 4
feature = 5
meta = 1
model = 3
param = 2
class zensols.deeplearn.cli.JupyterManager(cli_harness, cli_args_fn=<function FacadeApplicationManager.<lambda>>, reset_torch=True, allocation_tracking=False, logger_name='notebook', default_logging_level='WARNING', progress_bar_cols=120, config_overwrites=<factory>, reduce_logging=False, browser_width=95)[source]

Bases: FacadeApplicationManager

A facade application manager that provides additional convenience functionality.

__init__(cli_harness, cli_args_fn=<function FacadeApplicationManager.<lambda>>, reset_torch=True, allocation_tracking=False, logger_name='notebook', default_logging_level='WARNING', progress_bar_cols=120, config_overwrites=<factory>, reduce_logging=False, browser_width=95)
browser_width: int = 95

The width of the browser windows as a percentage.

create_facade(*args, **kwargs)[source]

Create and return a facade. This deallocates and cleans up state from any previous facade creation as a side effect.

Parameters:

args – given to the cli_args_fn function to create arguments passed to the CLI

Return type:

ModelFacade

reduce_logging: bool = False

Whether to disable most information logging so the progress bar is more prevalent.

static set_browser_width(width=95)[source]

Use the entire width of the browser to create more real estate.

Parameters:

width (int) – the width as a percent ([0, 100]) to use as the width in the notebook

zensols.deeplearn.domain module

This file contains classes that configure the network and classifier runs.

class zensols.deeplearn.domain.ActivationNetworkSettings(name, config_factory, torch_config, activation)[source]

Bases: NetworkSettings

A network settings that contains a activation setting and creates a activation layer.

__init__(name, config_factory, torch_config, activation)
activation: Union[Callable, Module, str]

The function between all layers, or None for no activation.

property activation_function: Callable
static get_activation_function(activation)[source]
class zensols.deeplearn.domain.BatchNormNetworkSettings(name, config_factory, torch_config, batch_norm_d, batch_norm_features)[source]

Bases: NetworkSettings

A network settings that contains a batchnorm setting and creates a batchnorm layer.

__init__(name, config_factory, torch_config, batch_norm_d, batch_norm_features)
batch_norm_d: int

The dimension of the batch norm or None to disable. Based on this one of the following is used as a layer:

  • torch.nn.BatchNorm1d

  • torch.nn.BatchNorm2d

  • torch.nn.BatchNorm3d

batch_norm_features: int

The number of features to use in the batch norm layer.

property batch_norm_layer
static create_batch_norm_layer(batch_norm_d, batch_norm_features)[source]
create_new_batch_norm_layer(batch_norm_d=None, batch_norm_features=None)[source]
class zensols.deeplearn.domain.DatasetSplitType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: Enum

Indicates an action on the model, which is first trained, validated, then tested.

Implementation note: for now test is used for both testing the model and ad-hoc prediction

test = 3
train = 1
validation = 2
exception zensols.deeplearn.domain.DeepLearnError[source]

Bases: APIError

Raised for any frame originated error.

__annotations__ = {}
__module__ = 'zensols.deeplearn.domain'
class zensols.deeplearn.domain.DropoutNetworkSettings(name, config_factory, torch_config, dropout)[source]

Bases: NetworkSettings

A network settings that contains a dropout setting and creates a dropout layer.

__init__(name, config_factory, torch_config, dropout)
dropout: float

The droput used in all layers or None to disable.

property dropout_layer
exception zensols.deeplearn.domain.EarlyBailError[source]

Bases: DeepLearnError

Convenience used for helping debug the network.

__annotations__ = {}
__init__()[source]
__module__ = 'zensols.deeplearn.domain'
exception zensols.deeplearn.domain.ModelError[source]

Bases: DeepLearnError

Raised for any model related error.

__annotations__ = {}
__module__ = 'zensols.deeplearn.domain'
class zensols.deeplearn.domain.ModelSettings(name, config_factory, model_name, path, learning_rate, epochs, append_model_path=None, max_consecutive_increased_count=9223372036854775807, nominal_labels=True, batch_iteration_class_name=None, criterion_class_name=None, optimizer_class_name=None, optimizer_params=None, clip_gradient_threshold=None, scale_gradient_params=None, scheduler_class_name=None, scheduler_params=None, reduce_outcomes='argmax', shuffle_training=False, batch_limit=9223372036854775807, batch_iteration='cpu', prediction_mapper_name=None, cache_batches=True, gc_level=0, observer_manager=<factory>, store_model_result='test', store_report=True)[source]

Bases: Writeback, PersistableContainer

This configures and instance of ModelExecutor. This differes from NetworkSettings in that it configures the model parameters, and not the neural network parameters.

Another reason for these two separate classes is data in this class is not needed to rehydrate an instance of torch.nn.Module.

The loss function strategy across parameters nominal_labels, criterion_class and optimizer_class, must be consistent. The defaults uses nominal labels, which means a single integer, rather than one hot encoding, is used for the labels. Most loss function, including the default torch.nn.CrossEntropyLoss` uses nominal labels. The optimizer defaults to torch.optim.Adam.

However, if nominal_labels is set to False, it is expected that the label output is a Long one hot encoding of the class label that must be decoded with BatchIterator._decode_outcomes() and uses a loss function such as torch.nn.BCEWithLogitsLoss, which applies a softmax over the output to narow to a nominal.

If the criterion_class is left as the default, the class the corresponding class across these two is selected based on nominal_labels.

Note: Instances of this class are pickled as parts of the results in zensols.deeplearn.result.domain.ModelResult, so they must be able to serialize. However, they are not used to restore the executor or model, which are instead, recreated from the configuration for each (re)load (see the package documentation for more information).

See:

NetworkSettings

__init__(name, config_factory, model_name, path, learning_rate, epochs, append_model_path=None, max_consecutive_increased_count=9223372036854775807, nominal_labels=True, batch_iteration_class_name=None, criterion_class_name=None, optimizer_class_name=None, optimizer_params=None, clip_gradient_threshold=None, scale_gradient_params=None, scheduler_class_name=None, scheduler_params=None, reduce_outcomes='argmax', shuffle_training=False, batch_limit=9223372036854775807, batch_iteration='cpu', prediction_mapper_name=None, cache_batches=True, gc_level=0, observer_manager=<factory>, store_model_result='test', store_report=True)
append_model_path: str = None

Whether and how to append the model’s name to the end of path. If this value is verbatim, append the model name as provided with model_name, if normalize use normalize_name() to normalize it, and if None do not append anything.

batch_iteration: str = 'cpu'

How the batches are buffered, which is one of:

  • gpu, buffers all data in the GPU

  • cpu, which means keep all batches in CPU memory (the default)

  • buffered which means to buffer only one batch at a time (only for very large data).

batch_iteration_class_name: InitVar = None

A string fully qualified class name of type BatchIterator. This must be set to a class such as ScoredBatchIterator to handle descrete states in the output layer such as terminating CRF states. The default is BatchIterator, which expects continuous output layers.

batch_limit: Union[int, float] = 9223372036854775807

The max number of batches to train, validate and test on, which is useful for limiting while debuging; defaults to sys.maxsize. If this value is a float, it is assumed to be a number between [0, 1] and the number of batches is multiplied by the value.

cache_batches: bool = True

If True cache unthawed/processed batches when possible.

clip_gradient_threshold: float = None

Parameters passed to torch.nn.utils.clip_grad_value_() to clip gradients above this threshold.

criterion_class_name: InitVar = None

The loss function class name (see class doc).

epochs: int

The number of epochs to train the network.

gc_level: int = 0

The frequency by with the garbage collector is invoked. The higher the value, the more often it will be run during training, testing and validation.

learning_rate: float

Learning_rate used for the gradient descent step (done in the optimzer).

max_consecutive_increased_count: int = 9223372036854775807

The maximum number of times the validation loss can increase per epoch before the executor “gives up” and early stops training.

model_name: str

A human readable name for the model.

nominal_labels: bool = True

True if using numbers to identify the class as an enumeration rather than a one hot encoded array.

property normal_model_name: str

Return the normalized model_name using normalize_name().

static normalize_name(name)[source]

Normalize the name in to a string that is more file system friendly. This is used for the model_name by API components that write data to the file system about the model this class configures such as ModelResultManager.

Return type:

str

Returns:

the normalized name

observer_manager: ModelObserverManager

The model observer used by the entire train, test, validation process.

optimizer_class_name: InitVar = None

The optimization algorithm class name (see class doc).

optimizer_params: Dict[str, Any] = None

The parameters given as **kwargs when creating the optimizer. Do not add the learning rate, instead see learning_rate.

path: Path

The path to save and load the model.

prediction_mapper_name: str = None

Creates data points from a client for the purposes of prediction. This value is the string class name of an instance of PredictionMapper used to create predictions. While optional, if not set, ad-hoc predictions (i.e. from the command line) can not be created.

Instances of PredictionMapper are created and managed in the ModelFacade.

reduce_outcomes: str = 'argmax'

The method by which the labels and output is reduced. The output is optionally reduced, which is one of the following:

  • argmax: uses the index of the largest value, which is used for classification models and the default

  • softmax: just like argmax but applies a softmax

  • none: return the identity.

scale_gradient_params: Dict[str, Union[float, bool]] = None

Parameters passed to torch.nn.utils.clip_grad_norm_() to scale the gradient norm.

scheduler_class_name: str = None

The fully qualified class name of the learning rate scheduler used for the optimizer (if not None) such as:

  • torch.optim.lr_scheduler.StepLR or,

  • torch.optim.lr_scheduler.ReduceLROnPlateau.

See:

scheduler_params

scheduler_params: Dict[str, Any] = None

The parameters given as **kwargs when creating the scheduler (if any).

See:

scheduler_class_name

shuffle_training: bool = False

If True shuffle the training data set split before the training process starts. The shuffling only happens once for all epocs.

store_model_result: str = 'test'

Whether to store the ModelResult instance in the state file, which is one of:

  • test: only tested models, as apposed to using train_production()

  • always: always save results, even in production models

  • never: there will be no training or validation results in output

The results are also stored as .dat files in the results directory.

See:

store_report

store_report: bool = True

Whether to store the contents of the results report when the results aren’t persisted with the model. The report is store as key model_result_report in the state.pt that lives in the model directory with weight.pt.

See:

store_model_result

class zensols.deeplearn.domain.NetworkSettings(name, config_factory, torch_config)[source]

Bases: Writeback, PersistableContainer

A container settings class for network models. This abstract class must return the fully qualified (with module name) PyTorch model (`torch.nn.Module`) that goes along with these settings. An instance of this class is saved in the model file and given back to it when later restored.

Note: Instances of this class are pickled as parts of the results in zensols.deeplearn.result.domain.ModelResult, so they must be able to serialize. However, they are not used to restore the executor or model, which are instead, recreated from the configuration for each (re)load (see the package documentation for more information).

See:

ModelSettings

__init__(name, config_factory, torch_config)
config_factory: ConfigFactory

The configuration factory used to create the module.

create_module(*args, **kwargs)[source]

Create a new instance of the network model.

Return type:

Module

abstract get_module_class_name()[source]

Returns the fully qualified class name of the module to create by ModelManager. This module takes as the first parameter an instance of this class.

Important: This method is not used for nested modules. You must declare specific class names in the configuration for those nested class naems.

Return type:

str

torch_config: TorchConfig

The GPU configuration for torch.nn.Module layers.

zensols.deeplearn.observer module

Contains a simple but effective observer pattern set of classes for training, testing and validating models.

class zensols.deeplearn.observer.DumperObserver(events=<factory>, flatten=True, flatten_short_classes=True, output_file=PosixPath('dumper-observer.csv'), file_mode='append', trigger_events=<factory>, trigger_callers=None, mkdir=True, add_columns=None)[source]

Bases: RecorderObserver

A class that dumps all data when certain events are received as a CSV to the file sytsem.

__init__(events=<factory>, flatten=True, flatten_short_classes=True, output_file=PosixPath('dumper-observer.csv'), file_mode='append', trigger_events=<factory>, trigger_callers=None, mkdir=True, add_columns=None)
add_columns: Dict[str, Any] = None

Additional columns to add to the data frame across all rows if given.

file_mode: str = 'append'

If append, then append data to the output .CSV file. Otherwise, if overwrite then overwrite the data.

mkdir: bool = True

If True then create the parent directories if they don’t exist.

notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

output_file: Path = PosixPath('dumper-observer.csv')

The path to where the (flattened data) is written.

trigger_callers: Set[Union[str, Type]] = None

A set of all callers’ fully qualified class names. If set to None the caller is not a constraint that precludes the dump.

trigger_events: Set[str]

A set of all events received that trigger a dump.

class zensols.deeplearn.observer.FilterModelObserver(delegate, include_events=<factory>)[source]

Bases: ModelObserver

Filters messages from the client to a delegate observer.

__init__(delegate, include_events=<factory>)
delegate: ModelObserver

The delegate observer to notify on notifications from this observer.

include_events: Set[str]

A set of events used to indicate to notify delegate.

notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

class zensols.deeplearn.observer.LogModelObserver(logger=<Logger zensols.deeplearn.observer.event (WARNING)>, level=20, add_context_format='{event}: {context}')[source]

Bases: ModelObserver

Logs notifications to logging system.

__init__(logger=<Logger zensols.deeplearn.observer.event (WARNING)>, level=20, add_context_format='{event}: {context}')
add_context_format: str = '{event}: {context}'

If not None, use the string to format the log message.

level: int = 20

The level used for logging.

logger: Logger = <Logger zensols.deeplearn.observer.event (WARNING)>

The logger that receives notifications.

notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

class zensols.deeplearn.observer.ModelObserver[source]

Bases: ABC

Recipient of notifications by the model framework.

abstract notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

class zensols.deeplearn.observer.ModelObserverManager(observers=<factory>)[source]

Bases: object

__init__(observers=<factory>)
add(observer)[source]

Add an observer to be notified of event changes.

notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

observers: List[ModelObserver]

A list of observers that get notified of all model lifecycle and process events.

class zensols.deeplearn.observer.RecorderObserver(events=<factory>, flatten=True, flatten_short_classes=True)[source]

Bases: ModelObserver

Records notifications and provides them as output.

__init__(events=<factory>, flatten=True, flatten_short_classes=True)
events: List[Tuple[datetime, str, Any, Any]]

All events received by this observer thus far.

events_as_df()[source]
Return type:

DataFrame

flatten: bool = True

Whether or not make the caller and context in to a strings before storing them in events.

flatten_short_classes: bool = True

If True, then only use the class name sans module. Otherwise, use the fully qualified class name.

notify(event, caller, context=None)[source]

Notify all registered observers of an event.

Parameters:

  • event (str) – the unique identifier of the event using underscore spacing and prefixed by a unique identifier per caller

  • caller (Any) – the object calling this method

  • context (Any) – any object specific to the call and understood by the client on a per client basis

zensols.deeplearn.observer.event_logger = <Logger zensols.deeplearn.observer.event (WARNING)>

`.LogModelObserver.

Type:

Event logger for the

Type:

class

zensols.deeplearn.observer.mod_logger = <Logger zensols.deeplearn.observer.status (WARNING)>

Logger for this module.

zensols.deeplearn.torchconfig module

CUDA access and utility module.

class zensols.deeplearn.torchconfig.CudaInfo[source]

Bases: Writable

A utility class that provides information about the CUDA configuration for the current (hardware) environment.

get_devices(format=False)[source]
Return type:

Dict[int, Dict[str, Any]]

property gpu_available: bool
property num_devices: int

Return number of devices connected.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Class representation as number of devices connected and about them.

See:

cuda

class zensols.deeplearn.torchconfig.TorchConfig(use_gpu=True, data_type=torch.float32, cuda_device_index=None, device_name=None)[source]

Bases: PersistableContainer, Writable

A utility class that provides access to CUDA APIs. It provides information on the current CUDA configuration and convenience methods to create, copy and modify tensors. These are handy for any given CUDA configuration and can back off to the CPU when CUDA isn’t available.

__init__(use_gpu=True, data_type=torch.float32, cuda_device_index=None, device_name=None)[source]

Initialize this configuration.

Parameters:

  • use_gpu (bool) – whether or not to use CUDA/GPU

  • data_type (type) – the default data type to use when creating new tensors in this configuration

  • cuda_device_index (int) – the CUDA device to use, which defaults to 0 if CUDA if use_gpu is True

  • device_name (str) – the string name of the device to use (i.e. cpu or mps); if provided, overrides cuda_device_index

cat(*args, **kwargs)[source]

Concatenate tensors in to one tensor using torch.cat.

Return type:

Tensor

clone(tensor, requires_grad=True)[source]

Clone a tensor.

Return type:

Tensor

static close(a, b)[source]

Return whether or not two tensors are equal. This does an exact cell comparison.

Return type:

bool

property cpu_device: torch.device

Return the CPU CUDA device, which is the device type configured to utilize the CPU (rather than the GPU).

classmethod cpu_device_name()[source]

The string name of the torch CPU device.

Return type:

str

cross_entropy_pad(size)[source]

Create a padded tensor of size size using the repeated pad ignore_index.

Return type:

Tensor

property cuda_configs: Tuple[TorchConfig]

Return a new set of configurations, one for each CUDA device.

property cuda_device_index: int | None

Return the CUDA device index if CUDA is being used for this configuration. Otherwise return None.

property cuda_devices: Tuple[device]

Return all cuda devices.

property device: device

Return the torch device configured.

empty(*args, **kwargs)[source]

Return a new tesor using torch.empty.

Return type:

Tensor

static empty_cache()[source]

Empty the CUDA torch cache. This releases memory in the GPU and should not be necessary to call for normal use cases.

static equal(a, b)[source]

Return whether or not two tensors are equal. This does an exact cell comparison.

Return type:

bool

float(*args, **kwargs)[source]

Return a new tensor using torch.tensor as a float type.

Return type:

Tensor

property float_type: Type

Return the float type that represents this configuration, converting to the corresponding precision from integer if necessary.

Returns:

the float that represents this data, or None if neither float nor int

from_iterable(array)[source]

Return a one dimenstional tensor created from array using the type and device in the current instance configuration.

Return type:

Tensor

from_numpy(arr)[source]

Return a new tensor generated from a numpy aray using torch.from_numpy. The array type is converted if necessary.

Return type:

Tensor

classmethod get_random_seed()[source]

Get the cross system random seed, meaning the seed applied to CUDA and the Python random library.

Return type:

int

classmethod get_random_seed_context()[source]

Return the random seed context given to set_random_seed() to restore across models for consistent results.

Return type:

Dict[str, Any]

property gpu_available: bool

Return whether or not CUDA GPU access is available.

static in_memory_tensors()[source]

Returns all in-memory tensors and parameters.

See:

show_leaks()

Return type:

List[Tensor]

property info: CudaInfo

Return the CUDA information, which include specs of the device.

classmethod init(spawn_multiproc='spawn', seed_kwargs={})[source]

Initialize the PyTorch framework. This includes:

  • Configuration of PyTorch multiprocessing so subprocesses can access the GPU, and

  • Setting the random seed state.

The needs to be initialized at the very beginning of your program.

Example:

def main():
    from zensols.deeplearn import TorchConfig
    TorchConfig.init()

Note: this method is separate from set_random_seed() because that method is called by the framework to reset the seed after a model is unpickled.

See:

torch.multiprocessing

See:

set_random_seed()

int(*args, **kwargs)[source]

Return a new tensor using torch.tensor as a int type.

Return type:

Tensor

property int_type: Type

Return the int type that represents this configuration, converting to the corresponding precision from integer if necessary.

Returns:

the int that represents this data, or None if neither int nor float

classmethod is_on_cpu(arr)[source]

Return True if the passed tensor is on the CPU.

Return type:

bool

is_sparse(arr)[source]

Return whether or not a tensor a sparse.

Return type:

bool

property numpy_data_type: Type[dtype]

Return the numpy type that corresponds to this instance’s configured data_type.

ones(*args, **kwargs)[source]

Return a new tensor of zeros using torch.ones.

Return type:

Tensor

same_device(tensor_or_model)[source]

Return whether or not a tensor or model is in the same memory space as this configuration instance.

Return type:

bool

classmethod set_random_seed(seed=0, disable_cudnn=True, rng_state=True)[source]

Set the random number generator for PyTorch.

Parameters:

  • seed (int) – the random seed to be set

  • disable_cudnn (bool) – if True disable NVidia’s backend cuDNN hardware acceleration, which might have non-deterministic features

  • rng_state (bool) – set the CUDA random state array to zeros

See:

Torch Random Seed

See:

Reproducibility

singleton(*args, **kwargs)[source]

Return a new tensor using torch.tensor.

Return type:

Tensor

sparse(indicies, values, shape)[source]

Create a sparce tensor from indexes and values.

property tensor_class: Type[dtype]

Return the class type based on the current configuration of this instance. For example, if using torch.float32 on the GPU, torch.cuda.FloatTensor is returned.

to(tensor_or_model)[source]

Copy the tensor or model to the device this to that of this configuration.

Return type:

Union[Module, Tensor]

classmethod to_cpu_deallocate(*arrs)[source]

Safely copy detached memory to the CPU and delete local instance (possibly GPU) memory to speed up resource deallocation. If the tensor is already on the CPU, it’s simply passed back. Otherwise the tensor is deleted.

This method is robust with None, which are skipped and substituted as None in the output.

Parameters:

arrs (Tuple[Tensor]) – the tensors the copy to the CPU (if not already)

Return type:

Union[Tuple[Tensor], Tensor]

Returns:

the singleton tensor if only one arrs is passed; otherwise, the CPU copied tensors from the input

to_type(arr)[source]

Convert the type of the given array to the type of this instance.

Return type:

Tensor

property using_cpu: bool

Return True if this configuration is using the CPU device.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Write the contents of this instance to writer using indention depth.

Parameters:

  • depth (int) – the starting indentation depth

  • writer (TextIOBase) – the writer to dump the content of this writable

write_device_tensors(writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)[source]

Like write_in_memory_tensors(), but filter on this instance’s device.

Parameters:

filter_device – if given, write only tensors matching this device

See:

TorchConfig

classmethod write_in_memory_tensors(writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, filter_device=None)[source]

Prints in-memory tensors and parameters.

Parameters:

filter_device (device) – if given, write only tensors matching this device

See:

TorchConfig

zeros(*args, **kwargs)[source]

Return a new tensor of zeros using torch.zeros.

Return type:

Tensor

class zensols.deeplearn.torchconfig.printopts(**kwargs)[source]

Bases: object

Object used with a with scope that sets options, then sets them back.

Example:

with printopts(profile='full', linewidth=120):
    print(tensor)
See:

PyTorch Documentation

DEFAULTS = {'edgeitems': 3, 'linewidth': 80, 'precision': 4, 'profile': 'default', 'sci_mode': None, 'threshold': 1000}
__init__(**kwargs)[source]

zensols.deeplearn.torchtype module

CUDA access and utility module.

class zensols.deeplearn.torchtype.TorchTypes[source]

Bases: object

A utility class to convert betwen numpy and torch classes. It also provides metadata for types that make other conversions, such as same precision cross types (i.e. int64 -> float64).

FLOAT_TO_INT = {torch.float16: torch.int16, torch.float32: torch.int32, torch.float64: torch.int64}
FLOAT_TYPES = frozenset({torch.float16, torch.float32, torch.float64})
INT_TO_FLOAT = {torch.int16: torch.float16, torch.int32: torch.float32, torch.int64: torch.float64}
INT_TYPES = frozenset({torch.int16, torch.int32, torch.int64})
NAME_TO_TYPE = {'bool': {'cpu': <class 'torch.BoolTensor'>, 'desc': 'Boolean', 'gpu': <class 'torch.cuda.BoolTensor'>, 'name': 'bool', 'numpy': <class 'bool'>, 'types': {torch.bool}}, 'float16': {'cpu': <class 'torch.HalfTensor'>, 'desc': '16-bit floating point', 'gpu': <class 'torch.cuda.HalfTensor'>, 'name': 'float16', 'numpy': <class 'numpy.float16'>, 'sparse': <class 'torch.sparse.HalfTensor'>, 'types': {torch.float16}}, 'float32': {'cpu': <class 'torch.FloatTensor'>, 'desc': '32-bit floating point', 'gpu': <class 'torch.cuda.FloatTensor'>, 'name': 'float32', 'numpy': <class 'numpy.float32'>, 'sparse': <class 'torch.sparse.FloatTensor'>, 'types': {torch.float32}}, 'float64': {'cpu': <class 'torch.DoubleTensor'>, 'desc': '64-bit floating point', 'gpu': <class 'torch.cuda.DoubleTensor'>, 'name': 'float64', 'numpy': <class 'numpy.float64'>, 'sparse': <class 'torch.sparse.DoubleTensor'>, 'types': {torch.float64}}, 'int16': {'cpu': <class 'torch.ShortTensor'>, 'desc': '16-bit integer (signed)', 'gpu': <class 'torch.cuda.ShortTensor'>, 'name': 'int16', 'numpy': <class 'numpy.int16'>, 'sparse': <class 'torch.sparse.ShortTensor'>, 'types': {torch.int16}}, 'int32': {'cpu': <class 'torch.IntTensor'>, 'desc': '32-bit integer (signed)', 'gpu': <class 'torch.cuda.IntTensor'>, 'name': 'int32', 'numpy': <class 'numpy.int32'>, 'sparse': <class 'torch.sparse.IntTensor'>, 'types': {torch.int32}}, 'int64': {'cpu': <class 'torch.LongTensor'>, 'desc': '64-bit integer (signed)', 'gpu': <class 'torch.cuda.LongTensor'>, 'name': 'int64', 'numpy': <class 'numpy.int64'>, 'sparse': <class 'torch.sparse.LongTensor'>, 'types': {torch.int64}}, 'int8': {'cpu': <class 'torch.CharTensor'>, 'desc': '8-bit integer (signed)', 'gpu': <class 'torch.cuda.CharTensor'>, 'name': 'int8', 'numpy': <class 'numpy.int8'>, 'sparse': <class 'torch.sparse.CharTensor'>, 'types': {torch.int8}}, 'uint8': {'cpu': <class 'torch.ByteTensor'>, 'desc': '8-bit integer (unsigned)', 'gpu': <class 'torch.cuda.ByteTensor'>, 'name': 'uint8', 'numpy': <class 'numpy.uint8'>, 'sparse': <class 'torch.sparse.ByteTensor'>, 'types': {torch.uint8}}}

A map of type to metadata.

TYPES = [{'cpu': <class 'torch.FloatTensor'>, 'desc': '32-bit floating point', 'gpu': <class 'torch.cuda.FloatTensor'>, 'name': 'float32', 'numpy': <class 'numpy.float32'>, 'sparse': <class 'torch.sparse.FloatTensor'>, 'types': {torch.float32}}, {'cpu': <class 'torch.DoubleTensor'>, 'desc': '64-bit floating point', 'gpu': <class 'torch.cuda.DoubleTensor'>, 'name': 'float64', 'numpy': <class 'numpy.float64'>, 'sparse': <class 'torch.sparse.DoubleTensor'>, 'types': {torch.float64}}, {'cpu': <class 'torch.HalfTensor'>, 'desc': '16-bit floating point', 'gpu': <class 'torch.cuda.HalfTensor'>, 'name': 'float16', 'numpy': <class 'numpy.float16'>, 'sparse': <class 'torch.sparse.HalfTensor'>, 'types': {torch.float16}}, {'cpu': <class 'torch.ByteTensor'>, 'desc': '8-bit integer (unsigned)', 'gpu': <class 'torch.cuda.ByteTensor'>, 'name': 'uint8', 'numpy': <class 'numpy.uint8'>, 'sparse': <class 'torch.sparse.ByteTensor'>, 'types': {torch.uint8}}, {'cpu': <class 'torch.CharTensor'>, 'desc': '8-bit integer (signed)', 'gpu': <class 'torch.cuda.CharTensor'>, 'name': 'int8', 'numpy': <class 'numpy.int8'>, 'sparse': <class 'torch.sparse.CharTensor'>, 'types': {torch.int8}}, {'cpu': <class 'torch.ShortTensor'>, 'desc': '16-bit integer (signed)', 'gpu': <class 'torch.cuda.ShortTensor'>, 'name': 'int16', 'numpy': <class 'numpy.int16'>, 'sparse': <class 'torch.sparse.ShortTensor'>, 'types': {torch.int16}}, {'cpu': <class 'torch.IntTensor'>, 'desc': '32-bit integer (signed)', 'gpu': <class 'torch.cuda.IntTensor'>, 'name': 'int32', 'numpy': <class 'numpy.int32'>, 'sparse': <class 'torch.sparse.IntTensor'>, 'types': {torch.int32}}, {'cpu': <class 'torch.LongTensor'>, 'desc': '64-bit integer (signed)', 'gpu': <class 'torch.cuda.LongTensor'>, 'name': 'int64', 'numpy': <class 'numpy.int64'>, 'sparse': <class 'torch.sparse.LongTensor'>, 'types': {torch.int64}}, {'cpu': <class 'torch.BoolTensor'>, 'desc': 'Boolean', 'gpu': <class 'torch.cuda.BoolTensor'>, 'name': 'bool', 'numpy': <class 'bool'>, 'types': {torch.bool}}]

A list of dicts containig conversions between types.

classmethod all_types()[source]
Return type:

List[dict]

classmethod float_to_int(torch_type)[source]
Return type:

Type

classmethod get_numpy_type(torch_type)[source]
Return type:

Type

classmethod get_sparse_class(torch_type)[source]
Return type:

Type

classmethod get_tensor_class(torch_type, cpu_type)[source]
Return type:

Type

classmethod int_to_float(torch_type)[source]
Return type:

Type

classmethod is_float(torch_type)[source]
Return type:

bool

classmethod is_int(torch_type)[source]
Return type:

bool

classmethod type_from_string(type_name)[source]
Return type:

dtype

classmethod types()[source]
Return type:

Dict[str, List[dict]]

Module contents

A deep learning framework to make training, validating and testing models with PyTorch easier.