zensols.deeplearn.result package

Submodules

zensols.deeplearn.result.ablation module

Feature ablation graph construction using summary results.

class zensols.deeplearn.result.ablation.FeatureAblationPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, palette=None, data=<factory>, x_axis_name=None, y_axis_name=None, x_column_name=None, y_column_name=None, key_title=None, sample_rate=0, plot_params=<factory>, test_performance=None, test_performance_column='features', test_performance_plot_params=<factory>)

Bases: PointPlot

Plot ablation as feature sets with peformance scores. The lines dataframe columns are:

• epoch: for the X values

• performance: for the Y values

The add_line() method adds a dataset split curves where the name is the split name and line are the loss values.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, palette=None, data=<factory>, x_axis_name=None, y_axis_name=None, x_column_name=None, y_column_name=None, key_title=None, sample_rate=0, plot_params=<factory>, test_performance=None, test_performance_column='features', test_performance_plot_params=<factory>)

test_performance: DataFrame = None

The performance for each feature set on the test set. The dataframe has the following columns:

• test_performance_column: the feature sets

• epoch: see class docs

• performance: see class docs

test_performance_column: str = 'features'

The name of the column in test_performance with the feature sets.

test_performance_plot_params: Dict[str, Any]

Parameters given to the test performance plot.

class zensols.deeplearn.result.ablation.FeatureAblationResults(reporter, average='weighted', metric='f1', append_metric_to_legend=True, conv_flatten=False, epoch_start=0, epoch_end=None, plot_params=<factory>)

Bases: Dictable

Create feature ablation results.

See:

plots.FeatureAblationPlot

__init__(reporter, average='weighted', metric='f1', append_metric_to_legend=True, conv_flatten=False, epoch_start=0, epoch_end=None, plot_params=<factory>)

add_plot(fig)

Add a new ablation plot to fig.

Return type:

Plot

append_metric_to_legend: bool = True

Whether or not to append the metric on the test dataset to the legend.

average: str = 'weighted'

The type of metric average to use for the performance metric.

conv_flatten: bool = False

Whether to max performance metrics to the last highest for epochs after the converged epoch.

epoch_end: int = None

The right boundary used to clamp the epochs reported.

epoch_start: int = 0

The left boundary used to clamp the epochs reported.

metric: str = 'f1'

The name of the metric to use for the performance values.

plot_params: Dict[str, Any]

.Figure.create call.

Type:

Additional args in the “meth

reporter: ModelResultReporter

The reporter used to obtain the results to create the graph.

property summary_dataframe: DataFrame

The multi-run summary used to create the graph. It is modified to be used in the ablation graph.

zensols.deeplearn.result.compare module

Result diff utilities.

class zensols.deeplearn.result.compare.ModelResultComparer(rm, res_id_a, res_id_b)

Bases: Writable

This class performs a diff on two classes and reports the differences.

__init__(rm, res_id_a, res_id_b)

res_id_a: str

The result ID of the first archived result set to diff.

res_id_b: str

The result ID of the second archived result set to diff.

rm: ModelResultManager

The manager used to retrieve the model results.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write the contents of this instance to writer using indention depth.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

zensols.deeplearn.result.confusion module

Tools to generate and visualize confusion matrices.

class zensols.deeplearn.result.confusion.ConfusionMatrix(preds)

Bases: Dictable

Create confusion matrix and optionally heat map representation.

__init__(preds)

add_plot(fig)

Ad a heat map plot of matrix to figure fig.

Return type:

Plot

property matrix: DataFrameDescriber

The confusion matrix of the predictions in preds.

preds: DataFrameDescriber

A dataframe of gold labels (column label) and predictions (column pred).

zensols.deeplearn.result.domain module

Contains contain classes for results generated from training and testing a model.

class zensols.deeplearn.result.domain.ClassificationMetrics(labels, predictions, n_outcomes)

Bases: Metrics

Real valued prediction results for ModelType.CLASSIFICATION result.

__init__(labels, predictions, n_outcomes)

property accuracy: float

Return the accuracy metric (num correct / total).

create_metrics(average)

Create a score metrics with the given average.

Return type:

ScoreMetrics

property macro: ScoreMetrics

Compute macro F1, precision and recall.

property micro: ScoreMetrics

Compute micro F1, precision and recall.

property n_correct: int

The number or correct predictions for the classification.

n_outcomes: int

The number of outcomes given for this metrics set.

property weighted: ScoreMetrics

Compute weighted F1, precision and recall.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.domain.DatasetResult(context)

Bases: ResultsContainer

Contains results for a dataset, such as training, validating and test.

__init__(context)

append(epoch_result)

clone()

Return a clone of the current container. Sub containers (lists) are deep copied in sub classes, but everything is shallow copied.

This is needed to create a temporary container to persist whose end() gets called by the ModelExecutor.

Return type:

ResultsContainer

property contains_results

True if this container has results.

property converged_epoch: EpochResult

Return the last epoch that arrived at the lowest loss.

property convergence: int

Return the Nth epoch index this result set convergened. If used on a EpocResult it is the Nth iteration.

end()

Record the time at which processing started for the metrics populated in this container.

See:

obj:is_ended

property losses: List[float]

Return the loss for each epoch of the run. If used on a EpocResult it is the Nth iteration.

property results: List[EpochResult]

property statistics: Dict[str, Any]

Return the statistics of the data set result.

Returns:

a dictionary with the following:

• n_epochs: the number of epoch results

• n_epoch_converged: the 0 based index for which epoch converged (lowest validation loss before it went back up)

• n_batches: the number of batches on which were trained,

  tested or validated

• ave_data_points: the average number of data pointes on

  which were trained, tested or validated per batch

• n_total_data_points: the number of data pointes on which

  were trained, tested or validated

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_details=False, converged_epoch=True, include_metrics=True, include_all_metrics=False)

Write the results data.

Parameters:

• depth (int) – the number of indentation levels

• writer (TextIOBase) – the data sink

• include_settings – whether or not to include model and network settings in the output

• include_config – whether or not to include the configuration in the output

class zensols.deeplearn.result.domain.EpochResult(context, index, split_type, batch_losses=<factory>, batch_ids=<factory>, n_data_points=<factory>)

Bases: ResultsContainer

Contains results recorded from an epoch of a neural network model. This is during a training/validation or test cycle.

Note that there is a terminology difference between what the model and the result set call outcomes. For the model, outcomes are the mapped/refined results, which are usually the argmax of the softmax of the logits. For results, these are the predictions of the given data to be compared against the gold labels.

__init__(context, index, split_type, batch_losses=<factory>, batch_ids=<factory>, n_data_points=<factory>)

batch_ids: List[int]

The ID of the batch from each iteration of the epoch.

property batch_labels: List[ndarray]

The batch labels given in the shape as output from the model.

batch_losses: List[float]

The losses generated from each iteration of the epoch.

property batch_outputs: List[ndarray]

property batch_predictions: List[ndarray]

The batch predictions given in the shape as output from the model.

clone()

Return a clone of the current container. Sub containers (lists) are deep copied in sub classes, but everything is shallow copied.

This is needed to create a temporary container to persist whose end() gets called by the ModelExecutor.

Return type:

ResultsContainer

end()

Record the time at which processing started for the metrics populated in this container.

See:

obj:is_ended

index: int

The Nth epoch of the run (across training, validation, test).

property losses: List[float]

Return the loss for each epoch of the run. If used on a EpocResult it is the Nth iteration.

n_data_points: List[int]

The number of data points for each batch for the epoch.

split_type: DatasetSplitType

The name of the split type (i.e. train vs test).

update(batch, loss, labels, preds, outputs)

Add another set of statistics, predictions and gold labels to prediction_updates.

Parameters:

• batch (Batch) – the batch on which the stats/data where trained, tested or validated; used to update the loss as a multiplier on its size

• loss (Tensor) – the loss returned by the loss function

• labels (Tensor) – the gold labels, or None if this is a prediction run

• preds (Tensor) – the predictions, or None for scored models (see prediction_updates)

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_metrics=False)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.domain.Metrics(labels, predictions)

Bases: Dictable

A container class that provides results for data stored in a ResultsContainer.

__init__(labels, predictions)

property contains_results: bool

Return True if this container has results.

labels: ndarray

The labels or None if none were provided (i.e. during test/evaluation).

predictions: ndarray

The predictions from the model. This also flattens the predictions in to a 1D array for the purpose of computing metrics.

class zensols.deeplearn.result.domain.ModelResult(config, name, model_settings, net_settings, decoded_attributes, context)

Bases: Dictable

A container class used to capture the training, validation and test results. The data captured is used to report and plot curves.

RUNS: ClassVar[int] = 1

__init__(config, name, model_settings, net_settings, decoded_attributes, context)

clone()

Return type:

ModelResult

config: Configurable

Useful for retrieving hyperparameter settings later after unpersisting from disk.

property contains_results: bool

context: ResultContext

The context of the results.

decoded_attributes: Set[str]

The attributes that were coded and used in this model.

get_intermediate()

Return type:

ModelResult

classmethod get_num_runs()

property last_test: DatasetResult

Return either the test or validation results depending on what is available.

property last_test_name: str

Return the anem of the dataset that exists in the container, and thus, the last to be populated. In order, this is test and then validation.

model_settings: InitVar

The setttings used to configure the model.

name: str

The name of this result set.

net_settings: InitVar

The network settings used by the model for this result set.

property non_empty_dataset_result: Dict[str, DatasetResult]

reset(name)

Clear all results for data set name.

classmethod reset_runs()

Reset the run counter.

property test: DatasetResult

Return the testing run results.

property train: DatasetResult

Return the training run results.

property validation: DatasetResult

Return the validation run results.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_settings=False, include_converged=False, include_config=False, include_all_metrics=False)

Generate a human readable format of the results.

write_result_statistics(split_type, depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

exception zensols.deeplearn.result.domain.ModelResultError

Bases: DeepLearnError

“Thrown when results can not be compiled or computed.

__annotations__ = {}

__module__ = 'zensols.deeplearn.result.domain'

class zensols.deeplearn.result.domain.ModelType(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

The type of model give by the type of its output.

CLASSIFICTION = 2

MULTI_LABEL_CLASSIFICATION = 3

PREDICTION = 1

RANKING = 4

class zensols.deeplearn.result.domain.MultiLabelClassificationMetrics(labels, predictions, n_outcomes, context)

Bases: ClassificationMetrics

Metrics for multi-label classification. Note that precision, recall and F1 micro-average is used because sklearn’s classification_report is used and “it corresponds to accuracy otherwise and would be the same for all metrics”.

__init__(labels, predictions, n_outcomes, context)

context: ResultContext

The context of the results.

create_metrics(average)

Create a score metrics with the given average.

Return type:

ScoreMetrics

property dataframes: Dict[str, DataFrame]

A multi-label classification report as a dataframe. See class documentation regarding reported average (micro).

property multi_labels: Tuple[str, ...]

The labels used in the multi-label classification.

static reshape_labels_predictions(labels, preds, context)

Return type:

Tuple[ndarray, ndarray]

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, report=False)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.domain.MultiLabelScoreMetrics(labels, predictions, average)

Bases: ScoreMetrics

A container class that provides results for multi-label data.

__init__(labels, predictions, average)

exception zensols.deeplearn.result.domain.NoResultError(cls)

Bases: ModelResultError

Convenience used for helping debug the network.

__annotations__ = {}

__init__(cls)

__module__ = 'zensols.deeplearn.result.domain'

class zensols.deeplearn.result.domain.PredictionMetrics(labels, predictions)

Bases: Metrics

Real valued prediction results for ModelType.PREDICTION result.

__init__(labels, predictions)

property correlation: float

Return the correlation metric.

property mean_absolute_error: float

Return the mean absolute error metric.

property r2_score: float

Return the R^2 score metric.

property root_mean_squared_error: float

Return the root mean squared error metric.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.domain.ResultContext(multi_labels=())

Bases: Dictable

Chain-of-reponsibility context for generated result objects.

__init__(multi_labels=())

multi_labels: Tuple[str, ...] = ()

The labels for multi-label classification, otherwise None.

class zensols.deeplearn.result.domain.ResultsContainer(context)

Bases: Dictable

The base class for all metrics containers. It helps in calculating loss, finding labels, predictions and other utility helpers.

Every container has a start and stop time, which demarcates the duration the for which the populated metrics were being calculated.

FLOAT_TYPES = [<class 'numpy.float32'>, <class 'numpy.float64'>, <class 'float'>]

Used to determin the model_type.

__init__(context)

property ave_loss: float

The average loss of this result set.

property classification_metrics: ClassificationMetrics

Return classification based metrics.

clone()

Return a clone of the current container. Sub containers (lists) are deep copied in sub classes, but everything is shallow copied.

This is needed to create a temporary container to persist whose end() gets called by the ModelExecutor.

Return type:

ResultsContainer

property contains_results

True if this container has results.

context: ResultContext

The context of the results.

end()

Record the time at which processing started for the metrics populated in this container.

See:

obj:is_ended

Return type:

datetime

property is_ended: bool

The time at which processing ended for the metrics populated in this container.

See:

meth:end

property is_started: bool

The time at which processing started for the metrics populated in this container.

See:

meth:start

property labels: ndarray

The labels or None if none were provided (i.e. during test/evaluation).

property max_loss: float

The highest loss recorded in this container.

property metrics: Metrics

Return the metrics based on the model_type.

property min_loss: float

The lowest loss recorded in this container.

property model_type: ModelType

The type of the model based on what whether the outcome data is a float or integer.

property multi_label_classification_metrics: MultiLabelClassificationMetrics

Return classification based metrics.

property n_iterations: int

The number of iterations, which is different from the n_outcomes since a single (say training) iteration can produce multiple outcomes (for example sequence classification).

property n_outcomes: int

The number of outcomes.

property prediction_metrics: PredictionMetrics

Return prediction based metrics.

property predictions: ndarray

The predictions from the model. This also flattens the predictions in to a 1D array for the purpose of computing metrics.

Returns:

the flattened predictions

start()

Record the time at which processing started for the metrics populated in this container.

See:

obj:is_started

Return type:

datetime

class zensols.deeplearn.result.domain.ScoreMetrics(labels, predictions, average)

Bases: Metrics

Classification metrics having an f1, precision and recall for a configured weighted, micro or macro average.

WRITE_CONFIDENCE_INTERVALS: ClassVar[bool] = False

__init__(labels, predictions, average)

average: str

The type of average to apply to metrics produced by this class, which is one of macro or micro.

property f1: float

Return the F1 metric as either the micro or macro based on the average attribute.

property f1_ci: Tuple[float, float]

The confidence interval of f1.

property long_f1_name: str

property precision: float

Return the precision metric as either the micro or macro based on the average attribute.

property precision_ci: Tuple[float, float]

The confidence interval of precision.

property recall: float

Return the recall metric as either the micro or macro based on the average attribute.

property recall_ci: Tuple[float, float]

The confidence interval of recall.

property short_f1_name: str

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_confidence_intervals=None)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

zensols.deeplearn.result.graph module

Provides a class to graph the results.

class zensols.deeplearn.result.graph.ModelResultGrapher(name=None, figsize=(15, 5), split_types=None, title=None, save_path=None)

Bases: object

Graphs the an instance of ModelResult. This creates subfigures, one for each of the results given as input to plot().

__init__(name=None, figsize=(15, 5), split_types=None, title=None, save_path=None)

figsize: Tuple[int, int] = (15, 5)

the size of the top level figure (not the panes)

name: str = None

The name that goes in the title of the graph.

plot_loss(containers)

save()

Save the plot to disk.

save_path: Path = None

Where the plot is saved.

show()

Render and display the plot.

split_types: List[DatasetSplitType] = None

The splits to graph (list of size 2); defaults to [DatasetSplitType.train, DatasetSplitType.validation].

title: str = None

The title format used to create each sub pane graph.

zensols.deeplearn.result.hypsig module

Model hypothesis significance testing. This module has a small framework for the hypothesis testing the model results (typically the results from the test dataset). The outcome of disproving the null hypothesis (which is that two classifiers perform the same) means that a classifier has statistically significant better (or worse) performance compared to a second.

class zensols.deeplearn.result.hypsig.AnovaSignificanceTest(data)

Bases: SignificanceTest

One-way ANOVA test.

class zensols.deeplearn.result.hypsig.ChiSquareEvaluation(pvalue, alpha, statistic=None, dof=None, expected=None, contingency_table=None)

Bases: Evaluation

The statistics gathered from scipy.stats.chi2_contingency() and created in ChiSquareCalculator.

__init__(pvalue, alpha, statistic=None, dof=None, expected=None, contingency_table=None)

property adjusted_residuals: DataFrame

The adjusted residuals (see class docs).

property associated: bool

Whether or not the variables are assocated (rejection of the null hypotheis).

contingency_table: DataFrame = None

The contigency table used for the results.

property contribs: DataFrame

The contribution of each cell to the results of the chi-square computation.

dof: int = None

Degrees of freedom

expected: ndarray = None

The expected frequencies, based on the marginal sums of the table. It has the same shape as ChiSquareCalculator.observations.

property pearson_residuals: DataFrame

Pearson residuals, aka standardized residuals.

property raw_residuals: DataFrame

The raw residuals as computed as the difference between the observations and the expected cell values.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

write_associated(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write how the variables relate as a result of the chi-square computation.

See:

write()

class zensols.deeplearn.result.hypsig.ChiSquareSignificanceTest(data)

Bases: SignificanceTest

A ChiSquare test using the 2x2 contigency table as input.

class zensols.deeplearn.result.hypsig.Evaluation(pvalue, alpha, statistic=None)

Bases: DataFrameDictable

An evaluation metric returned by an implementation of SignificanceTest.

__init__(pvalue, alpha, statistic=None)

alpha: float

Independency threshold for asserting the null hypothesis.

property disprove_null_hyp: bool

Whether the evaluation shows the test disproves the null hypothesis.

pvalue: float

The probabily value (p-value).

statistic: float = None

A method specific statistic.

class zensols.deeplearn.result.hypsig.McNemarSignificanceTest(data)

Bases: SignificanceTest

McNemar’s test.

Citation:

Quinn McNemar (1947) Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, June.

exception zensols.deeplearn.result.hypsig.SignificanceError

Bases: APIError

Raised for inconsistent or bad data while testing significance.

__annotations__ = {}

__module__ = 'zensols.deeplearn.result.hypsig'

class zensols.deeplearn.result.hypsig.SignificanceTest(data)

Bases: DataFrameDictable

A statistical significance hypothesis test for models using test set data results.

__init__(data)

data: SignificanceTestData

Contains the data to be used for the significance hypothesis testing.

property evaluation: Evaluation

property name: str

The name of the test.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_contingency=True, include_conclusion=True)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

write_conclusion(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write an intuitive explanation of the results.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.hypsig.SignificanceTestData(a, b, id_col='id', gold_col='label', pred_col='pred', alpha=0.05, null_hypothesis='classifiers have a similar proportion of errors on the test set')

Bases: DataFrameDictable

Metadata needed to create significance tests.

See:

SignificanceTest.

__init__(a, b, id_col='id', gold_col='label', pred_col='pred', alpha=0.05, null_hypothesis='classifiers have a similar proportion of errors on the test set')

a: DataFrame

Test set results from the first model.

alpha: float = 0.05

Used to compare with the p-value to disprove the null hypothesis.

b: DataFrame

Test set results from the second model.

property contingency_table: DataFrame

Return the contingency table using correct columns from correct_table`.

property correct_table: DataFrame

Return a tuple of a dataframe of the correct values in columns a_correct and b_correct.

gold_col: str = 'label'

The column of the gold label/data.

id_col: str = 'id'

The dataset column that contains the unique identifier of the data point. If this is not None, an assertion on the id’s of a and b is performed.

null_hypothesis: str = 'classifiers have a similar proportion of errors on the test set'

A human readable string of the hypothesis.

pred_col: str = 'pred'

The column of the prediction.

class zensols.deeplearn.result.hypsig.SignificanceTestSuite(data, test_names=None)

Bases: DataFrameDictable

A suite of significance tests that use one or more SignificanceTest.

__init__(data, test_names=None)

property available_test_names: Set[str]

All avilable names of tests (see test_names).

data: SignificanceTestData

Contains the data to be used for the significance hypothesis testing.

property describer: DataFrameDescriber

A dataframe describer of all significance evaluations.

test_names: Tuple[str, ...] = None

The test names (SignificanceTest.name) to be in this suite.

property tests: Tuple[SignificanceTest, ...]

The tests used in this suite

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

class zensols.deeplearn.result.hypsig.StudentTTestSignificanceTest(data)

Bases: SignificanceTest

Student’s T-Test, which measure the difference in the mean. This test violates the independence assumption, but it is included as it is still used in papers as a metric.

Citation:

Student (1908) The Probable Error of a Mean. Biometrika, 6(1):1–25.

class zensols.deeplearn.result.hypsig.WilcoxSignificanceTest(data)

Bases: SignificanceTest

Wilcoxon signed-rank test, which is a non-parametric version of Student’s T-Test.

Citation:

Frank Wilcoxon (1945) Individual Comparisons by Ranking Methods. Biometrics Bulletin, 1(6):80–83.

zensols.deeplearn.result.manager module

A class that persists results in various formats.

class zensols.deeplearn.result.manager.ArchivedResult(id, name, txt_path, result_path, model_path, png_path, json_path)

Bases: Dictable

An archived result that provides access to the outcomes the training, validation and optionally test phases of a model execution

See:

ModelResultManager

__init__(id, name, txt_path, result_path, model_path, png_path, json_path)

clear()

Return type:

List[Path]

get_paths(excludes=frozenset({}))

Get all paths in the result as an iterable.

Parameters:

excludes (Set[str]) – the extensions to exclude from the returned paths

Return type:

Iterable[Path]

id: int

The result incremented identitifer.

json_path: Path

The path to the results as a parsable JSON file.

model_path: Path

The path to the directory with the PyTorch model and state files.

property model_result: ModelResult

The results container of the run.

name: str

The result’s unique name, which includes id.

png_path: Path

The path to the training/validation loss results.

result_path: Path

The path to pickled results file.

txt_path: Path

The path results as a text file.

class zensols.deeplearn.result.manager.ModelResultManager(path, pattern='{name}.dat', name=None, model_path=True, save_text=True, save_plot=True, save_json=True, file_pattern='{prefix}-{key}.{ext}', file_regex=re.compile('^(.+)-(.+?)\\\\.([^.]+)$'))

Bases: IncrementKeyDirectoryStash

Saves and loads results from runs (ModelResult) of the ModelExecutor. Keys incrementing integers, one for each save, which usually corresponds to the run of the model executor.

The stash’s path points to where results are persisted with all file format versions.

__init__(path, pattern='{name}.dat', name=None, model_path=True, save_text=True, save_plot=True, save_json=True, file_pattern='{prefix}-{key}.{ext}', file_regex=re.compile('^(.+)-(.+?)\\\\.([^.]+)$'))

create_results_stash(prefix=None)

Return a stash that provides access to previous results (not just the last results). The stash iterates over the model results directory with ArchivedResult values.

Parameters:

prefix (str) – the prefix to use when creating the basename’s stem file portion; if None use a file name version of name

Return type:

Stash

dump(result)

If only one argument is given, it is used as the data and the key name is derived from get_last_key.

file_pattern: str = '{prefix}-{key}.{ext}'

The pattern used to store the model and results files.

file_regex: Pattern = re.compile('^(.+)-(.+?)\\.([^.]+)$')

An regular expression analogue to file_pattern.

get_grapher(figsize=(15, 5), title=None)

Return an instance of a model grapher. This class can plot results of res using matplotlib.

See:

ModelResultGrapher

Return type:

ModelResultGrapher

get_last_id()

Get the last result ID.

Return type:

str

get_next_graph_path()

Return a path to the available graph file to be written.

Return type:

Path

get_next_json_path()

Return a path to the available JSON file to be written.

Return type:

Path

get_next_model_path()

Return a path to the available model file to be written.

Return type:

Path

get_next_text_path()

Return a path to the available text file to be written.

Return type:

Path

model_path: Path = True

The path to where the results are stored.

name: str = None

The name of the manager in the configuration.

parse_file_name(res_id, raise_ex=True)

Return type:

Tuple[str, str, str]

property results_stash: Stash

The canonical results stash for the application configured prefix.

See:

create_results_stash()

save_json: bool = True

If True save the results as a JSON file.

save_json_result(result)

Save the results of the model in JSON format.

save_plot: bool = True

If True save the plot to the file system.

save_plot_result(result)

Plot and save results of the validation and training loss.

save_text: bool = True

If True save the results as a text file.

save_text_result(result)

Save the text results of the model.

static to_file_name(name)

Return a file name string from human readable name.

Return type:

str

zensols.deeplearn.result.plots module

Deep learning plots

class zensols.deeplearn.result.plots.LossPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, palette=None, data=<factory>, x_axis_name=None, y_axis_name=None, x_column_name=None, y_column_name=None, key_title=None, sample_rate=0, plot_params=<factory>)

Bases: PointPlot

A training and validation loss plot. The data dataframe columns are:

• epoch: for the X values

• loss: for the Y values

The add_line() method adds a dataset split curves where the name is the split name and line are the loss values.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, palette=None, data=<factory>, x_axis_name=None, y_axis_name=None, x_column_name=None, y_column_name=None, key_title=None, sample_rate=0, plot_params=<factory>)

zensols.deeplearn.result.pred module

This creates Pandas dataframes containing predictions.

class zensols.deeplearn.result.pred.MultiLabelPredictionsDataFrameFactory(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

Bases: PredictionsDataFrameFactory

Like the super class but create predictions multilabel on sentences and documents.

__init__(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

class zensols.deeplearn.result.pred.PredictionsDataFrameFactory(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

Bases: object

Create a Pandas pandas.DataFrame containing the labels and predictions from the model.ModelExecutor test data set output . The data frame contains the feature IDs, labels, predictions mapped back to their original value from the feature data item.

Currently only classification models are supported.

CORRECT_COL: ClassVar[str] = 'correct'

The correct/incorrect indication column in the generated dataframe in dataframe and metrics_dataframe.

ID_COL: ClassVar[str] = 'id'

The data point ID in the generated dataframe in dataframe and metrics_dataframe.

LABEL_COL: ClassVar[str] = 'label'

The gold label column in the generated dataframe in dataframe and metrics_dataframe.

METRICS_DF_COLUMNS: ClassVar[Tuple[str, ...]] = ('label', 'wF1', 'wP', 'wR', 'mF1', 'mP', 'mR', 'MF1', 'MP', 'MR', 'acc', 'correct', 'count')

metrics_dataframe

Type:

see

METRICS_DF_MACRO_COLUMNS: ClassVar[Tuple[str, ...]] = ('MF1', 'MP', 'MR')

Macro performance metrics columns.

METRICS_DF_MICRO_COLUMNS: ClassVar[Tuple[str, ...]] = ('mF1', 'mP', 'mR')

Micro performance metrics columns.

METRICS_DF_WEIGHTED_COLUMNS: ClassVar[Tuple[str, ...]] = ('wF1', 'wP', 'wR')

Weighed performance metrics columns.

METRIC_AVERAGE_TO_COLUMN: ClassVar[Dict[str, str]] = {'macro': 'M', 'micro': 'm', 'weighted': 'w'}

Name to abbreviation average mapping.

METRIC_COLUMNS: ClassVar[Tuple[str, ...]] = ('wF1', 'wP', 'wR', 'mF1', 'mP', 'mR', 'MF1', 'MP', 'MR', 'acc')

Weighted, micro, macro and accuracy metrics columns.

METRIC_DESCRIPTIONS: ClassVar[Dict[str, str]] = {'MF1': 'macro F1', 'MF1t': 'macro F1 on the test set', 'MF1v': 'macro F1 on the validation set', 'MP': 'macro precision', 'MPt': 'macro precision on the test set', 'MPv': 'macro precision on the validation set', 'MR': 'macro recall', 'MRt': 'macro recall on the test set', 'MRv': 'macro recall on the validation set', 'acc': 'accuracy', 'acct': 'accuracy on the test set', 'accv': 'accuracy on the validation set', 'converged': 'last epoch with the lowest loss', 'correct': 'number of correct classifications', 'count': 'number of data points in the test set', 'features': 'features used in the model', 'label': 'model class', 'mF1': 'micro F1', 'mF1t': 'micro F1 on the test set', 'mF1v': 'micro F1 on the validation set', 'mP': 'micro precision', 'mPt': 'micro precision on the test set', 'mPv': 'micro precision on the validation set', 'mR': 'micro recall', 'mRt': 'micro recall on the test set', 'mRv': 'micro recall on the validation set', 'name': 'model or result set name', 'resid': 'result ID and file name prefix', 'test_end': 'when the testingstarted', 'test_occurs': 'the number of data points used to test', 'test_start': 'when the testing started', 'train_end': 'when the trainingstarted', 'train_occurs': 'the number of data points used to train', 'train_start': 'when the training started', 'validation_occurs': 'the number of data points used to validate', 'wF1': 'weighted F1', 'wF1t': 'weighted F1 on the test set', 'wF1v': 'weighted F1 on the validation set', 'wP': 'weighted precision', 'wPt': 'weighted precision on the test set', 'wPv': 'weighted precision on the validation set', 'wR': 'weighted recall', 'wRt': 'weighted recall on the test set', 'wRv': 'weighted recall on the validation set'}

Dictionary of performance metrics column names to human readable descriptions.

METRIC_NAME_TO_COLUMN: ClassVar[Dict[str, str]] = {'f1': 'f1', 'p': 'precision', 'r': 'recall'}

Name to abbreviation metric mapping.

PREDICTION_COL: ClassVar[str] = 'pred'

The prediction column in the generated dataframe in dataframe and metrics_dataframe.

TEST_METRIC_COLUMNS: ClassVar[Tuple[str, ...]] = ('wF1t', 'wPt', 'wRt', 'mF1t', 'mPt', 'mRt', 'MF1t', 'MPt', 'MRt', 'acct')

Test set performance metric columns.

VALIDATION_METRIC_COLUMNS: ClassVar[Tuple[str, ...]] = ('wF1v', 'wPv', 'wRv', 'mF1v', 'mPv', 'mRv', 'MF1v', 'MPv', 'MRv', 'accv')

Validation set performance metric columns.

__init__(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

batch_limit: int = 9223372036854775807

The max number of batches of results to output.

column_names: List[str] = None

The list of string column names for each data item the list returned from data_point_transform to be added to the results for each label/prediction.

data_point_transform: Callable[[DataPoint], tuple] = None

A function that returns a tuple, each with an element respective of column_names to be added to the results for each label/prediction; if None (the default), str used (see the Iris Jupyter Notebook example)

property dataframe: DataFrame

The predictions and labels as a dataframe. The first columns are generated from data_point_tranform, and the remaining columns are:

• id: the ID of the feature (not batch) data item

• label: the label given by the feature data item

• pred: the prediction

• correct: whether or not the prediction was correct

property dataframe_describer: DataFrameDescriber

Same as dataframe, but return the data with metadata.

epoch_result: EpochResult = None

The epoch containing the results. If none given, take it from the test results.

label_vectorizer_name: str = None

The name of the vectorizer that encodes the labels, which is used to reverse map from integers to their original string nominal values.

property majority_label_metrics_describer: DataFrameDescriber

Compute metrics of the majority label of the test dataset.

metric_metadata: Dict[str, str] = None

Additional metadata when creating instances of DataFrameDescriber in addition to METRIC_DESCRIPTIONS.

property metrics_dataframe_describer: DataFrameDescriber

Get a dataframe describer of metrics (see metrics_dataframe).

classmethod metrics_to_describer(metrics)

Create a dataframe describer using a metrics instance with standard column naming and metadata. Use METRIC_AVERAGE_TO_COLUMN and METRIC_NAME_TO_COLUMN to create a single dataframe row of performance metrics.

Return type:

DataFrameDescriber

metrics_to_series(lab, mets)

Create a single row dataframe from classification metrics.

Return type:

Series

name: str = None

The name of the results or the result ID (res_id). If not provided, it is taken from ModelResult.

result: ModelResult

The epoch containing the results.

source: Path

The source file from where the results were unpickled.

stash: BatchStash

The batch stash used to generate the results from the ModelExecutor. This is used to get the vectorizer to reverse map the labels.

class zensols.deeplearn.result.pred.SequencePredictionsDataFrameFactory(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

Bases: PredictionsDataFrameFactory

Like the super class but create predictions for sequence based models.

See:

SequenceNetworkModule

__init__(source, result, stash, column_names=None, data_point_transform=None, batch_limit=9223372036854775807, epoch_result=None, label_vectorizer_name=None, metric_metadata=None, name=None)

zensols.deeplearn.result.report module

A utility class to summarize all results in a directory.

class zensols.deeplearn.result.report.ModelResultReporter(result_manager, include_validation=True)

Bases: object

Summarize all results in a directory from the output of model execution from ModelExectuor.

The class iterates through the pickled binary output files from the run and summarizes in a Pandas dataframe, which is handy for reporting in papers.

__init__(result_manager, include_validation=True)

property cross_validate_describer: DataDescriber

A data describer with the results of a cross-validation.

The describer returned includes the metrics for each fold and summary statitics for all folds. The statistics DataFrameDescriber (describer with name cross-validation-stats) contain the following 95% mean confidence interval calculations given in the stat row:

• t-ci: use the t-scores from a t-distribution (assumes a normal distribution across scores)

property dataframe: DataFrame

Return the summarized results (see class docs).

Returns:

the Pandas dataframe of the results

property dataframe_describer: DataFrameDescriber

Get a dataframe describer of metrics (see dataframe).

dump(path)

Create the summarized results and write them to the file system.

Return type:

DataFrame

include_validation: bool = True

Whether or not to include validation performance metrics.

result_manager: ModelResultManager

Contains the results to report on–and specifically the path to directory where the results were persisted.

Module contents

This package provides container classes used for the model execution results.

see:

zensols.deeplearn.result.ModelResult