zensols.datdesc package

Submodules

zensols.datdesc.app module

Generate LaTeX tables in a .sty file from CSV files. The paths to the CSV files to create tables from and their metadata is given as a YAML configuration file. Paraemters are both files or both directories. When using directories, only files that match *-table.yml are considered.

class zensols.datdesc.app.Application(config_factory, renderable_factory)

Bases: object

Generate LaTeX tables files from CSV files and hyperparameter .sty files.

__init__(config_factory, renderable_factory)

config_factory: ConfigFactory

Creates table and figure factories.

generate_figures(input_path, output_path, output_image_format=None, output_sty=None)

Generate figures.

Parameters:

• input_path (Path) – YAML definitions or JSON serialized file

• output_path (Path) – output file or directory

• output_image_format (str) – the output format (defaults to svg)

• output_sty (str) – name of .sty output file, - for source names, default to no output

generate_tables(input_path, output_path, output_format=OutputFormat.table)

Create LaTeX tables.

Parameters:

• input_path (Path) – YAML definitions or JSON serialized file

• output_path (Path) – output file or directory

list_figures(input_path)

List figures.

Parameters:

• input_path (Path) – YAML definitions or JSON serialized file

• output_path – output file or directory

• output_image_format – the output format (defaults to svg)

renderable_factory: RenderableFactory

Creates instances of Renderable from file paths.

show_table(name=None)

Print a list of example LaTeX tables.

Parameters:

name (str) – the name of the example table or a listing of tables if omitted

write_excel(input_path, output_file=None, output_latex_format=False)

Create an Excel file from table data.

Parameters:

• input_path (Path) – YAML definitions or JSON serialized file

• output_file (Path) – the output file, which defaults to the input prefix with the approproate extension

• output_latex_format (bool) – whether to output with LaTeX commands

zensols.datdesc.cli module

Command line entry point to the application.

class zensols.datdesc.cli.ApplicationFactory(*args, **kwargs)

Bases: ApplicationFactory

__init__(*args, **kwargs)

classmethod get_renderable_factory(**kwargs)

Return a factory that creates instances of Renderable from file paths.

Return type:

RenderableFactory

zensols.datdesc.desc module

Metadata container classes.

class zensols.datdesc.desc.DataDescriber(describers, name='default', mangle_sheet_name=False)

Bases: PersistableContainer, Dictable

Container class for DataFrameDescriber instances. It also saves their instances as CSV data files and YAML configuration files.

DEFAULT_CSV_DIR: ClassVar[Path] = PosixPath('config/csv')

DEFAULT_EXCEL_DIR: ClassVar[Path] = PosixPath('results')

DEFAULT_YAML_DIR: ClassVar[Path] = PosixPath('results/config')

SHEET_NAME_MAXLEN: ClassVar[int] = 31

Maximum allowed characters in an Excel spreadsheet’s name.

__init__(describers, name='default', mangle_sheet_name=False)

add_summary()

Add a new metadata like DataFrameDescriber as a first entry in describers that describes what data this instance currently has.

Return type:

DataFrameDescriber

Returns:

the added metadata DataFrameDescriber instance

derive(**kwargs)

Create a new instance based on this instance and replace any non-None kwargs.

Parameters:

kwargs – the key word arguments to replace any field data from this instance

Return type:

DataDescriber

Returns:

a new instance with replaced data, or a clone if called with no key word arguments

derive_with_index_meta(index_format=None)

Applies DataFrameDescriber.derive_with_index_meta() to each element of describers.

Return type:

DataFrameDescriber

describers: tuple[DataFrameDescriber, ...]

The contained dataframe and metadata.

property describers_by_name: dict[str, DataFrameDescriber]

Data frame describers keyed by the describer name.

format_tables()

See DataFrameDescriber.format_table().

classmethod from_describer(dfd)

Create a singleton describer. The name is taken from the dfd DataFrameDescriber.name.

Return type:

DataDescriber

classmethod from_excel(path, *, dataset_name_fmt='{path.stem}', read_excel_kwargs=None, header_row=None, data_start_row=None, include_sheets=None, exclude_sheets=None, table_desc_fmt="From {path.name} (sheet '{sheet}')", head=None, mangle_sheet_name=False, mangle_file_names=False)

Read all sheets from an Excel workbook using pandas, and populate:

• DataFrameDescriber.meta from COMMENTS (a.k.a. Excel “notes”) on header cells

• DataFrameDescriber.index_meta from COMMENTS on index-column cells (one per row)

Conventions (workbook layout assumptions):

• Header row contains column names. Put each column description as a cell comment on the corresponding header cell.

• If you set pandas index_col=…, put row descriptions as cell

  comments on the cells of that index column for each data row.

Note: openpyxl supports the legacy Excel “notes” comment objects, but not newer threaded comments. Use Notes if you want this to work.

Parameters:

• read_excel_kwargs (Mapping[str, Any] | None) – pandas.read_excel() keyword params

• header_row (int | None) – start row of header; if omitted inferred from pandas If omitted, inferred from pandas header (default 0 => row 1)

• data_start_row (int | None) – start row of header; if omitted inferred from pandas If omitted, inferred from pandas header (default 0 => row 1)

Return type:

DataDescriber

classmethod from_json(reader)

Unserialize a JSON stream into a data descriptor.

Parameters:

reader (TextIOWrapper) – the file / data stream

Return type:

DataDescriber

classmethod from_json_file(path)

Like from_yaml_file() but a JSON file written with meth:save_json.

Return type:

DataDescriber

classmethod from_tables(tables, name=None)

Create a data descriptor from a Table.

Return type:

DataDescriber

classmethod from_yaml_file(path)

Create a data descriptor from a previously written YAML/CSV files using save().

See:

save()

See:

DataFrameDescriber.from_table()

Return type:

DataDescriber

items()

Return type:

Iterable[tuple[str, DataFrameDescriber]]

keys()

Return type:

Sequence[str]

mangle_sheet_name: bool = False

Whether to normalize the Excel sheet names when xlsxwriter.exceptions.InvalidWorksheetName is raised.

name: str = 'default'

The name of the dataset.

save(csv_dir=None, yaml_dir=None, excel_path=None)

Save both the CSV and YAML configuration file.

Parameters:

• csv_dir (Path) – the directory of where to save the data

• yaml_dir (Path) – the directory of where to save the YAML files

• excel_path (bool | Path) – where to write the Excel file if not None or False, otherwise create in a new results directory with name

See:

save_csv()

See:

save_yaml()

Return type:

list[Path]

save_csv(csv_dir)

Save all provided dataframe describers to an CSV files.

Parameters:

csv_dir (Path) – the directory of where to save the data

Return type:

list[Path]

save_excel(output_file=None)

Save all provided dataframe describers to an Excel file.

Parameters:

output_file (Path) – the Excel file to write; .xlsx will be postpend if no extension exists

Return type:

Path

save_json(output_path=PosixPath('.'))

Serialize the this instance as a JSON file. Most or all of the state is kept in tact and unserialized / restored with from_json().

Return type:

Path

save_yaml(csv_dir, yaml_dir)

Save all provided dataframe describers YAML files used by the datdesc command.

Parameters:

• csv_dir (Path) – the directory of where to save the data

• yaml_dir (Path) – the directory where the YAML config files are saved

Return type:

list[Path]

to_json(writer)

Serialize the object to JSON that can be re-instantiated using from_json().

Parameters:

• writer (TextIOBase) – the data sink

• kwargs – the key word arguments to give to json.dump()

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, df_params=None)

Parameters:

df_params (dict[str, Any]) – the formatting pandas options, which defaults to max_colwidth=80

class zensols.datdesc.desc.DataFrameDescriber(name, df, desc, head=None, meta_path=None, meta=None, table_kwargs=<factory>, index_meta=None, mangle_file_names=False)

Bases: PersistableContainer, Dictable

A class that contains a Pandas dataframe, a description of the data, and descriptions of all the columns in that dataframe.

property T: DataFrameDescriber

See transpose().

__init__(name, df, desc, head=None, meta_path=None, meta=None, table_kwargs=<factory>, index_meta=None, mangle_file_names=False)

classmethod class_to_name(name, replace_char='-')

Return type:

str

property column_descriptions: dict[str, str]

A dictionary of name to Descriptions of the column metadata created from meta. Any missing column metadata will result in None dictionary values.

create_table(**kwargs)

Create a table from the metadata using:

• csv_path as Table.path

• df as Table.dataframe

• desc as Table.caption

• asdict() as Table.column_renames

Parameters:

kwargs – key word arguments that override the default parameterized data passed to Table

Return type:

Table

property csv_path: Path

The CVS file that contains the data this instance describes.

derive(*, name=None, df=None, desc=None, meta=None, index_meta=None)

Create a new instance based on this instance and replace any non-None kwargs.

If meta is provided, it is merged with the metadata of this instance. However, any metadata provided must match in both column names and descriptions.

Parameters:

• name (str) – name

• df (DataFrame) – df

• desc (str) – desc

• meta (DataFrame | tuple[tuple[str, str], ...]) – meta

Raises:

DataDescriptionError – if multiple metadata columns with differing descriptions are found

Return type:

DataFrameDescriber

derive_with_index_meta(index_format=None)

Like derive(), but the dataframe is generated with df_with_index_meta() using index_format as a parameter.

Parameters:

index_format (str) – see df_with_index_meta()

Return type:

DataFrameDescriber

desc: str

The description of the data frame.

df: pd.DataFrame

The dataframe to describe.

df_with_index_meta(index_format=None)

Create a dataframe with the first column containing index metadata. This uses index_meta to create the column values.

Parameters:

index_format (str) – the new index column format using index and value, which defaults to {index}

Return type:

DataFrame

Returns:

the dataframe with a new first column of the index metadata, or df if index_meta is None

format_table()

Replace (in place) dataframe df with the formatted table obtained with Table.formatted_dataframe. The Table is created by with create_table().

classmethod from_columns(source, name=None, desc=None)

Create a new instance by transposing a column data into a new dataframe describer. If source is a dataframe, it that has the following columns:

• column: the column names of the resulting describer

• meta: the description that makes up the meta

• data: Sequence’s of the data

Otherwise, each element of the sequence is a row of column, meta descriptions, and data sequences.

Parameters:

• source (DataFrame | Sequence[Sequence[Any]]) – the data as columns

• name (str) – used for name

• desc (str) – used for desc

Return type:

DataFrameDescriber

classmethod from_dataclasses(data, meta=None, member_names=None)

Create a new instance from dataclasses.

Parameters:

• data (Sequence[Any]) – at least one @dataclass object from dataclasses

• meta (DataclassMetadata) – dataclass metadata, or if None created from data[0]

• member_names (Sequence[str]) – dataclass members to copy, or all if not given

Return type:

DataFrameDescriber

Returns:

a new instance containing data

classmethod from_table(table)

Create a frame descriptor from a Table.

Return type:

DataFrameDescriber

get_table_name(form)

The table derived from name.

Parameters:

form (str) – specifies the format: file means file-friendly, camel is for reverse camel notation

Return type:

str

head: str = None

A short summary of the table and used in Table.head.

index_meta: dict[Any, str] = None

The index metadata, which maps index values to descriptions of the respective row.

mangle_file_names: bool = False

Whether to normalize output file names.

merge(objs, on=None)

Merge using the same semantics as pandas.DataFrame.merge(). Only the column metadata and columns are merged. The remaining data (i.e. name, desc, index_meta) are taken from this instance.

Parameters:

• objs (Iterable[DataFrameDescriber]) – dataframes to merge to this

• on (str | list[str] | None) – column used to merge; if None, the indexes are used

Return type:

DataFrameDescriber

Returns:

a new describer with objs merged

property meta: DataFrame

The column metadata for dataframe, which needs columns name and description. If this is not provided, it is read from file meta_path. If this is set to a tuple of tuples, a dataframe is generated from the form:

((<column name 1>, <column description 1>),
 (<column name 2>, <column description 2>) ...

If both this and meta_path are not provided, the following is used:

(('description', 'Description'),
 ('value', 'Value')))

meta_path: Path | None = None

A path to use to create meta metadata.

See:

meta

name: str

The description of the data this describer holds.

save_csv(output_dir=PosixPath('.'))

Save as a CSV file using csv_path.

Return type:

Path

save_excel(output_path=PosixPath('.'), is_dir=True)

Save as an Excel file. To add column labels use instances of this object with DataDescriber.save_excel().

Parameters:

• output_path (Path) – where to write the Excel file

• is_dir (bool) – whether output_path should be treated as a file or directory; if a directory use csv_path as the file name

See:

DataDescriber.save_excel()

Return type:

Path

table_kwargs: dict[str, Any]

Additional key word arguments given when creating a table in create_table().

transpose(row_names=((0, 'value', 'Value'),), name_column='name', name_description='Name', index_column='description')

Transpose all data in this descriptor by transposing df and swapping meta with index_meta as a new instance.

Parameters:

• row_names (tuple[int, str, str]) – a tuple of (row index in df, the column in the new df and the metadata description of that column in the new df; the default takes only the first row

• name_column (str) – the column name this instance’s df

• description_column – the column description this instance’s df

• index_column (str) – the name of the new index in the returned instance

Return type:

DataFrameDescriber

Returns:

a new derived instance of the transposed data

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, df_params=None)

Parameters:

df_params (dict[str, Any]) – the formatting pandas options, which defaults to max_colwidth=80

write_pretty(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_metadata=False, title_format='{name} ({desc})', **tabulate_params)

Like write(), but generate a visually appealing table and optionally column metadata.

class zensols.datdesc.desc.RenderableDataFrameDescriber(path)

Bases: Renderable

Reads instances serialized with DataDescriber.to_json() and writes the CVS, YAML and Excel files.

__init__(path)

get_artifacts()

Return artifacts created found on path.

Return type:

Iterable[Any]

get_data_describer()

Return the data describer serialized in path.

Return type:

DataDescriber

render(output)

Write the rendered output.

Parameters:

output (Path) – either a file or directory (depending on the subclass)

Return type:

Path

zensols.datdesc.dfstash module

A stash implementation that uses a Pandas dataframe and stored as a CSV file.

class zensols.datdesc.dfstash.DataFrameStash(path, dataframe=None, key_column='key', columns=('value',), sort_columns=(), mkdirs=True, auto_commit=True, single_column_index=0)

Bases: CloseableStash, Dictable

A backing stash that persists to a CSV file via a Pandas dataframe. All modification go through the pandas.DataFrame and then saved with commit() or close().

__init__(path, dataframe=None, key_column='key', columns=('value',), sort_columns=(), mkdirs=True, auto_commit=True, single_column_index=0)

auto_commit: bool = True

Whether to save to the file system after any modification.

clear()

Delete all data from the from the stash.

Important: Exercise caution with this method, of course.

close()

Close all resources created by the stash.

columns: tuple[str, ...] = ('value',)

The columns to create in the spreadsheet. These must be consistent when the data is restored.

commit()

Commit changes to the file system.

property dataframe: DataFrame

The dataframe to proxy in memory. This is settable on instantiation but read-only afterward. If this is not set an empty dataframe is created with the metadata in this class.

delete(name=None)

Delete the resource for data pointed to by name or the entire resource if name is not given.

dump(name, inst)

Persist data value inst with key name.

exists(name)

Return True if data with key name exists.

Implementation note: This Stash.exists() method is very inefficient and should be overriden.

Return type:

bool

get(name, default=None)

Load an object or a default if key name doesn’t exist. Semantically, this method tries not to re-create the data if it already exists. This means that if a stash has built-in caching mechanisms, this method uses it.

See:

load()

Return type:

Any | tuple[Any, ...]

key_column: str = 'key'

The spreadsheet column name used to store stash keys.

keys()

Return an iterable of keys in the collection.

Return type:

Iterable[str]

load(name)

Load a data value from the pickled data with key name. Semantically, this method loads the using the stash’s implementation. For example DirectoryStash loads the data from a file if it exists, but factory type stashes will always re-generate the data.

See:

get()

Return type:

Any | tuple[Any, ...]

mkdirs: bool = True

Whether to recusively create the directory where path is stored if it does not already exist.

path: Path

The path of the file from which to read and write.

single_column_index: int | None = 0

If this is set, then a single type is assumed for loads and restores. Otherwise, if set to None, multiple columns are saved and retrieved.

sort_columns: tuple[str, ...] = ()

If none-None, sort rows by the given column.

values()

Return the values in the hash.

Return type:

Iterable[Any | tuple[Any, ...]]

class zensols.datdesc.dfstash.DataclassStash(path, dataframe=None, key_column='key', columns=('value',), sort_columns=(), mkdirs=True, auto_commit=True, single_column_index=0, data_name=None, desc=None, metadata=None)

Bases: DataFrameStash

Map CSV rows in DataFrameStash stashes to Python dataclasses.

__init__(path, dataframe=None, key_column='key', columns=('value',), sort_columns=(), mkdirs=True, auto_commit=True, single_column_index=0, data_name=None, desc=None, metadata=None)

data_name: str = None

The name used for describer. If not provided, taken from the class name.

desc: str = None

The description used in describer. If not provided, it is taken from the class docstring.

get_describer()

Get a data describer representation of the current data.

Return type:

DataFrameDescriber

metadata: DataclassMetadata = None

Get the metadata of dataclass data_class.

zensols.datdesc.domain module

Application domain.

exception zensols.datdesc.domain.DataDescriptionError

Bases: APIError

Thrown for any application level error.

__firstlineno__ = 8

__module__ = 'zensols.datdesc.domain'

__static_attributes__ = ()

exception zensols.datdesc.domain.FigureError(reason, figure=None, definition_file=None)

Bases: DataDescriptionError

Thrown for any application level error related to creating figures.

__annotations__ = {}

__firstlineno__ = 26

__init__(reason, figure=None, definition_file=None)

__module__ = 'zensols.datdesc.domain'

__static_attributes__ = ('figure',)

exception zensols.datdesc.domain.LatexTableError(reason, table=None)

Bases: DataDescriptionError

Thrown for any application level error related to creating tables.

__annotations__ = {}

__firstlineno__ = 15

__init__(reason, table=None)

__module__ = 'zensols.datdesc.domain'

__static_attributes__ = ('table',)

class zensols.datdesc.domain.OutputFormat(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: Enum

The output format for tables (mostly hyperparameter data).

json = 3

short = 1

sphinx = 5

table = 6

verbose = 2

yaml = 4

zensols.datdesc.figure module

A simple object oriented plotting API.

class zensols.datdesc.figure.Figure(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, config_factory=None, title_font_size=0, height=5, width=5, padding=5.0, metadata=<factory>, plots=(), image_dir=PosixPath('.'), image_format='svg', image_file_norm=True, seaborn=<factory>, subplot_params=<factory>, savefig_params=<factory>)

Bases: RenderableLatexArtifact

Manage a Matplotlib figure and its collection of plots.

A figure is the top-level rendering object for one or more Plot instances. Each plot is assigned a row and column and rendered on the corresponding matplotlib.axes.Axes created by matplotlib.pyplot.subplots().

The figure controls the overall image dimensions, subplot layout, optional Seaborn styling, image metadata, and parameters used when saving the resulting image. Plots can be supplied at construction time, added with add_plot(), or created from configuration with FigureFactory.

Plot definitions loaded by FigureFactory support several code hooks. These run at two different stages of the figure lifecycle.

code_pre

Executed by FigureFactory._parse_plot() while parsing the plot definition, before the Plot instance is created. The YAML mapping is first wrapped in a Settings instance and made available as plot. The code can therefore modify constructor parameters using attribute notation. After the hook executes, the settings are converted back to a dictionary and passed to FigureFactory.create().

code_post

Executed by FigureFactory._parse_plot() immediately after the Plot instance has been created. Here plot refers to the newly instantiated Plot, so the hook can modify the actual plot object.

code_pre_render

Stored on the Plot instance and executed by _render() immediately before Plot.render(). At this point the Figure, Matplotlib figure, and axes have already been created. The execution context provides plot, fig, and axes.

code_post_render

Stored on the Plot instance and executed by _render() immediately after Plot.render(), with the same rendering context. This is useful for modifying Matplotlib objects after the plot has populated its axes.

Thus, code_pre and code_post are configuration/instantiation hooks handled by FigureFactory, whereas code_pre_render and code_post_render are rendering hooks invoked later by this class.

For example:

example_fig:
  width: 6
  height: 4
  plots:
    - type: bar
      data: 'dataframe: data.csv'
      code_pre: |
        plot.y_axis_label = 'Score'
      code_post: |
        plot.title = 'Results'
      code_post_render: |
        axes.grid(False)

Rendering is lazy: the underlying matplotlib.figure.Figure and axes are created when first needed and reused until the figure is reset. Calling save() renders all managed plots and writes the image to path.

In addition to image generation, this class is a RenderableLatexArtifact, allowing the generated image to be referenced by the LaTeX rendering infrastructure.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, config_factory=None, title_font_size=0, height=5, width=5, padding=5.0, metadata=<factory>, plots=(), image_dir=PosixPath('.'), image_format='svg', image_file_norm=True, seaborn=<factory>, subplot_params=<factory>, savefig_params=<factory>)

add_plot(plot)

Add to the collection of managed plots. This is needed for the plot to work if not created from this manager instance.

Parameters:

plot (Plot) – the plot to be managed

clear()

Remove all plots and reset the matplotlib module.

config_factory: ConfigFactory = None

The configuration factory used to create plots.

create(name, **kwargs)

Create a plot using the arguments of Plot.

Parameters:

• name (str | type[Plot]) – the configuration section name of the plot

• kwargs – the initializer keyword arguments when creating the plot

Return type:

Plot

deallocate()

Deallocate all resources for this instance.

height: int = 5

The height in inches of the entire figure.

image_dir: Path = PosixPath('.')

The default image save directory.

image_file_norm: bool = True

Whether to normalize the image output file name.

image_format: str = 'svg'

The image format to use when saving plots.

metadata: dict[str, str]

Metadata added to the image when saved.

padding: float = 5.0

Tight layout padding.

plots: tuple[Plot, ...] = ()

The plots managed by this object instance. Use add_plot() to add new plots.

save()

Save the figure of subplot(s) to at location path.

Param:

if provided, overrides the save location path

Return type:

Path

Returns:

the value of path

savefig_params: dict[str, Any]

Additional parameters given to matplotlib.figure.Figure.savefig().

seaborn: dict[str, Any]

Seaborn (seaborn) rendering configuration. It has the following optional keys:

• style: parameters used with sns.set_style()

• context: parameters used with sns.set_context()

show()

Render and display the plot.

subplot_params: dict[str, Any]

Additional parameters given to matplotlib.pyplot.subplots().

title_font_size: int = 0

The font size name. A size of 0 means do not render the title. Typically a font size of 16 is appropriate.

width: int = 5

The width in inches of the entire figure.

class zensols.datdesc.figure.FigureFactory(config_factory, plot_section_regex)

Bases: Dictable

Create instances of :.Figure using create() or from configuration files with from_file(). See the usage documentation for information about the configuration files used by from_file().

__init__(config_factory, plot_section_regex)

config_factory: ConfigFactory

The configuration factory used to create Table instances.

create(type, **params)

Create a plot from the application configuration.

Parameters:

• type (str) – the name used to find the plot by section

• params (dict[str, Any]) – the keyword arguments used to create the plot

Return type:

Plot

Returns:

a new instance of the plot defined by the template

See:

get_plot_names()

classmethod default_instance()

Get the singleton instance.

Return type:

FigureFactory

from_dict(figure_config)

Return figures parsed from nested builtins.dict (see class documentation).

Parameters:

figure_config (dict[str, Any]) – the same structure as what comes in a YAML file

Return type:

Iterable[Figure]

from_file(figure_path)

Like from_dict() but read from a YAML file.

Parameters:

figure_path (Path) – the file containing the figure configurations

Return type:

Iterable[Figure]

get_plot_names()

Return names of plots used in :meth:create.

Return type:

Iterable[str]

plot_section_regex: Pattern

A regular expression that matches plot entries.

classmethod reset_default_instance()

Force default_instance() to re-instantiate a new instance on a subsequent call.

class zensols.datdesc.figure.Plot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None)

Bases: Dictable

An abstract base class for plots. The subclass overrides plot() to generate the plot. Then the client can use save() or render() it. The plot is created as a subplot providing attributes for space to be taken in rows, columns, height and width.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None)

code_post_render: str = None

If provided, execute code after the plot has been created. The code is executed with variable ax set the Axes, fig set to matplotlib.figure.Figure and plot set to this instance.

code_pre_render: str = None

If provided, execute code before the plot has been created. The code is executed with variable ax set the Axes, fig set to matplotlib.figure.Figure and plot set to this instance.

column: int = 0

The column grid position of the plot.

legend_params: dict[str, Any]

Parameters given to legend().

post_hooks: list[Callable]

Methods to invoke after rendering.

render(axes)

row: int = 0

The row grid position of the plot.

title: str = None

The title to render in the plot.

class zensols.datdesc.figure.RenderableFigure(path, factory, image_format=None, output_sty=False)

Bases: Renderable

A renderable for figures. The output is either a directory where all figures will be written, or a file.

__init__(path, factory, image_format=None, output_sty=False)

factory: FigureFactory

.Figure using create() or from configuration files with from_file().

Type:

Create instances of

get_artifacts()

Return artifacts created found on path.

Return type:

Iterable[Any]

get_figures()

Get figures configured in file path.

Return type:

Iterable[Figure]

image_format: str = None

The default image file output format.

output_sty: bool = False

whether to generate command .sty files.

render(output)

Write the rendered output.

Parameters:

output (Path) – either a file or directory (depending on the subclass)

Return type:

tuple[Path, ...]

static write_sty(output, figures, package_name=None)

class zensols.datdesc.figure.RenderableFigureLatexPackage(artifacts, name, description='{date} Artifacts', sty_file=None)

Bases: RenderableLatexPackage

A figure package that makes figure paths relative to sty_file.

__init__(artifacts, name, description='{date} Artifacts', sty_file=None)

sty_file: Path = None

The sty file to write; used to make fig paths relative.

zensols.datdesc.hyperparam module

Hyperparameter metadata: access and documentation. This package was designed for the following purposes:

• Provide a basic scaffolding to update model hyperparameters such as hyperopt.

• Generate LaTeX tables of the hyperparamers and their descriptions for academic papers.

The object instance graph hierarchy is:

HyperparamSet
|
+-- HyperparamModel
    |
    +-- Hyperparam

Access to the hyperparameters is done by calling the set or model levels with a dotted path notation string. For example, svm.C first navigates to model svm, then to the hyperparameter named C.

class zensols.datdesc.hyperparam.Hyperparam(name, type, doc, choices=None, value=None, interval=None)

Bases: Dictable

A hyperparameter’s metadata, documentation and value. The value is accessed (retrieval and setting) at runtime. Do not use this class explicitly. Instead use HyperparamModel.

The index access only applies when type is list or dict. Otherwise, the value member has the value of the hyperparameter.

CLASS_MAP: ClassVar[dict[str, type]] = {'bool': <class 'bool'>, 'choice': <class 'str'>, 'dict': <class 'dict'>, 'float': <class 'float'>, 'int': <class 'int'>, 'list': <class 'list'>, 'str': <class 'str'>}

A mapping for values set in type to their Python class equivalents.

VALID_TYPES: ClassVar[str] = frozenset({'bool', 'choice', 'dict', 'float', 'int', 'list', 'str'})

Valid settings for type.

__init__(name, type, doc, choices=None, value=None, interval=None)

choices: tuple[str, ...] = None

When type is choice, the value strings used in value.

property cls: type

The Python equivalent class of type.

doc: str

The human readable documentation for the hyperparameter. This is used in documentation generation tasks.

get_type_str(short=True)

Return type:

str

interval: tuple[float | int, int] = None

Valid intervals for value as an inclusive interval.

property interval_str: str

name: str

The name of the hyperparameter (i.e. C or learning_rate).

type: str

The type of value (i.e. float, or int).

property value: str | float | int | bool | list | dict | None

The value of the hyperparamer used in the application.

write_sphinx(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

class zensols.datdesc.hyperparam.HyperparamContainer

Bases: Dictable

A container class for Hyperparam instances.

__init__()

abstract flatten(deep=False)

Return a flattened directory with the dotted path notation (see module docs).

Parameters:

deep (bool) – if True recurse in to dict and list hyperparameter values

Return type:

dict[str, Any]

update(params)

Update parameter values.

Parameters:

params (dict[str, Any] | HyperparamContainer) – a dict of dotted path notation keys

abstract write_sphinx(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write Sphinx autodoc used in a class as dataclasses.dataclass field.

exception zensols.datdesc.hyperparam.HyperparamError

Bases: DataDescriptionError

Raised for any error related hyperparameter access.

__annotations__ = {}

__firstlineno__ = 53

__module__ = 'zensols.datdesc.hyperparam'

__static_attributes__ = ()

class zensols.datdesc.hyperparam.HyperparamModel(name, doc, desc=None, params=<factory>, table=None)

Bases: HyperparamContainer

The model level class that contains the parameters. This class represents a machine learning model such as a SVM with hyperparameters such as C and maximum iterations.

__init__(name, doc, desc=None, params=<factory>, table=None)

clone()

Make a copy of this instance.

Return type:

HyperparamModel

create_dataframe_describer()

Return an object with metadata fully describing the hyperparameters of this model.

Return type:

DataFrameDescriber

desc: str = None

name is not sufficient. Since name has naming constraints, this can be used as in place during documentation generation.

Type:

The description the model used in the documentation when obj

doc: str

The human readable documentation for the model.

flatten(deep=False)

Return a flattened directory with the dotted path notation (see module docs).

Parameters:

deep (bool) – if True recurse in to dict and list hyperparameter values

Return type:

dict[str, Any]

get(name)

Return type:

HyperparamModel

property metadata_dataframe: DataFrame

A dataframe describing the values_dataframe.

name: str

The name of the model (i.e. svm). This name can have only alpha-numeric and underscore charaters.

params: dict[str, Hyperparam]

The hyperparameters keyed by their names.

table: dict[str, Any | None] = None

Overriding data used when creating a Table from DataFrameDescriber.create_table().

property values_dataframe: DataFrame

A dataframe with parameter data. This includes the name, type, value and documentation.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_doc=False)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

write_sphinx(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write Sphinx autodoc used in a class as dataclasses.dataclass field.

class zensols.datdesc.hyperparam.HyperparamSet(models=<factory>, name=None)

Bases: HyperparamContainer

The top level in the object graph hierarchy (see module docs). This contains a set of models and typically where calls by packages such as hyperopt are used to update the hyperparameters of the model(s).

__init__(models=<factory>, name=None)

create_describer(meta_path=None)

Return an object with metadata fully describing the hyperparameters of this model.

Parameters:

meta_path (Path) – if provided, set the path on the returned instance

Return type:

DataDescriber

flatten(deep=False)

Return a flattened directory with the dotted path notation (see module docs).

Parameters:

deep (bool) – if True recurse in to dict and list hyperparameter values

Return type:

dict[str, Any]

get(name)

Return type:

HyperparamModel

models: dict[str, HyperparamModel]

The models containing hyperparameters for this set.

name: str | None = None

The name fo the hyperparameter set.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, include_doc=False)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

write_sphinx(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write Sphinx autodoc used in a class as dataclasses.dataclass field.

class zensols.datdesc.hyperparam.HyperparamSetLoader(data, config=None, updates=())

Bases: object

Loads a set of hyperparameters from a YAML pathlib.Path, dict or stream io.TextIOBase.

__init__(data, config=None, updates=())

config: Configurable = None

The application configuration used to update the hyperparameters from other sections.

data: dict[str, Any] | Path | TextIOBase

The source of data to load, which is a YAML pathlib.Path, dict or stream io.TextIOBase.

See:

updates

load(**kwargs) → HyperparamSet

Load and return the hyperparameter object graph from data.

Return type:

HyperparamSet

updates: Sequence[dict[str, Any]] = ()

A sequence of dictionaries with keys as HyperparamModel names and values as sections with values to set after loading using data.

exception zensols.datdesc.hyperparam.HyperparamValueError

Bases: HyperparamError

Raised for bad values set on a hyperparameter.

__annotations__ = {}

__firstlineno__ = 60

__module__ = 'zensols.datdesc.hyperparam'

__static_attributes__ = ()

class zensols.datdesc.hyperparam.RenderableHyperparamSet(path, hyperparam_table_default=None)

Bases: Renderable

Reads HyperparamSet instances and writes latex tables.

__init__(path, hyperparam_table_default=None)

hyperparam_table_default: dict[str, str] = None

Default settings for hyperparameter Table instances.

render(output, output_format=OutputFormat.short)

Write the rendered output.

Parameters:

output (Path) – either a file or directory (depending on the subclass)

Return type:

tuple[Path, ...]

zensols.datdesc.latex module

Contains the manager classes that invoke the tables to generate.

class zensols.datdesc.latex.LatexTable(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None, row_range=(1, -1), row_deletes=frozenset({}), booktabs=False)

Bases: Table

This subclass generates LaTeX tables.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None, row_range=(1, -1), row_deletes=frozenset({}), booktabs=False)

booktabs: bool = False

Whether or not to use the booktabs style table and to format using its style.

format_scientific(x, sig_digits=1)

Format x in scientific notation.

Parameters:

• x (float) – the number to format

• sig_digits (int) – the number of digits after the decimal point

Return type:

str

row_deletes: set[int] = frozenset({})

Rows to delete in the output produced by tabulate.

row_range: tuple[int, int] = (1, -1)

The range of rows to add to output produced by tabulate.

class zensols.datdesc.latex.RenderableLatexTable(path, factory)

Bases: Renderable

A renderable for table definitions in yaml files. The output is a latex .sty file with the output table as a command (see table_factory).

__init__(path, factory)

factory: TableFactory

Reads the table definitions file and writes a Latex .sty file of the generated tables from the CSV data.

get_artifacts()

Return artifacts created found on path.

get_tables()

Return the tables configured in path.

Return type:

Iterable[Table]

render(output)

Write the rendered output.

Parameters:

output (Path) – either a file or directory (depending on the subclass)

Return type:

Path

class zensols.datdesc.latex.SlackTable(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None, row_range=(1, -1), row_deletes=frozenset({}), booktabs=False, slack_column=0)

Bases: LatexTable

An instance of the table that fills up space based on the widest column.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None, row_range=(1, -1), row_deletes=frozenset({}), booktabs=False, slack_column=0)

property columns: str

Return the columns field in the Latex environment header.

slack_column: int = 0

Which column elastically grows or shrinks to make the table fit.

zensols.datdesc.opt module

zensols.datdesc.optscore module

zensols.datdesc.plots module

Common used plots for ML.

class zensols.datdesc.plots.BarPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, x_axis_label=None, y_axis_label=None, x_column_name=None, y_column_name=None, hue_column_name=None, x_label_rotation=0, key_title=None, log_scale=None, render_value_font_size=None, render_value_format=None, hue_palette=False, plot_params=<factory>)

Bases: PaletteContainerPlot, DataFramePlot

Create a bar plot using seaborn.barplot().

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, x_axis_label=None, y_axis_label=None, x_column_name=None, y_column_name=None, hue_column_name=None, x_label_rotation=0, key_title=None, log_scale=None, render_value_font_size=None, render_value_format=None, hue_palette=False, plot_params=<factory>)

hue_column_name: str = None

The column in data used for the data hue (each data category will have a unique in palette.

hue_palette: bool = False

Whether to use the hue to calculate the palette colors.

key_title: str = None

The title that goes in the key.

log_scale: float = None

The log scale of the Y-axis (see matplotlib.axes.Axis.set_yscale.

plot_params: dict[str, Any]

Parameters given to seaborn.barplot().

render_value_font_size: int = None

Whether to add Y-axis values to the bars and the font size to render.

render_value_format: str = None

The (f-string) format with value x for the Y-axis value labels.

x_axis_label: str = None

The axis name with the X label.

x_column_name: str = None

The data column with the X values.

x_label_rotation: float = 0

The degree of label rotation.

y_axis_label: str = None

The axis name with the Y label.

y_column_name: str = None

The data column with the Y values.

class zensols.datdesc.plots.CodePlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None, code='', context=None)

Bases: PaletteContainerPlot

A plotter that executes code_render as a Python block.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None, code='', context=None)

code: str = ''

The code used to render the plot. The code is executed with variable ax set the Axes```plot` set to this instance.

context: Any = None

Data that is useful, and made available, when rendering the plot.

class zensols.datdesc.plots.DataFramePlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None)

Bases: Plot

A base class for plots that render data from a Pandas dataframe.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None)

data: DataFrame = None

The data to plot.

class zensols.datdesc.plots.HeatMapPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, format='.2f', x_label_rotation=0, params=<factory>)

Bases: PaletteContainerPlot, DataFramePlot

Create heat map plot and optionally normalize. This uses seaborn’s heatmap.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, format='.2f', x_label_rotation=0, params=<factory>)

format: str = '.2f'

The format of the plots’s cell numerical values.

params: dict[str, Any]

Additional parameters to give to seaborn.heatmap().

x_label_rotation: float = 0

The degree of label rotation.

class zensols.datdesc.plots.HistPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None, data=<factory>, x_axis_label=None, y_axis_label=None, key_title=None, log_scale=None, plot_params=<factory>)

Bases: PaletteContainerPlot

Create a histogram plot using seaborn.histplot().

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None, data=<factory>, x_axis_label=None, y_axis_label=None, key_title=None, log_scale=None, plot_params=<factory>)

add(name, data)

Add occurances to use in the histogram.

Parameters:

• name (str) – the variable name

• data (Iterable[float]) – the data to render

data: list[tuple[str, DataFrame]]

The data to plot. Each element is tuple first components with the plot name.

key_title: str = None

The title that goes in the key.

log_scale: float = None

See the seaborn.histplot() log_scale parameter. This is also used to update the ticks if provided.

plot_params: dict[str, Any]

Parameters given to seaborn.histplot().

x_axis_label: str = None

The axis name with the X label.

y_axis_label: str = None

The axis name with the Y label.

class zensols.datdesc.plots.PaletteContainerPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None)

Bases: Plot

A base class that supports creating a color palette for subclasses.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, palette=None)

palette: str | Callable = None

Either the a list of color characters or a callable that takes the number of colors as input. For example, the Seaborn color palette (such as sns.color_palette('tab10', n_colors=n)). This is used as the palette parameter in the sns.pointplot call.

class zensols.datdesc.plots.PointPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, point_data=None, x_axis_name=None, y_axis_name=None, x_column_name='x', y_column_name='y', key_title=None, sample_rate=0, plot_params=<factory>, hue_name=None, hue_names=None)

Bases: PaletteContainerPlot, DataFramePlot

An abstract base class that renders overlapping lines that uses a seaborn pointplot.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, point_data=None, x_axis_name=None, y_axis_name=None, x_column_name='x', y_column_name='y', key_title=None, sample_rate=0, plot_params=<factory>, hue_name=None, hue_names=None)

add(name, line, x_vals=None)

Add the losses of a dataset by adding X values as incrementing integers the size of line.

Parameters:

• name (str) – the line name

• line (Iterable[float]) – the Y values for the line

• x_vals (Sequence) – the values used for the X axes, which defaults to range(1, n+ 1)

hue_name: str = None

The name of the heu given to seaborn.pointplot.

hue_names: tuple[str, ...] = None

Hue names give to seaborn.pointplot.

key_title: str = None

The title that goes in the key.

plot_params: dict[str, Any]

Parameters given to seaborn.plotpoint(). The default are decorative parameters for the marker size and line width.

point_data: list[tuple[str, DataFrame]] = None

The data to plot. Each element is tuple first components with the plot name. The second component is a dataframe with columns:

• x_column_name: the X values of the graph, usually an incrementing number

• y_column_name: a list loss float values

Optionally use add_line() to populate this list.

sample_rate: int = 0

Every $n$ data point in the list of losses is added to the plot.

set_data(df)

Set the point data using df.

x_axis_name: str = None

The axis name with the X label.

x_column_name: str = 'x'

The data column with the X values.

y_axis_name: str = None

The axis name with the Y label.

y_column_name: str | Sequence[tuple[str, str]] = 'y'

The data column(s) with the Y values.

class zensols.datdesc.plots.RadarPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, key_title=None, frame='circle', render_value_font_size=None, label_gap=None, alpha=0.25)

Bases: DataFramePlot

A radar plot (a.k.a. spider plolt).

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, key_title=None, frame='circle', render_value_font_size=None, label_gap=None, alpha=0.25)

alpha: float = 0.25

The fill alpha for each row of the dataframe.

frame: str = 'circle'

Shape of frame surrounding axes (circle or polygon).

key_title: str = None

The title that goes in the key.

label_gap: int = None

The spacing of the labels from the center of the plot.

render_value_font_size: int = None

Whether to add Y-axis values to the bars.

class zensols.datdesc.plots.ScatterPlot(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, scatter_data=None, x_axis_name=None, y_axis_name=None, x_column_name='x', y_column_name='y', key_title=None, sample_rate=0, plot_params=<factory>, hue_name=None, hue_names=None)

Bases: PaletteContainerPlot, DataFramePlot

Render a scatter plot using seaborn scatterplot.

This is a simpler version of PointPlot for when only point rendering is needed.

__init__(title=None, row=0, column=0, post_hooks=<factory>, legend_params=<factory>, code_pre_render=None, code_post_render=None, data=None, palette=None, scatter_data=None, x_axis_name=None, y_axis_name=None, x_column_name='x', y_column_name='y', key_title=None, sample_rate=0, plot_params=<factory>, hue_name=None, hue_names=None)

add(name, values, x_vals=None)

Add a series of Y values to the scatter plot.

Parameters:

• name (str) – the series name

• values (Iterable[float]) – the Y values for the series

• x_vals (Sequence) – the values used for the X axis, which defaults to range(1, n + 1)

hue_name: str = None

The name of the hue given to seaborn.scatterplot.

hue_names: tuple[str, ...] = None

Hue names given to seaborn.scatterplot.

key_title: str = None

The title that goes in the key.

plot_params: dict[str, Any]

Parameters given to seaborn.scatterplot().

sample_rate: int = 0

Every $n$ data point in the list is added to the plot.

scatter_data: list[tuple[str, DataFrame]] = None

The data to plot. Each element is a tuple with the plot name and a dataframe with columns:

• x_column_name: the X values of the graph

• y_column_name: the Y values of the graph

Optionally use add() to populate this list.

set_data(df)

Set the scatter data using df.

x_axis_name: str = None

The axis name with the X label.

x_column_name: str = 'x'

The data column with the X values.

y_axis_name: str = None

The axis name with the Y label.

y_column_name: str | Sequence[tuple[str, str]] = 'y'

The data column(s) with the Y values.

zensols.datdesc.proto module

class zensols.datdesc.proto.PrototypeApplication(app)

Bases: object

CLI_META = {'is_usage_visible': False}

__init__(app)

app: Application

proto()

Prototype test.

zensols.datdesc.render module

Classes to create first class object and process files.

class zensols.datdesc.render.Renderable(path)

Bases: Dictable

Creates rendered output from a machine readble input file.

__init__(path)

get_artifacts()

Return artifacts created found on path.

Return type:

Iterable[RenderableArtifact]

path: Path

The input definition of the object to render.

abstract render(output)

Write the rendered output.

Parameters:

output (Path) – either a file or directory (depending on the subclass)

Return type:

Path

class zensols.datdesc.render.RenderableArtifact(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>)

Bases: PersistableContainer, Dictable

Data that can be rendered from a Renderable.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>)

caption: str = ''

The human readable string used to the caption in the figure.

definition_file: Path = None

The YAML file from which this instance was created.

head: str = None

The header to use for the table, which is used as the text in the list of tables and made bold in the table.

name: str

The name of the artifiact, also used in reference labels.

property path: Path

The file that has the data used to populate this artifiact.

template: str

The figure template, which lives in the application configuration obj.yml.

template_params: dict[str, str]

Parameters used in the template.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write this instance as either a Writable or as a Dictable. If class attribute _DICTABLE_WRITABLE_DESCENDANTS is set as True, then use the write() method on children instead of writing the generated dictionary. Otherwise, write this instance by first creating a dict recursively using asdict(), then formatting the output.

If the attribute _DICTABLE_WRITE_EXCLUDES is set, those attributes are removed from what is written in the write() method.

Note that this attribute will need to be set in all descendants in the instance hierarchy since writing the object instance graph is done recursively.

Parameters:

• depth (int) – the starting indentation depth

• writer (TextIOBase) – the writer to dump the content of this writable

writes: list[str]

A list of what to output for this artifact. Each must be a method in the subclass and the default only renders the template. This is configurable so a client can decide what from the artifact is output.

class zensols.datdesc.render.RenderableFactory(config_factory, types, name_format)

Bases: Dictable

Creates instances of Renderable from file paths.

__init__(config_factory, types, name_format)

config_factory: ConfigFactory

Creates table and figure factories.

name_format: str

obj`config_factory`.

Type:

Section format used to create instances with

types: dict[str, Pattern | str]

Application config to regular expression mapping.

zensols.datdesc.renderlatex module

Things rendered for LaTeX files (usually .sty).

class zensols.datdesc.renderlatex.RenderableLatexArtifact(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>)

Bases: RenderableArtifact

Subclasses implement behavior to render LaTeX files. There are also utility method for generating latex commands.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>)

default_params: Sequence[Sequence[str]]

Default parameters to be substituted in the template that are interpolated by (i.e. LaTeX) numeric values such as #1, #2, etc. This is a sequence (list or tuple) of (<name>, [<default>]) where name is substituted by name in the template and default is the default if not given in params.

For any that have no default, which is a singleton paramenter name, arguments are created for the function that must be supplied to the function.

property package_name: str

Return the package name for the table in table_path.

params: dict[str, str]

Parameters used in the template that override of the default_params.

uses: list[str]

Comma separated list of packages to use.

class zensols.datdesc.renderlatex.RenderableLatexPackage(artifacts, name, description='{date} Artifacts')

Bases: Dictable

Generate a Latex table from a CSV file.

__init__(artifacts, name, description='{date} Artifacts')

artifacts: Sequence[RenderableLatexArtifact]

A list of instances to create Latex definitions.

description: str = '{date} Artifacts'

The package description.

name: str

The name Latex .sty package.

write(depth=0, writer=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)

Write the Latex table to the writer given in the initializer.

zensols.datdesc.table module

This module contains classes that generate tables.

class zensols.datdesc.table.Table(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None)

Bases: RenderableLatexArtifact

Generate and render a formatted table from tabular data.

A table manages the transformation of a dataframe into a rendered artifact, typically a LaTeX table. The source dataframe passes through several processing stages before being rendered with tabulate and the configured table template.

The dataframe processing stages are:

nascent

The original dataframe before any table-specific processing.

unformatted

The dataframe after code_pre has executed, but before numeric formatting is applied.

postformat

The dataframe after numeric formatting and code_post have been applied.

formatted

The final dataframe after structural and presentation formatting, including column manipulation, capitalization, font formatting, and code_format. This dataframe is used to generate the table.

Several Python code hooks allow customization at specific stages of the processing pipeline:

code_pre

Executed before numeric or presentation formatting. The local variable df contains the dataframe and may be modified or replaced.

code_post

Executed after numeric formatting, including rounding, percentages, thousands separators, and scientific notation, but before the final structural and presentation formatting steps.

code_format

Executed after all built-in dataframe formatting has completed. This is the final opportunity to modify df before it is passed to tabulate.

code_render

Executed after tabulate has rendered the dataframe to text but before the surrounding table template is rendered. The local variable table contains the rendered table lines and may be modified in place. This is useful for output-specific changes such as inserting LaTeX \multicolumn commands.

For example:

results:
  path: results.csv
  round_column_names:
    score: 2
  code_pre: |
    df = df[df['enabled']]
  code_post: |
    df['score'] = df['score'].astype(str)

Table variables may also reference values from intermediate processing stages through variables, allowing generated commands or rendered values to use either raw or formatted dataframe values. Subclasses provide output-specific behavior such as scientific-number formatting, row extraction for tabulate, and rendering of table and variable content.

__init__(name, path, template, template_params=<factory>, caption='', definition_file=None, head=None, writes=<factory>, uses=<factory>, default_params=<factory>, params=<factory>, type=None, hlines=<factory>, double_hlines=<factory>, rules=<factory>, column_keeps=None, column_removes=<factory>, column_renames=<factory>, column_value_replaces=<factory>, column_aligns=None, round_column_names=<factory>, percent_column_names=(), make_percent_column_names=<factory>, format_thousands_column_names=<factory>, format_scientific_column_names=<factory>, read_params=<factory>, tabulate_params=<factory>, replace_nan=None, blank_columns=<factory>, bold_cells=<factory>, bold_max_columns=<factory>, capitalize_columns=<factory>, index_col_name=None, variables=<factory>, code_pre=None, code_post=None, code_format=None, code_render=None)

asflatdict(*args, **kwargs)

Like asdict() but flatten in to a data structure suitable for writing to JSON or YAML.

Return type:

dict[str, Any]

blank_columns: list[int]

A list of column indexes to set to the empty string (i.e. 0th to fixed the Unnamed: 0 issues).

bold_cells: list[tuple[int, int]]

A list of row/column cells to bold.

bold_max_columns: list[str]

A list of column names that will have its max value bolded.

capitalize_columns: dict[str, bool]

Capitalize either sentences (False values) or every word (True values). The keys are column names.

code_format: str = None

Like code_post but modifies the table after this class’s all formatting of the table (including those applied by this class).

code_post: str = None

Like code_pre but modifies the table after this class’s modifications of the table.

code_pre: str = None

Python code executed that manipulates the table’s dataframe before modifications made by this class. The code has a local df variable and the returned value is used as the replacement. This is usually a one-liner used to subset the data etc. The code is evaluated with eval().

code_render: str = None

Like code_format but modifies the rendered table as a typing.List of lines with variable name table. Modify this list to change the final rendered output of the table, such as applying Latex multicolumn lines.

column_aligns: str = None

The alignment/justification (i.e. |l|l| for two columns). If not provided, they are automatically generated based on the columns of the table.

column_keeps: list[str] | None = None

If provided, only keep the columns in the list

column_removes: list[str]

The name of the columns to remove from the table, if any.

column_renames: dict[str, str]

Columns to rename, if any.

column_value_replaces: dict[str, dict[Any, Any]]

Data values to replace in the dataframe. It is keyed by the column name and values are the replacements. Each value is a dict with orignal value keys and the replacements as values.

property columns: str

Return the columns field in the Latex environment header.

property dataframe: DataFrame

The Pandas dataframe that holds the CSV data.

double_hlines: Sequence[int]

Indexes of rows to put double horizontal line breaks.

abstract format_scientific(x, sig_digits=1)

Format x in scientific notation.

Parameters:

• x (float) – the number to format

• sig_digits (int) – the number of digits after the decimal point

Return type:

str

format_scientific_column_names: dict[str, int | None]

Format a column using LaTeX formatted scientific notation using format_scientific(). Keys are column names and values is the mantissa length or 1 if None.

static format_thousand(x, apply_k=True, add_comma=True, round=None, max_k=1000)

Format a number as a string with comma separating thousands.

Parameters:

• x (int) – the number to format

• apply_k (bool) – add a K to the end of large numbers

• add_comma (bool) – whether to add a comma

• round (int) – the number to round the mantissa if given

• max_k (int) – the maximum value needed to reformat with K at the end

Return type:

str

format_thousands_column_names: dict[str, dict[str, Any] | None]

Columns to format using thousands, and optionally round. The keys are the column names of the table and the values are either None or the keyword arguments to format_thousand().

property formatted_dataframe: DataFrame

The dataframe with the formatting applied to it used to create the Latex table. Modifications such as string replacements for adding percents is done.

hlines: Sequence[int]

Indexes of rows to put horizontal line breaks.

index_col_name: str = None

If set, add an index column with the given name.

make_percent_column_names: dict[str, int | str]

Each columnn in the map will get rounded to the value * 100 of the name. For example, {'ann_per': 3} will round column ann_per to 3 decimal places.

If the values are strings then it is interpreted as a Python f-string using v as the value. For example, {'ann_per': '{v:.1f}'} gives a percentage to the first decimal without the percentage sign (%).

percent_column_names: Sequence[str] = ()

Column names that have a percent sign to be escaped.

read_params: dict[str, str]

Keyword arguments used in the read_csv() call when reading the CSV file.

replace_nan: str = None

Replace NaN values with a the value of this field as tabulate() is not using the missing value due to some bug I assume.

round_column_names: dict[str, tuple[int, int] | int]

Each column in the map will get rounded to their respective values.

For tuple values the number will be rounded as an integer if higher than a cutoff (second element), otherwise it is rounded to the decimal (first element).

rules: dict[int, str]

Like hlines but allows other horizontal lines such as toprule. Each key/value is a tuple of row and the verbatim text to add at that place.

tabulate_params: dict[str, str]

Keyword arguments used in the tabulate() call when writing the table. The default tells tabulate to not parse/format numerical data.

type: str = None

The type of table (i.e. one_column).

variables: dict[str, tuple[int, int] | str]

A mapping of variable names to a dataframe cell or Python code snipped that is evaluated with exec(). In LaTeX, this is done by setting a newcommand (see LatexTable).

If set to a tuple of (<row>, <column>) the value of the pre-formatted dataframe is used (see unformatted below).

If a Python evalution string, the code values must set variables v to the variable value. A variable stages is a Dict used to get one of the dataframes created at various stages of formatting the table with entries:

• nascent: same as dataframe

• unformatted: after the pre-evaluation but before any formatting

• postformat: after number formatting and post evaluation, but before remaining column and cell modifications

• formatted: same as formatted_dataframe

For example, the following uses the value at row 2 and column 3 of the unformatted dataframe:

v = stages['unformatted'].iloc[2, 3]

writes: list[str]

A list of what to output for this table. Entries are table and varaibles.

class zensols.datdesc.table.TableFactory(config_factory, table_section_regex, default_table_type)

Bases: Dictable

Reads the table definitions file and writes a Latex .sty file of the generated tables from the CSV data. Tables are created with either usage() or from_file(). See the usage documentation for information about the configuration files used by from_file().

__init__(config_factory, table_section_regex, default_table_type)

config_factory: ConfigFactory

The configuration factory used to create Table instances.

create(type=None, **params)

Create a table from the application configuration.

Parameters:

• type (str) – the name used to find the table by section

• params (dict[str, Any]) – the keyword arguments used to create the table

Return type:

Table

Returns:

a new instance of the table defined by the template

See:

get_table_names()

classmethod default_instance()

Get the singleton instance.

Return type:

TableFactory

default_table_type: str

The default name, which resolves to a section name, to use when creating anonymous tables.

from_file(table_path)

Return tables parsed from a YAML file (see class documentation).

Parameters:

table_path (Path) – the file containing the table configurations

Return type:

Iterable[Table]

get_table_names()

Return names of tables used in :meth:create.

Return type:

Iterable[str]

classmethod reset_default_instance()

Force default_instance() to re-instantiate a new instance on a subsequent call.

table_section_regex: Pattern

A regular expression that matches table entries.

to_file(table, table_path)

Save table as a YAML file to table_path.

Module contents

Generate Latex tables in a .sty file from CSV files. The paths to the CSV files to create tables from and their metadata is given as a YAML configuration file.