Clear the in memory instance data. If key :wipe-persistent? is true all fold and test/train split data is also cleared.

Return maps representing the data set distribution by class label. Each element of the returned sequence has the following keys:

• :class-label the class-label fo the instances
• :count the number of instances for :class-label

Divide the data into folds and initialize the current fold in the dataset split state. Using this kind of dataset split is useful for cross fold validation.

• folds number of folds to use, which defaults to 10

See set-fold

Divide the data into test and training buckets. The respective train/test buckets are dictated by the :set-type label given in parameter given to the :create-instances-fn as documented in elasticsearch-connection.

Divide the dataset into a test and training buckets.

• train-ratio this is the percentage of data in the train bucket, which defaults to 0.5

Keys

• :dist-type one of the following symbols: even each test/training set has an even distribution by class label uneven each test/training set has an uneven distribution by class label
• :shuffle? if true then shuffle the set before partitioning, otherwise just update the demarcation boundary
• :filter-fn if given a filter function that takes a key as input
• :max-instances the maximum number of instances per class
• :seed if given, seed the random number generator, otherwise don’t return random documents

Create a connection to the dataset DB cache.

Parameters

• index-name the name of the Elasticsearch index

Keys

• :create-instances-fn a function that computes the instance set (i.e. parses the utterance) and invoked by instances-load; this function takes a single argument, which is also a function that is used to load utterance in the DB; this function takes the following forms:

  • (fn [instance class-label] …
  • (fn [id instance class-label] …
  • (fn [id instance class-label set-type] …
    • id the unique identifier of the data point
    • instance is the data set instance (can be an N-deep map)
    • class-label the label of the class (can be nominal, double, integer)
    • set-type either :test, :train, :train-test (all) used to presort the data with divide-by-preset; note that it isn’t necessary to call divide-by-preset for the first invocation of instances-load

• :url the URL to the DB (defaults to http://localhost:9200)

• :mapping-type map type name (see ES docs)
• :cache-inst an atom used to cache instances by ID; if given this retrieves instances from the in memory map stored in the atom; otherwise it goes to ElasticSearch each time

Example

Create a connection that produces a list of 20 instances:

(defn- create-iter-connection []
  (letfn [(load-fn [add-fn]
            (doseq [i (range 20)]
              (add-fn (str i) (format "inst %d" i) (format "class %d" i))))]
    (elasticsearch-connection "tmp" :create-instances-fn load-fn)))

Distille the current data set (data and test/train splits) in a output-file. See freeze-dataset-to-writer.

Distille the current data set (data and test/train splits) in writer to be later restored withzensols.dataset.thaw/taw-connection.

Return all IDs based on the dataset split (see class docs).

Keys

• :set-type is either :train, :test, :train-test (all) and defaults to set-default-set-type or :train if not set

Get a specific instance by its ID.

This returns a map that has the following keys:

• :instance the instance data, which was set with :create-instances-fn in elasticsearch-connection

Get the number of total instances in the database. This result is independent of the dataset split state.

Return all instance data based on the dataset split (see class docs).

See instance-by-id for the data in each map sequence returned.

Keys

• :set-type is either :train, :test, :train-test (all) and defaults to set-default-set-type or :train if not set
• :include-ids? if non-nil return keys in the map as well

Return a map with class-labels for keys and corresponding instances for that class-label.

Keys

• :max-instances the maximum number of instances per class
• :seed if given, seed the random number generator, otherwise don’t return random documents

Return the number of datasets in the DB.

Parse and load the dataset in the DB.

Set the default connection.

Parameter conn is used in place of what is set with with-connection. This is very convenient and saves typing, but will get clobbered if a with-connection is used further down in the stack frame.

If the parameter is missing, it’s unset.

Set the default bucket (training or testing) to get data.

• :set-type is either :train (default) or :test; see elasticsearch-connection

See ids

Set the current fold in the dataset split state.

You must call divide-by-fold before calling this.

See the namespace docs for more information.

Set how much of the data from the DB to use. This is useful for cases where your dataset or corpus is huge and you only want to start with a small chunk until you get your models debugged.

Parameters

• ratio a number between (0-1]; by default this is 1

Note This removes any stored dataset split state

Get training vs testing dataset split statistics.

Execute a body with the form (with-connection connection …)

• connection is created with elasticsearch-connection

Write the data set to a spreadsheet. If the file name ends with a .csv a CSV file is written, otherwise an Excel file is written.

Keys

• :output-file where to write the file and defaults to res/resource-path :analysis-report
• :single? if true then create a single sheet, otherwise the training and testing buckets are split between sheets