Class Dataset

Initialize dataset instance

There are basically two different way to create a dataset:

1. Create a new dataset from samples and sample attributes. In this mode a two-dimensional ndarray has to be passed to the samples keyword argument and the corresponding samples attributes are provided via the labels and chunks arguments.

2. Copy contructor mode

   The second way is used internally to perform quick coyping of datasets, e.g. when performing feature selection. In this mode and the two dictionaries (data and dsattr) are required. For performance reasons this mode bypasses most of the sanity check performed by the previous mode, as for internal operations data integrity is assumed.

Each of the Keywords arguments overwrites what is/might be already in the data container.

Parameters:

• data (dict) - Dictionary with an arbitrary number of entries. The value for each key in the dict has to be an ndarray with the same length as the number of rows in the samples array. A special entry in this dictionary is 'samples', a 2d array (samples x features). A shallow copy is stored in the object.
• dsattr (dict) - Dictionary of dataset attributes. An arbitrary number of arbitrarily named and typed objects can be stored here. A shallow copy of the dictionary is stored in the object.
• dtype, type, |, None - If None -- do not change data type if samples is an ndarray. Otherwise convert samples to dtype.
• samples (ndarray) - 2d array (samples x features)
• labels - An array or scalar value defining labels for each samples. Generally labels should be numeric, unless labels_map is used
• labels_map (None or bool or dict) - Map original labels into numeric labels. If True, the mapping is computed if labels are literal. If is False, no mapping is computed. If dict instance -- provided mapping is verified and applied. If you want to have labels_map just be present given already numeric labels, just assign labels_map dictionary to existing dataset instance
• chunks - An array or scalar value defining chunks for each sample

Overrides: object.__init__

To verify if dataset is in the same state as when smth else was done

Like if classifier was trained on the same dataset as in question

Decorators:

• @property

Find samples which are on the boundaries of the blocks

Such samples might need to be removed. By default (with prior=0, post=0) ids of the first samples in a 'block' are reported

Parameters:

• prior (int) - how many samples prior to transition sample to include
• post (int) - how many samples post the transition sample to include
• attributes_to_track (list of basestring) - which attributes to track to decide on the boundary condition
• affected_labels (list of basestring) - for which labels to perform selection. If None - for all
• revert (bool) - either to revert the meaning and provide ids of samples which are found to not to be boundary samples

Overrides: object.__str__

Overrides: object.__repr__

(inherited documentation)

Parameters:

• uniq (bool) - Include summary over data attributes which have unique
• idhash (bool) - Include idhash value for dataset and samples
• stats (bool) - Include some basic statistics (mean, std, var) over dataset samples
• lstats (bool) - Include statistics on chunks/labels
• maxc (int) - Maximal number of chunks when provide details on labels/chunks
• maxl (int) - Maximal number of labels when provide details on labels/chunks

Parameters:

• maxc (int) - Maximal number of chunks when provide details
• maxl (int) - Maximal number of labels when provide details

Merge the samples of one Dataset object to another (in-place).

No dataset attributes, besides labels_map, will be merged! Additionally, a new set of unique origids will be generated.

Merge the samples two Dataset objects.

All data of both datasets is copied, concatenated and a new Dataset is returned.

NOTE: This can be a costly operation (both memory and time). If performance is important consider the '+=' operator.

Parameters:

• deep (bool) - deep flag is provided to __init__ for copy_{samples,data,dsattr}. By default full copy is done.

Select a number of features from the current set.

Returns a new Dataset object with a copy of corresponding features

from the original samples array.

WARNING: The order of ids determines the order of features in the returned dataset. This might be useful sometimes, but can also cause major headaches! Order would is verified when running in non-optimized code (if __debug__)

Parameters:

• ids - iterable container to select ids
• sort (bool) - if to sort Ids. Order matters and selectFeatures assumes incremental order. If not such, in non-optimized code selectFeatures would verify the order and sort

Obtain new dataset by applying mappers over features and/or samples.

While featuresmappers leave the sample attributes information unchanged, as the number of samples in the dataset is invariant, samplesmappers are also applied to the samples attributes themselves!

Applying a featuresmapper will destroy any feature grouping information.

TODO: selectFeatures is pretty much

applyMapper(featuresmapper=MaskMapper(...))

Parameters:

• featuresmapper (Mapper) - Mapper to somehow transform each sample's features
• samplesmapper (Mapper) - Mapper to transform each feature across samples
• train (bool) - Flag whether to train the mapper with this dataset before applying it.

Choose a subset of samples defined by samples IDs.

Returns a new dataset object containing the selected sample subset.

TODO: yoh, we might need to sort the mask if the mask is a list of ids and is not ordered. Clarify with Michael what is our intent here!

Returns:

tuple of (samples indexes, features indexes). Each item could be also None, if no selection on samples or features was requested (to discriminate between no selected items, and no selections)

Universal selector

WARNING: if you need to select duplicate samples (e.g. samples=[5,5]) or order of selected samples of features is important and has to be not ordered (e.g. samples=[3,2,1]), please use selectFeatures or selectSamples functions directly

Examples:

Mimique plain selectSamples:

dataset.select([1,2,3])
dataset[[1,2,3]]

Mimique plain selectFeatures:

dataset.select(slice(None), [1,2,3])
dataset.select('all', [1,2,3])
dataset[:, [1,2,3]]

Mixed (select features and samples):

dataset.select([1,2,3], [1, 2])
dataset[[1,2,3], [1, 2]]

Select samples matching some attributes:

dataset.select(labels=[1,2], chunks=[2,4])
dataset.select('labels', [1,2], 'chunks', [2,4])
dataset['labels', [1,2], 'chunks', [2,4]]

Mixed -- out of first 100 samples, select only those with labels 1 or 2 and belonging to chunks 2 or 4, and select features 2 and 3:

dataset.select(slice(0,100), [2,3], labels=[1,2], chunks=[2,4])
dataset[:100, [2,3], 'labels', [1,2], 'chunks', [2,4]]

Obtain indexes of interesting samples/features. See select() for more information

XXX somewhat obsoletes idsby...

Convinience dataset parts selection

See select for more information

Permute the labels.

TODO: rename status into something closer in semantics.

Parameters:

• status (bool) - Calling this method with set to True, the labels are permuted among all samples. If 'status' is False the original labels are restored.
• perchunk (bool) - If True permutation is limited to samples sharing the same chunk value. Therefore only the association of a certain sample with a label is permuted while keeping the absolute number of occurences of each label value within a certain chunk constant.
• assure_permute (bool) - If True, assures that labels are permutted, ie any one is different from the original one

Select a random set of samples.

If 'nperlabel' is an integer value, the specified number of samples is randomly choosen from the group of samples sharing a unique label value ( total number of selected samples: nperlabel x len(uniquelabels).

If 'nperlabel' is a list which's length has to match the number of unique label values. In this case 'nperlabel' specifies the number of samples that shall be selected from the samples with the corresponding label.

The method returns a Dataset object containing the selected samples.

Set labels map.

Checks for the validity of the mapping -- values should cover all existing labels in the dataset

Assign definition to featuregroups

XXX Feature-groups was not finished to be useful

Parameters:

• ids, list, or, 1d, array - To be selected features ids.

Returns:

ndarray: dtype='bool'

All selected features are set to True; False otherwise.

Parameters:

• mask, 1d, ndarray - Feature mask.

Returns:

ndarray: integer

Ids of non-zero (non-False) mask elements.