fiftyone.utils.eval.classification

Classification evaluation.

Copyright 2017-2024, Voxel51, Inc.

voxel51.com

Functions:

evaluate_classifications(samples, pred_field)

Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.

Classes:

ClassificationEvaluationConfig(pred_field, …)	Base class for configuring ClassificationEvaluation instances.
ClassificationEvaluation(config)	Base class for classification evaluation methods.
SimpleEvaluationConfig(pred_field, gt_field, …)	Simple classification evaluation config.
SimpleEvaluation(config)	Standard classification evaluation.
TopKEvaluationConfig(pred_field, gt_field[, k])	Top-k classification evaluation config.
TopKEvaluation(config)	Top-k classification evaluation.
BinaryEvaluationConfig(pred_field, gt_field, …)	Binary evaluation config.
BinaryEvaluation(config)	Binary classification evaluation.
ClassificationResults(samples, config, …)	Class that stores the results of a classification evaluation.
BinaryClassificationResults(samples, config, …)	Class that stores the results of a binary classification evaluation.

fiftyone.utils.eval.classification.evaluate_classifications(samples, pred_field, gt_field='ground_truth', eval_key=None, classes=None, missing=None, method=None, progress=None, **kwargs)

Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.

By default, this method simply compares the ground truth and prediction for each sample, but other strategies such as binary evaluation and top-k matching can be configured via the method parameter.

You can customize the evaluation method by passing additional parameters for the method’s config class as kwargs.

The natively provided method values and their associated configs are:

• "simple": SimpleEvaluationConfig

• "top-k": TopKEvaluationConfig

• "binary": BinaryEvaluationConfig

If an eval_key is specified, then this method will record some statistics on each sample:

• When evaluating sample-level fields, an eval_key field will be populated on each sample recording whether that sample’s prediction is correct.

• When evaluating frame-level fields, an eval_key field will be populated on each frame recording whether that frame’s prediction is correct. In addition, an eval_key field will be populated on each sample that records the average accuracy of the frame predictions of the sample.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

• gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Classification instances

• eval_key (None) – an evaluation key to use to refer to this evaluation

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

• missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

• method (None) – a string specifying the evaluation method to use. The supported values are fo.evaluation_config.classification_backends.keys() and the default is fo.evaluation_config.default_classification_backend

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

• **kwargs – optional keyword arguments for the constructor of the ClassificationEvaluationConfig being used

Returns

a ClassificationResults

class fiftyone.utils.eval.classification.ClassificationEvaluationConfig(pred_field, gt_field, **kwargs)

Bases: fiftyone.core.evaluation.EvaluationMethodConfig

Base class for configuring ClassificationEvaluation instances.

Parameters

• pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

• gt_field – the name of the field containing the ground truth fiftyone.core.labels.Classification instances

Attributes:

type	The type of run.
cls	The fully-qualified name of this BaseRunConfig class.
method	The name of the method.
run_cls	The BaseRun class associated with this config.

Methods:

attributes()	Returns the list of class attributes that will be serialized by serialize().
base_config_cls(type)	Returns the config class for the given run type.
build()	Builds the BaseRun instance associated with this config.
builder()	Returns a ConfigBuilder instance for this class.
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
default()	Returns the default config instance.
from_dict(d)	Constructs a BaseRunConfig from a serialized JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_kwargs(**kwargs)	Constructs a Config object from keyword arguments.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
load_credentials(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
load_default()	Loads the default config instance from file.
parse_array(d, key[, default])	Parses a raw array attribute.
parse_bool(d, key[, default])	Parses a boolean value.
parse_categorical(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
parse_dict(d, key[, default])	Parses a dictionary attribute.
parse_int(d, key[, default])	Parses an integer attribute.
parse_mutually_exclusive_fields(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
parse_number(d, key[, default])	Parses a number attribute.
parse_object(d, key, cls[, default])	Parses an object attribute.
parse_object_array(d, key, cls[, default])	Parses an array of objects.
parse_object_dict(d, key, cls[, default])	Parses a dictionary whose values are objects.
parse_path(d, key[, default])	Parses a path attribute.
parse_raw(d, key[, default])	Parses a raw (arbitrary) JSON field.
parse_string(d, key[, default])	Parses a string attribute.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
validate_all_or_nothing_fields(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

property type

The type of run.

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

static base_config_cls(type)

Returns the config class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunConfig subclass

build()

Builds the BaseRun instance associated with this config.

Returns

a BaseRun instance

classmethod builder()

Returns a ConfigBuilder instance for this class.

property cls

The fully-qualified name of this BaseRunConfig class.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters

d – a JSON dict

Returns

a BaseRunConfig

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)

Constructs a Config object from keyword arguments.

Parameters

**kwargs – keyword arguments that define the fields expected by cls

Returns

an instance of cls

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters

**kwargs – subclass-specific credentials

classmethod load_default()

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

property method

The name of the method.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw array attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)

Parses a boolean value.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

• d – a JSON dictionary

• key – the key to parse

• choices – either an iterable of possible values or an enum-like class whose attributes define the possible values

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)

Parses a dictionary attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)

Parses an integer attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Returns

the (field, value) that was set

Raises

ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)

Parses a number attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of d[key]

• default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the elements of list d[key]

• default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the values of dictionary d[key]

• default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw (arbitrary) JSON field.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)

Parses a string attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls

The BaseRun class associated with this config.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

static validate_all_or_nothing_fields(fields)

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Raises

ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.classification.ClassificationEvaluation(config)

Bases: fiftyone.core.evaluation.EvaluationMethod

Base class for classification evaluation methods.

Parameters

config – a ClassificationEvaluationConfig

Methods:

register_samples(samples, eval_key)	Registers the collection on which evaluation will be performed.
evaluate_samples(samples[, eval_key, …])	Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.
get_fields(samples, eval_key)	Gets the fields that were involved in the given run.
rename(samples, eval_key, new_eval_key)	Performs any necessary operations required to rename this run’s key.
cleanup(samples, eval_key)	Cleans up the results of the run with the given key from the collection.
delete_run(samples, key[, cleanup])	Deletes the results associated with the given run key from the collection.
delete_runs(samples[, cleanup])	Deletes all runs from the collection.
ensure_requirements()	Ensures that any necessary packages to execute this run are installed.
ensure_usage_requirements()	Ensures that any necessary packages to use existing results for this run are installed.
from_config(config)	Instantiates a Configurable class from a <cls>Config instance.
from_dict(d)	Instantiates a Configurable class from a <cls>Config dict.
from_json(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
from_kwargs(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
get_run_info(samples, key)	Gets the BaseRunInfo for the given key on the collection.
has_cached_run_results(samples, key)	Determines whether BaseRunResults for the given key are cached on the collection.
list_runs(samples[, type, method])	Returns the list of run keys on the given collection.
load_run_results(samples, key[, cache, …])	Loads the BaseRunResults for the given key on the collection.
load_run_view(samples, key[, select_fields])	Loads the fiftyone.core.view.DatasetView on which the specified run was performed.
parse(class_name[, module_name])	Parses a Configurable subclass name string.
register_run(samples, key[, overwrite, cleanup])	Registers a run of this method under the given key on the given collection.
run_info_cls()	The BaseRunInfo class associated with this class.
save_run_info(samples, run_info[, …])	Saves the run information on the collection.
save_run_results(samples, key, run_results)	Saves the run results on the collection.
update_run_config(samples, key, config)	Updates the BaseRunConfig for the given run on the collection.
update_run_key(samples, key, new_key)	Replaces the key for the given run with a new key.
validate(config)	Validates that the given config is an instance of <cls>Config.
validate_run(samples, key[, overwrite])	Validates that the collection can accept this run.

register_samples(samples, eval_key)

Registers the collection on which evaluation will be performed.

This method will be called before calling evaluate_samples(). Subclasses can extend this method to perform any setup required for an evaluation run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key – the evaluation key for this evaluation

evaluate_samples(samples, eval_key=None, classes=None, missing=None, progress=None)

Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key (None) – an evaluation key for this evaluation

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used for results purposes

• missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a ClassificationResults instance

get_fields(samples, eval_key)

Gets the fields that were involved in the given run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a list of fields

rename(samples, eval_key, new_eval_key)

Performs any necessary operations required to rename this run’s key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

cleanup(samples, eval_key)

Cleans up the results of the run with the given key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

classmethod delete_run(samples, key, cleanup=True)

Deletes the results associated with the given run key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() method

classmethod delete_runs(samples, cleanup=True)

Deletes all runs from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() methods

ensure_requirements()

Ensures that any necessary packages to execute this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

ensure_usage_requirements()

Ensures that any necessary packages to use existing results for this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

classmethod from_config(config)

Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)

Instantiates a Configurable class from a <cls>Config dict.

Parameters

d – a dict to construct a <cls>Config

Returns

an instance of cls

classmethod from_json(json_path)

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters

json_path – path to a JSON file for type <cls>Config

Returns

an instance of cls

classmethod from_kwargs(**kwargs)

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters

**kwargs – keyword arguments that define the fields of a <cls>Config dict

Returns

an instance of cls

classmethod get_run_info(samples, key)

Gets the BaseRunInfo for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a BaseRunInfo

classmethod has_cached_run_results(samples, key)

Determines whether BaseRunResults for the given key are cached on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

True/False

classmethod list_runs(samples, type=None, method=None, **kwargs)

Returns the list of run keys on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• type (None) –

  a specific run type to match, which can be:

  • a string fiftyone.core.runs.BaseRunConfig.type

  • a fiftyone.core.runs.BaseRun class or its fully-qualified class name string

• method (None) – a specific fiftyone.core.runs.BaseRunConfig.method string to match

• **kwargs – optional config parameters to match

Returns

a list of run keys

classmethod load_run_results(samples, key, cache=True, load_view=True, **kwargs)

Loads the BaseRunResults for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cache (True) – whether to cache the results on the collection

• load_view (True) – whether to load the run view in the results (True) or the full dataset (False)

• **kwargs – keyword arguments for the run’s BaseRunConfig.load_credentials() method

Returns

a BaseRunResults, or None if the run did not save results

classmethod load_run_view(samples, key, select_fields=False)

Loads the fiftyone.core.view.DatasetView on which the specified run was performed.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• select_fields (False) – whether to exclude fields involved in other runs of the same type

Returns

a fiftyone.core.view.DatasetView

static parse(class_name, module_name=None)

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters

• class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”

• module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns

the Configurable class config_cls: the Config class associated with cls

Return type

cls

register_run(samples, key, overwrite=True, cleanup=True)

Registers a run of this method under the given key on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod run_info_cls()

The BaseRunInfo class associated with this class.

classmethod save_run_info(samples, run_info, overwrite=True, cleanup=True)

Saves the run information on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• run_info – a BaseRunInfo

• overwrite (True) – whether to overwrite an existing run with the same key

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod save_run_results(samples, key, run_results, overwrite=True, cache=True)

Saves the run results on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• run_results – a BaseRunResults, or None

• overwrite (True) – whether to overwrite an existing result with the same key

• cache (True) – whether to cache the results on the collection

classmethod update_run_config(samples, key, config)

Updates the BaseRunConfig for the given run on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• config – a BaseRunConfig

classmethod update_run_key(samples, key, new_key)

Replaces the key for the given run with a new key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod validate(config)

Validates that the given config is an instance of <cls>Config.

Raises

ConfigurableError – if config is not an instance of <cls>Config

validate_run(samples, key, overwrite=True)

Validates that the collection can accept this run.

The run may be invalid if, for example, a run of a different type has already been run under the same key and thus overwriting it would cause ambiguity on how to cleanup the results.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

Raises

ValueError – if the run is invalid

class fiftyone.utils.eval.classification.SimpleEvaluationConfig(pred_field, gt_field, **kwargs)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluationConfig

Simple classification evaluation config.

Parameters

• pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

• gt_field – the name of the field containing the ground truth fiftyone.core.labels.Classification instances

Attributes:

method	The name of the method.
cls	The fully-qualified name of this BaseRunConfig class.
run_cls	The BaseRun class associated with this config.
type	The type of run.

Methods:

attributes()	Returns the list of class attributes that will be serialized by serialize().
base_config_cls(type)	Returns the config class for the given run type.
build()	Builds the BaseRun instance associated with this config.
builder()	Returns a ConfigBuilder instance for this class.
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
default()	Returns the default config instance.
from_dict(d)	Constructs a BaseRunConfig from a serialized JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_kwargs(**kwargs)	Constructs a Config object from keyword arguments.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
load_credentials(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
load_default()	Loads the default config instance from file.
parse_array(d, key[, default])	Parses a raw array attribute.
parse_bool(d, key[, default])	Parses a boolean value.
parse_categorical(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
parse_dict(d, key[, default])	Parses a dictionary attribute.
parse_int(d, key[, default])	Parses an integer attribute.
parse_mutually_exclusive_fields(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
parse_number(d, key[, default])	Parses a number attribute.
parse_object(d, key, cls[, default])	Parses an object attribute.
parse_object_array(d, key, cls[, default])	Parses an array of objects.
parse_object_dict(d, key, cls[, default])	Parses a dictionary whose values are objects.
parse_path(d, key[, default])	Parses a path attribute.
parse_raw(d, key[, default])	Parses a raw (arbitrary) JSON field.
parse_string(d, key[, default])	Parses a string attribute.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
validate_all_or_nothing_fields(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

property method

The name of the method.

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

static base_config_cls(type)

Returns the config class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunConfig subclass

build()

Builds the BaseRun instance associated with this config.

Returns

a BaseRun instance

classmethod builder()

Returns a ConfigBuilder instance for this class.

property cls

The fully-qualified name of this BaseRunConfig class.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters

d – a JSON dict

Returns

a BaseRunConfig

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)

Constructs a Config object from keyword arguments.

Parameters

**kwargs – keyword arguments that define the fields expected by cls

Returns

an instance of cls

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters

**kwargs – subclass-specific credentials

classmethod load_default()

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw array attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)

Parses a boolean value.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

• d – a JSON dictionary

• key – the key to parse

• choices – either an iterable of possible values or an enum-like class whose attributes define the possible values

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)

Parses a dictionary attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)

Parses an integer attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Returns

the (field, value) that was set

Raises

ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)

Parses a number attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of d[key]

• default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the elements of list d[key]

• default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the values of dictionary d[key]

• default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw (arbitrary) JSON field.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)

Parses a string attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls

The BaseRun class associated with this config.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

property type

The type of run.

static validate_all_or_nothing_fields(fields)

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Raises

ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.classification.SimpleEvaluation(config)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluation

Standard classification evaluation.

Parameters

config – a SimpleClassificationEvaluationConfig

Methods:

register_samples(samples, eval_key)	Registers the collection on which evaluation will be performed.
evaluate_samples(samples[, eval_key, …])	Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.
cleanup(samples, eval_key)	Cleans up the results of the run with the given key from the collection.
delete_run(samples, key[, cleanup])	Deletes the results associated with the given run key from the collection.
delete_runs(samples[, cleanup])	Deletes all runs from the collection.
ensure_requirements()	Ensures that any necessary packages to execute this run are installed.
ensure_usage_requirements()	Ensures that any necessary packages to use existing results for this run are installed.
from_config(config)	Instantiates a Configurable class from a <cls>Config instance.
from_dict(d)	Instantiates a Configurable class from a <cls>Config dict.
from_json(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
from_kwargs(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
get_fields(samples, eval_key)	Gets the fields that were involved in the given run.
get_run_info(samples, key)	Gets the BaseRunInfo for the given key on the collection.
has_cached_run_results(samples, key)	Determines whether BaseRunResults for the given key are cached on the collection.
list_runs(samples[, type, method])	Returns the list of run keys on the given collection.
load_run_results(samples, key[, cache, …])	Loads the BaseRunResults for the given key on the collection.
load_run_view(samples, key[, select_fields])	Loads the fiftyone.core.view.DatasetView on which the specified run was performed.
parse(class_name[, module_name])	Parses a Configurable subclass name string.
register_run(samples, key[, overwrite, cleanup])	Registers a run of this method under the given key on the given collection.
rename(samples, eval_key, new_eval_key)	Performs any necessary operations required to rename this run’s key.
run_info_cls()	The BaseRunInfo class associated with this class.
save_run_info(samples, run_info[, …])	Saves the run information on the collection.
save_run_results(samples, key, run_results)	Saves the run results on the collection.
update_run_config(samples, key, config)	Updates the BaseRunConfig for the given run on the collection.
update_run_key(samples, key, new_key)	Replaces the key for the given run with a new key.
validate(config)	Validates that the given config is an instance of <cls>Config.
validate_run(samples, key[, overwrite])	Validates that the collection can accept this run.

register_samples(samples, eval_key)

Registers the collection on which evaluation will be performed.

This method will be called before calling evaluate_samples(). Subclasses can extend this method to perform any setup required for an evaluation run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key – the evaluation key for this evaluation

evaluate_samples(samples, eval_key=None, classes=None, missing=None, progress=None)

Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key (None) – an evaluation key for this evaluation

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used for results purposes

• missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a ClassificationResults instance

cleanup(samples, eval_key)

Cleans up the results of the run with the given key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

classmethod delete_run(samples, key, cleanup=True)

Deletes the results associated with the given run key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() method

classmethod delete_runs(samples, cleanup=True)

Deletes all runs from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() methods

ensure_requirements()

Ensures that any necessary packages to execute this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

ensure_usage_requirements()

Ensures that any necessary packages to use existing results for this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

classmethod from_config(config)

Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)

Instantiates a Configurable class from a <cls>Config dict.

Parameters

d – a dict to construct a <cls>Config

Returns

an instance of cls

classmethod from_json(json_path)

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters

json_path – path to a JSON file for type <cls>Config

Returns

an instance of cls

classmethod from_kwargs(**kwargs)

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters

**kwargs – keyword arguments that define the fields of a <cls>Config dict

Returns

an instance of cls

get_fields(samples, eval_key)

Gets the fields that were involved in the given run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a list of fields

classmethod get_run_info(samples, key)

Gets the BaseRunInfo for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a BaseRunInfo

classmethod has_cached_run_results(samples, key)

Determines whether BaseRunResults for the given key are cached on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

True/False

classmethod list_runs(samples, type=None, method=None, **kwargs)

Returns the list of run keys on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• type (None) –

  a specific run type to match, which can be:

  • a string fiftyone.core.runs.BaseRunConfig.type

  • a fiftyone.core.runs.BaseRun class or its fully-qualified class name string

• method (None) – a specific fiftyone.core.runs.BaseRunConfig.method string to match

• **kwargs – optional config parameters to match

Returns

a list of run keys

classmethod load_run_results(samples, key, cache=True, load_view=True, **kwargs)

Loads the BaseRunResults for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cache (True) – whether to cache the results on the collection

• load_view (True) – whether to load the run view in the results (True) or the full dataset (False)

• **kwargs – keyword arguments for the run’s BaseRunConfig.load_credentials() method

Returns

a BaseRunResults, or None if the run did not save results

classmethod load_run_view(samples, key, select_fields=False)

Loads the fiftyone.core.view.DatasetView on which the specified run was performed.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• select_fields (False) – whether to exclude fields involved in other runs of the same type

Returns

a fiftyone.core.view.DatasetView

static parse(class_name, module_name=None)

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters

• class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”

• module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns

the Configurable class config_cls: the Config class associated with cls

Return type

cls

register_run(samples, key, overwrite=True, cleanup=True)

Registers a run of this method under the given key on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

rename(samples, eval_key, new_eval_key)

Performs any necessary operations required to rename this run’s key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod run_info_cls()

The BaseRunInfo class associated with this class.

classmethod save_run_info(samples, run_info, overwrite=True, cleanup=True)

Saves the run information on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• run_info – a BaseRunInfo

• overwrite (True) – whether to overwrite an existing run with the same key

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod save_run_results(samples, key, run_results, overwrite=True, cache=True)

Saves the run results on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• run_results – a BaseRunResults, or None

• overwrite (True) – whether to overwrite an existing result with the same key

• cache (True) – whether to cache the results on the collection

classmethod update_run_config(samples, key, config)

Updates the BaseRunConfig for the given run on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• config – a BaseRunConfig

classmethod update_run_key(samples, key, new_key)

Replaces the key for the given run with a new key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod validate(config)

Validates that the given config is an instance of <cls>Config.

Raises

ConfigurableError – if config is not an instance of <cls>Config

validate_run(samples, key, overwrite=True)

Validates that the collection can accept this run.

The run may be invalid if, for example, a run of a different type has already been run under the same key and thus overwriting it would cause ambiguity on how to cleanup the results.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

Raises

ValueError – if the run is invalid

class fiftyone.utils.eval.classification.TopKEvaluationConfig(pred_field, gt_field, k=5, **kwargs)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluationConfig

Top-k classification evaluation config.

Parameters

• pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

• gt_field – the name of the field containing the ground truth fiftyone.core.labels.Classification instances

• k (5) – the top-k value to use when assessing accuracy

Attributes:

method	The name of the method.
cls	The fully-qualified name of this BaseRunConfig class.
run_cls	The BaseRun class associated with this config.
type	The type of run.

Methods:

attributes()	Returns the list of class attributes that will be serialized by serialize().
base_config_cls(type)	Returns the config class for the given run type.
build()	Builds the BaseRun instance associated with this config.
builder()	Returns a ConfigBuilder instance for this class.
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
default()	Returns the default config instance.
from_dict(d)	Constructs a BaseRunConfig from a serialized JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_kwargs(**kwargs)	Constructs a Config object from keyword arguments.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
load_credentials(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
load_default()	Loads the default config instance from file.
parse_array(d, key[, default])	Parses a raw array attribute.
parse_bool(d, key[, default])	Parses a boolean value.
parse_categorical(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
parse_dict(d, key[, default])	Parses a dictionary attribute.
parse_int(d, key[, default])	Parses an integer attribute.
parse_mutually_exclusive_fields(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
parse_number(d, key[, default])	Parses a number attribute.
parse_object(d, key, cls[, default])	Parses an object attribute.
parse_object_array(d, key, cls[, default])	Parses an array of objects.
parse_object_dict(d, key, cls[, default])	Parses a dictionary whose values are objects.
parse_path(d, key[, default])	Parses a path attribute.
parse_raw(d, key[, default])	Parses a raw (arbitrary) JSON field.
parse_string(d, key[, default])	Parses a string attribute.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
validate_all_or_nothing_fields(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

property method

The name of the method.

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

static base_config_cls(type)

Returns the config class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunConfig subclass

build()

Builds the BaseRun instance associated with this config.

Returns

a BaseRun instance

classmethod builder()

Returns a ConfigBuilder instance for this class.

property cls

The fully-qualified name of this BaseRunConfig class.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters

d – a JSON dict

Returns

a BaseRunConfig

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)

Constructs a Config object from keyword arguments.

Parameters

**kwargs – keyword arguments that define the fields expected by cls

Returns

an instance of cls

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters

**kwargs – subclass-specific credentials

classmethod load_default()

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw array attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)

Parses a boolean value.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

• d – a JSON dictionary

• key – the key to parse

• choices – either an iterable of possible values or an enum-like class whose attributes define the possible values

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)

Parses a dictionary attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)

Parses an integer attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Returns

the (field, value) that was set

Raises

ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)

Parses a number attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of d[key]

• default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the elements of list d[key]

• default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the values of dictionary d[key]

• default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw (arbitrary) JSON field.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)

Parses a string attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls

The BaseRun class associated with this config.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

property type

The type of run.

static validate_all_or_nothing_fields(fields)

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Raises

ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.classification.TopKEvaluation(config)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluation

Top-k classification evaluation.

This method requires the logits field of each predicted object to be populated, and the list of class labels corresponding to these logits must be provided.

Parameters

config – a TopKEvaluationConfig

Methods:

register_samples(samples, eval_key)	Registers the collection on which evaluation will be performed.
evaluate_samples(samples[, eval_key, …])	Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.
cleanup(samples, eval_key)	Cleans up the results of the run with the given key from the collection.
delete_run(samples, key[, cleanup])	Deletes the results associated with the given run key from the collection.
delete_runs(samples[, cleanup])	Deletes all runs from the collection.
ensure_requirements()	Ensures that any necessary packages to execute this run are installed.
ensure_usage_requirements()	Ensures that any necessary packages to use existing results for this run are installed.
from_config(config)	Instantiates a Configurable class from a <cls>Config instance.
from_dict(d)	Instantiates a Configurable class from a <cls>Config dict.
from_json(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
from_kwargs(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
get_fields(samples, eval_key)	Gets the fields that were involved in the given run.
get_run_info(samples, key)	Gets the BaseRunInfo for the given key on the collection.
has_cached_run_results(samples, key)	Determines whether BaseRunResults for the given key are cached on the collection.
list_runs(samples[, type, method])	Returns the list of run keys on the given collection.
load_run_results(samples, key[, cache, …])	Loads the BaseRunResults for the given key on the collection.
load_run_view(samples, key[, select_fields])	Loads the fiftyone.core.view.DatasetView on which the specified run was performed.
parse(class_name[, module_name])	Parses a Configurable subclass name string.
register_run(samples, key[, overwrite, cleanup])	Registers a run of this method under the given key on the given collection.
rename(samples, eval_key, new_eval_key)	Performs any necessary operations required to rename this run’s key.
run_info_cls()	The BaseRunInfo class associated with this class.
save_run_info(samples, run_info[, …])	Saves the run information on the collection.
save_run_results(samples, key, run_results)	Saves the run results on the collection.
update_run_config(samples, key, config)	Updates the BaseRunConfig for the given run on the collection.
update_run_key(samples, key, new_key)	Replaces the key for the given run with a new key.
validate(config)	Validates that the given config is an instance of <cls>Config.
validate_run(samples, key[, overwrite])	Validates that the collection can accept this run.

register_samples(samples, eval_key)

Registers the collection on which evaluation will be performed.

This method will be called before calling evaluate_samples(). Subclasses can extend this method to perform any setup required for an evaluation run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key – the evaluation key for this evaluation

evaluate_samples(samples, eval_key=None, classes=None, missing=None, progress=None)

Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key (None) – an evaluation key for this evaluation

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used for results purposes

• missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a ClassificationResults instance

cleanup(samples, eval_key)

Cleans up the results of the run with the given key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

classmethod delete_run(samples, key, cleanup=True)

Deletes the results associated with the given run key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() method

classmethod delete_runs(samples, cleanup=True)

Deletes all runs from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() methods

ensure_requirements()

Ensures that any necessary packages to execute this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

ensure_usage_requirements()

Ensures that any necessary packages to use existing results for this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

classmethod from_config(config)

Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)

Instantiates a Configurable class from a <cls>Config dict.

Parameters

d – a dict to construct a <cls>Config

Returns

an instance of cls

classmethod from_json(json_path)

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters

json_path – path to a JSON file for type <cls>Config

Returns

an instance of cls

classmethod from_kwargs(**kwargs)

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters

**kwargs – keyword arguments that define the fields of a <cls>Config dict

Returns

an instance of cls

get_fields(samples, eval_key)

Gets the fields that were involved in the given run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a list of fields

classmethod get_run_info(samples, key)

Gets the BaseRunInfo for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a BaseRunInfo

classmethod has_cached_run_results(samples, key)

Determines whether BaseRunResults for the given key are cached on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

True/False

classmethod list_runs(samples, type=None, method=None, **kwargs)

Returns the list of run keys on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• type (None) –

  a specific run type to match, which can be:

  • a string fiftyone.core.runs.BaseRunConfig.type

  • a fiftyone.core.runs.BaseRun class or its fully-qualified class name string

• method (None) – a specific fiftyone.core.runs.BaseRunConfig.method string to match

• **kwargs – optional config parameters to match

Returns

a list of run keys

classmethod load_run_results(samples, key, cache=True, load_view=True, **kwargs)

Loads the BaseRunResults for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cache (True) – whether to cache the results on the collection

• load_view (True) – whether to load the run view in the results (True) or the full dataset (False)

• **kwargs – keyword arguments for the run’s BaseRunConfig.load_credentials() method

Returns

a BaseRunResults, or None if the run did not save results

classmethod load_run_view(samples, key, select_fields=False)

Loads the fiftyone.core.view.DatasetView on which the specified run was performed.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• select_fields (False) – whether to exclude fields involved in other runs of the same type

Returns

a fiftyone.core.view.DatasetView

static parse(class_name, module_name=None)

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters

• class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”

• module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns

the Configurable class config_cls: the Config class associated with cls

Return type

cls

register_run(samples, key, overwrite=True, cleanup=True)

Registers a run of this method under the given key on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

rename(samples, eval_key, new_eval_key)

Performs any necessary operations required to rename this run’s key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod run_info_cls()

The BaseRunInfo class associated with this class.

classmethod save_run_info(samples, run_info, overwrite=True, cleanup=True)

Saves the run information on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• run_info – a BaseRunInfo

• overwrite (True) – whether to overwrite an existing run with the same key

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod save_run_results(samples, key, run_results, overwrite=True, cache=True)

Saves the run results on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• run_results – a BaseRunResults, or None

• overwrite (True) – whether to overwrite an existing result with the same key

• cache (True) – whether to cache the results on the collection

classmethod update_run_config(samples, key, config)

Updates the BaseRunConfig for the given run on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• config – a BaseRunConfig

classmethod update_run_key(samples, key, new_key)

Replaces the key for the given run with a new key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod validate(config)

Validates that the given config is an instance of <cls>Config.

Raises

ConfigurableError – if config is not an instance of <cls>Config

validate_run(samples, key, overwrite=True)

Validates that the collection can accept this run.

The run may be invalid if, for example, a run of a different type has already been run under the same key and thus overwriting it would cause ambiguity on how to cleanup the results.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

Raises

ValueError – if the run is invalid

class fiftyone.utils.eval.classification.BinaryEvaluationConfig(pred_field, gt_field, **kwargs)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluationConfig

Binary evaluation config.

Parameters

• pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

• gt_field – the name of the field containing the ground truth

Attributes:

method	The name of the method.
cls	The fully-qualified name of this BaseRunConfig class.
run_cls	The BaseRun class associated with this config.
type	The type of run.

Methods:

attributes()	Returns the list of class attributes that will be serialized by serialize().
base_config_cls(type)	Returns the config class for the given run type.
build()	Builds the BaseRun instance associated with this config.
builder()	Returns a ConfigBuilder instance for this class.
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
default()	Returns the default config instance.
from_dict(d)	Constructs a BaseRunConfig from a serialized JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_kwargs(**kwargs)	Constructs a Config object from keyword arguments.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
load_credentials(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
load_default()	Loads the default config instance from file.
parse_array(d, key[, default])	Parses a raw array attribute.
parse_bool(d, key[, default])	Parses a boolean value.
parse_categorical(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
parse_dict(d, key[, default])	Parses a dictionary attribute.
parse_int(d, key[, default])	Parses an integer attribute.
parse_mutually_exclusive_fields(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
parse_number(d, key[, default])	Parses a number attribute.
parse_object(d, key, cls[, default])	Parses an object attribute.
parse_object_array(d, key, cls[, default])	Parses an array of objects.
parse_object_dict(d, key, cls[, default])	Parses a dictionary whose values are objects.
parse_path(d, key[, default])	Parses a path attribute.
parse_raw(d, key[, default])	Parses a raw (arbitrary) JSON field.
parse_string(d, key[, default])	Parses a string attribute.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
validate_all_or_nothing_fields(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

property method

The name of the method.

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

static base_config_cls(type)

Returns the config class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunConfig subclass

build()

Builds the BaseRun instance associated with this config.

Returns

a BaseRun instance

classmethod builder()

Returns a ConfigBuilder instance for this class.

property cls

The fully-qualified name of this BaseRunConfig class.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters

d – a JSON dict

Returns

a BaseRunConfig

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)

Constructs a Config object from keyword arguments.

Parameters

**kwargs – keyword arguments that define the fields expected by cls

Returns

an instance of cls

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters

**kwargs – subclass-specific credentials

classmethod load_default()

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw array attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)

Parses a boolean value.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

• d – a JSON dictionary

• key – the key to parse

• choices – either an iterable of possible values or an enum-like class whose attributes define the possible values

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)

Parses a dictionary attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)

Parses an integer attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Returns

the (field, value) that was set

Raises

ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)

Parses a number attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of d[key]

• default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the elements of list d[key]

• default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

• d – a JSON dictionary

• key – the key to parse

• cls – the class of the values of dictionary d[key]

• default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)

Parses a raw (arbitrary) JSON field.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)

Parses a string attribute.

Parameters

• d – a JSON dictionary

• key – the key to parse

• default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls

The BaseRun class associated with this config.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

property type

The type of run.

static validate_all_or_nothing_fields(fields)

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters

fields – a dictionary of pre-parsed fields

Raises

ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.classification.BinaryEvaluation(config)

Bases: fiftyone.utils.eval.classification.ClassificationEvaluation

Binary classification evaluation.

Evaluates the binary classification predictions in the given collection with respect to the specified ground truth labels.

Any missing ground truth or prediction labels are assumed to be examples of the negative class (with zero confidence, for predictions).

If an eval_key is specified, this method will record the TP/FP/FN/TN status of each prediction in this field.

This method requires that the (neg_label, pos_label) label strings for the task are provided via the classes argument.

Parameters

config – a BinaryEvaluationConfig

Methods:

register_samples(samples, eval_key)	Registers the collection on which evaluation will be performed.
evaluate_samples(samples[, eval_key, …])	Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.
cleanup(samples, eval_key)	Cleans up the results of the run with the given key from the collection.
delete_run(samples, key[, cleanup])	Deletes the results associated with the given run key from the collection.
delete_runs(samples[, cleanup])	Deletes all runs from the collection.
ensure_requirements()	Ensures that any necessary packages to execute this run are installed.
ensure_usage_requirements()	Ensures that any necessary packages to use existing results for this run are installed.
from_config(config)	Instantiates a Configurable class from a <cls>Config instance.
from_dict(d)	Instantiates a Configurable class from a <cls>Config dict.
from_json(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
from_kwargs(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
get_fields(samples, eval_key)	Gets the fields that were involved in the given run.
get_run_info(samples, key)	Gets the BaseRunInfo for the given key on the collection.
has_cached_run_results(samples, key)	Determines whether BaseRunResults for the given key are cached on the collection.
list_runs(samples[, type, method])	Returns the list of run keys on the given collection.
load_run_results(samples, key[, cache, …])	Loads the BaseRunResults for the given key on the collection.
load_run_view(samples, key[, select_fields])	Loads the fiftyone.core.view.DatasetView on which the specified run was performed.
parse(class_name[, module_name])	Parses a Configurable subclass name string.
register_run(samples, key[, overwrite, cleanup])	Registers a run of this method under the given key on the given collection.
rename(samples, eval_key, new_eval_key)	Performs any necessary operations required to rename this run’s key.
run_info_cls()	The BaseRunInfo class associated with this class.
save_run_info(samples, run_info[, …])	Saves the run information on the collection.
save_run_results(samples, key, run_results)	Saves the run results on the collection.
update_run_config(samples, key, config)	Updates the BaseRunConfig for the given run on the collection.
update_run_key(samples, key, new_key)	Replaces the key for the given run with a new key.
validate(config)	Validates that the given config is an instance of <cls>Config.
validate_run(samples, key[, overwrite])	Validates that the collection can accept this run.

register_samples(samples, eval_key)

Registers the collection on which evaluation will be performed.

This method will be called before calling evaluate_samples(). Subclasses can extend this method to perform any setup required for an evaluation run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key – the evaluation key for this evaluation

evaluate_samples(samples, eval_key=None, classes=None, missing=None, progress=None)

Evaluates the predicted classifications in the given samples with respect to the specified ground truth labels.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• eval_key (None) – an evaluation key for this evaluation

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used for results purposes

• missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a ClassificationResults instance

cleanup(samples, eval_key)

Cleans up the results of the run with the given key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

classmethod delete_run(samples, key, cleanup=True)

Deletes the results associated with the given run key from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() method

classmethod delete_runs(samples, cleanup=True)

Deletes all runs from the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• cleanup (True) – whether to execute the run’s BaseRun.cleanup() methods

ensure_requirements()

Ensures that any necessary packages to execute this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

ensure_usage_requirements()

Ensures that any necessary packages to use existing results for this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

classmethod from_config(config)

Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)

Instantiates a Configurable class from a <cls>Config dict.

Parameters

d – a dict to construct a <cls>Config

Returns

an instance of cls

classmethod from_json(json_path)

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters

json_path – path to a JSON file for type <cls>Config

Returns

an instance of cls

classmethod from_kwargs(**kwargs)

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters

**kwargs – keyword arguments that define the fields of a <cls>Config dict

Returns

an instance of cls

get_fields(samples, eval_key)

Gets the fields that were involved in the given run.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a list of fields

classmethod get_run_info(samples, key)

Gets the BaseRunInfo for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

a BaseRunInfo

classmethod has_cached_run_results(samples, key)

Determines whether BaseRunResults for the given key are cached on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

Returns

True/False

classmethod list_runs(samples, type=None, method=None, **kwargs)

Returns the list of run keys on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• type (None) –

  a specific run type to match, which can be:

  • a string fiftyone.core.runs.BaseRunConfig.type

  • a fiftyone.core.runs.BaseRun class or its fully-qualified class name string

• method (None) – a specific fiftyone.core.runs.BaseRunConfig.method string to match

• **kwargs – optional config parameters to match

Returns

a list of run keys

classmethod load_run_results(samples, key, cache=True, load_view=True, **kwargs)

Loads the BaseRunResults for the given key on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• cache (True) – whether to cache the results on the collection

• load_view (True) – whether to load the run view in the results (True) or the full dataset (False)

• **kwargs – keyword arguments for the run’s BaseRunConfig.load_credentials() method

Returns

a BaseRunResults, or None if the run did not save results

classmethod load_run_view(samples, key, select_fields=False)

Loads the fiftyone.core.view.DatasetView on which the specified run was performed.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• select_fields (False) – whether to exclude fields involved in other runs of the same type

Returns

a fiftyone.core.view.DatasetView

static parse(class_name, module_name=None)

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters

• class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”

• module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns

the Configurable class config_cls: the Config class associated with cls

Return type

cls

register_run(samples, key, overwrite=True, cleanup=True)

Registers a run of this method under the given key on the given collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

rename(samples, eval_key, new_eval_key)

Performs any necessary operations required to rename this run’s key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod run_info_cls()

The BaseRunInfo class associated with this class.

classmethod save_run_info(samples, run_info, overwrite=True, cleanup=True)

Saves the run information on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• run_info – a BaseRunInfo

• overwrite (True) – whether to overwrite an existing run with the same key

• cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod save_run_results(samples, key, run_results, overwrite=True, cache=True)

Saves the run results on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• run_results – a BaseRunResults, or None

• overwrite (True) – whether to overwrite an existing result with the same key

• cache (True) – whether to cache the results on the collection

classmethod update_run_config(samples, key, config)

Updates the BaseRunConfig for the given run on the collection.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• config – a BaseRunConfig

classmethod update_run_key(samples, key, new_key)

Replaces the key for the given run with a new key.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• new_key – a new run key

classmethod validate(config)

Validates that the given config is an instance of <cls>Config.

Raises

ConfigurableError – if config is not an instance of <cls>Config

validate_run(samples, key, overwrite=True)

Validates that the collection can accept this run.

The run may be invalid if, for example, a run of a different type has already been run under the same key and thus overwriting it would cause ambiguity on how to cleanup the results.

Parameters

• samples – a fiftyone.core.collections.SampleCollection

• key – a run key

• overwrite (True) – whether to allow overwriting an existing run of the same type

Raises

ValueError – if the run is invalid

class fiftyone.utils.eval.classification.ClassificationResults(samples, config, eval_key, ytrue, ypred, confs=None, weights=None, ytrue_ids=None, ypred_ids=None, classes=None, missing=None, backend=None)

Bases: fiftyone.utils.eval.base.BaseEvaluationResults

Class that stores the results of a classification evaluation.

Parameters

• samples – the fiftyone.core.collections.SampleCollection used

• config – the ClassificationEvaluationConfig used

• eval_key – the evaluation key

• ytrue – a list of ground truth labels

• ypred – a list of predicted labels

• confs (None) – an optional list of confidences for the predictions

• weights (None) – an optional list of sample weights

• ytrue_ids (None) – a list of IDs for the ground truth labels

• ypred_ids (None) – a list of IDs for the predicted labels

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

• missing (None) – a missing label string. Any None-valued labels are given this label for evaluation purposes

• backend (None) – a ClassificationEvaluation backend

Methods:

attributes()	Returns the list of class attributes that will be serialized by serialize().
base_results_cls(type)	Returns the results class for the given run type.
confusion_matrix([classes, include_other])	Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
from_dict(d, samples, config, key)	Builds a BaseRunResults from a JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
metrics([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
plot_confusion_matrix([classes, …])	Plots a confusion matrix for the evaluation results.
print_report([classes, digits])	Prints a classification report for the results via sklearn.metrics.classification_report().
report([classes])	Generates a classification report for the results via sklearn.metrics.classification_report().
save()	Saves the results to the database.
save_config()	Saves these results config to the database.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

backend	The BaseRun for these results.
cls	The fully-qualified name of this BaseRunResults class.
config	The BaseRunConfig for these results.
key	The run key for these results.
samples	The fiftyone.core.collections.SampleCollection associated with these results.

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The BaseRun for these results.

static base_results_cls(type)

Returns the results class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunResults subclass

property cls

The fully-qualified name of this BaseRunResults class.

property config

The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

• include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

• d – a JSON dict

• samples – the fiftyone.core.collections.SampleCollection for the run

• config – the BaseRunConfig for the run

• key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

• classes (None) – an optional list of classes to include in the calculations

• average ("micro") – the averaging strategy to use

• beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix

• include_other (None) –

  whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

  • None (default): only include a row/column for other labels if there are any

  • True: do include a row/column for other labels

  • False: do not include a row/column for other labels

• include_missing (None) –

  whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

  • None (default): only include a row/column for missing labels if there are any

  • True: do include a row/column for missing labels

  • False: do not include a row/column for missing labels

• other_label ("(other)") – the label to use for “other” predictions

• backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

• **kwargs –

  keyword arguments for the backend plotting method:

  • ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()

  • ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

• a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used

• a matplotlib figure, otherwise

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

• classes (None) – an optional list of classes to include in the report

• digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.classification.BinaryClassificationResults(samples, config, eval_key, ytrue, ypred, confs, classes, weights=None, ytrue_ids=None, ypred_ids=None, backend=None)

Bases: fiftyone.utils.eval.classification.ClassificationResults

Class that stores the results of a binary classification evaluation.

Any missing ground truth or prediction labels are assumed to be examples of the negative class (with zero confidence, for predictions).

Parameters

• samples – the fiftyone.core.collections.SampleCollection used

• config – the ClassificationEvaluationConfig used

• eval_key – the evaluation key

• ytrue – a list of ground truth labels

• ypred – a list of predicted labels

• confs – a list of confidences for the predictions

• classes – the (neg_label, pos_label) label strings for the task

• weights (None) – an optional list of sample weights

• ytrue_ids (None) – a list of IDs for the ground truth labels

• ypred_ids (None) – a list of IDs for the predicted labels

• backend (None) – a ClassificationEvaluation backend

Methods:

average_precision([average])	Computes the average precision for the results via sklearn.metrics.average_precision_score().
plot_pr_curve([average, backend])	Plots a precision-recall (PR) curve for the results.
plot_roc_curve([backend])	Plots a receiver operating characteristic (ROC) curve for the results.
attributes()	Returns the list of class attributes that will be serialized by serialize().
base_results_cls(type)	Returns the results class for the given run type.
confusion_matrix([classes, include_other])	Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
from_dict(d, samples, config, key)	Builds a BaseRunResults from a JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
metrics([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
plot_confusion_matrix([classes, …])	Plots a confusion matrix for the evaluation results.
print_report([classes, digits])	Prints a classification report for the results via sklearn.metrics.classification_report().
report([classes])	Generates a classification report for the results via sklearn.metrics.classification_report().
save()	Saves the results to the database.
save_config()	Saves these results config to the database.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

backend	The BaseRun for these results.
cls	The fully-qualified name of this BaseRunResults class.
config	The BaseRunConfig for these results.
key	The run key for these results.
samples	The fiftyone.core.collections.SampleCollection associated with these results.

average_precision(average='micro')

Computes the average precision for the results via sklearn.metrics.average_precision_score().

Parameters

average ("micro") – the averaging strategy to use

Returns

the average precision

plot_pr_curve(average='micro', backend='plotly', **kwargs)

Plots a precision-recall (PR) curve for the results.

Parameters

• average ("micro") – the averaging strategy to use when computing average precision

• backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

• **kwargs –

  keyword arguments for the backend plotting method:

  • ”plotly” backend: fiftyone.core.plots.plotly.plot_pr_curve()

  • ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_pr_curve()

Returns

• a fiftyone.core.plots.plotly.PlotlyNotebookPlot, if you are working in a notebook context and the plotly backend is used

• a plotly or matplotlib figure, otherwise

Return type

one of the following

plot_roc_curve(backend='plotly', **kwargs)

Plots a receiver operating characteristic (ROC) curve for the results.

Parameters

• backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

• **kwargs –

  keyword arguments for the backend plotting method:

  • ”plotly” backend: fiftyone.core.plots.plotly.plot_roc_curve()

  • ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_roc_curve()

Returns

• a fiftyone.core.plots.plotly.PlotlyNotebookPlot, if you are working in a notebook context and the plotly backend is used

• a plotly or matplotlib figure, otherwise

Return type

one of the following

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The BaseRun for these results.

static base_results_cls(type)

Returns the results class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunResults subclass

property cls

The fully-qualified name of this BaseRunResults class.

property config

The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

• include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

• d – a JSON dict

• samples – the fiftyone.core.collections.SampleCollection for the run

• config – the BaseRunConfig for the run

• key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

• classes (None) – an optional list of classes to include in the calculations

• average ("micro") – the averaging strategy to use

• beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix

• include_other (None) –

  whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

  • None (default): only include a row/column for other labels if there are any

  • True: do include a row/column for other labels

  • False: do not include a row/column for other labels

• include_missing (None) –

  whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

  • None (default): only include a row/column for missing labels if there are any

  • True: do include a row/column for missing labels

  • False: do not include a row/column for missing labels

• other_label ("(other)") – the label to use for “other” predictions

• backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

• **kwargs –

  keyword arguments for the backend plotting method:

  • ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()

  • ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

• a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used

• a matplotlib figure, otherwise

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

• classes (None) – an optional list of classes to include in the report

• digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()