fiftyone.utils.eval.base

Base evaluation methods.

Copyright 2017-2024, Voxel51, Inc.

voxel51.com

Classes:

BaseEvaluationResults(samples, config, …)

Base class for evaluation results.

class fiftyone.utils.eval.base.BaseEvaluationResults(samples, config, eval_key, ytrue, ypred, confs=None, weights=None, ytrue_ids=None, ypred_ids=None, classes=None, missing=None, backend=None)

Bases: fiftyone.core.evaluation.EvaluationResults

Base class for evaluation results.

Parameters

• samples (None) – the fiftyone.core.collections.SampleCollection used

• config – the fiftyone.core.evaluation.EvaluationMethodConfig used

• eval_key – the evaluation key

• ytrue – a list of ground truth labels

• ypred – a list of predicted labels

• confs (None) – an optional list of confidences for the predictions

• weights (None) – an optional list of sample weights

• ytrue_ids (None) – a list of IDs for the ground truth labels

• ypred_ids (None) – a list of IDs for the predicted labels

• classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

• missing (None) – a missing label string. Any None-valued labels are given this label for evaluation purposes

• samples – the fiftyone.core.collections.SampleCollection for which the results were computed

• backend (None) – a fiftyone.core.evaluation.EvaluationMethod backend

Methods:

report([classes])	Generates a classification report for the results via sklearn.metrics.classification_report().
metrics([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
print_report([classes, digits])	Prints a classification report for the results via sklearn.metrics.classification_report().
confusion_matrix([classes, include_other])	Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().
plot_confusion_matrix([classes, …])	Plots a confusion matrix for the evaluation results.
attributes()	Returns the list of class attributes that will be serialized by serialize().
base_results_cls(type)	Returns the results class for the given run type.
copy()	Returns a deep copy of the object.
custom_attributes([dynamic, private])	Returns a customizable list of class attributes.
from_dict(d, samples, config, key)	Builds a BaseRunResults from a JSON dict representation of it.
from_json(path, *args, **kwargs)	Constructs a Serializable object from a JSON file.
from_str(s, *args, **kwargs)	Constructs a Serializable object from a JSON string.
get_class_name()	Returns the fully-qualified class name string of this object.
save()	Saves the results to the database.
save_config()	Saves these results config to the database.
serialize([reflective])	Serializes the object into a dictionary.
to_str([pretty_print])	Returns a string representation of this object.
write_json(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

backend	The BaseRun for these results.
cls	The fully-qualified name of this BaseRunResults class.
config	The BaseRunConfig for these results.
key	The run key for these results.
samples	The fiftyone.core.collections.SampleCollection associated with these results.

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

• classes (None) – an optional list of classes to include in the calculations

• average ("micro") – the averaging strategy to use

• beta (1.0) – the F-beta value to use

Returns

a dict

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

• classes (None) – an optional list of classes to include in the report

• digits (2) – the number of digits of precision to print

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

• include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

• classes (None) – an optional list of classes to include in the confusion matrix

• include_other (None) –

  whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

  • None (default): only include a row/column for other labels if there are any

  • True: do include a row/column for other labels

  • False: do not include a row/column for other labels

• include_missing (None) –

  whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

  • None (default): only include a row/column for missing labels if there are any

  • True: do include a row/column for missing labels

  • False: do not include a row/column for missing labels

• other_label ("(other)") – the label to use for “other” predictions

• backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

• **kwargs –

  keyword arguments for the backend plotting method:

  • ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()

  • ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

• a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used

• a matplotlib figure, otherwise

Return type

one of the following

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The BaseRun for these results.

static base_results_cls(type)

Returns the results class for the given run type.

Parameters

type – a BaseRunConfig.type

Returns

a BaseRunResults subclass

property cls

The fully-qualified name of this BaseRunResults class.

property config

The BaseRunConfig for these results.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

• dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

• private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

• d – a JSON dict

• samples – the fiftyone.core.collections.SampleCollection for the run

• config – the BaseRunConfig for the run

• key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

• path – the path to the JSON file on disk

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

• s – a JSON string representation of a Serializable object

• *args – optional positional arguments for self.from_dict()

• **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

• **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

• path – the output path

• pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

• **kwargs – optional keyword arguments for self.serialize()