Table of Contents
Contents

fiftyone.utils.eval

Module contents

Evaluation utilities.

Copyright 2017-2023, Voxel51, Inc.
voxel51.com

Functions:

evaluate_classifications(samples, pred_field)

Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.

evaluate_detections(samples, pred_field[, …])

Evaluates the predicted detections in the given samples with respect to the specified ground truth detections.

evaluate_regressions(samples, pred_field[, …])

Evaluates the regression predictions in the given collection with respect to the specified ground truth values.

evaluate_segmentations(samples, pred_field)

Evaluates the specified semantic segmentation masks in the given collection with respect to the specified ground truth masks.

Classes:

ClassificationResults(samples, config, …)

Class that stores the results of a classification evaluation.

BinaryClassificationResults(samples, config, …)

Class that stores the results of a binary classification evaluation.

DetectionResults(samples, config, eval_key, …)

Class that stores the results of a detection evaluation.

RegressionResults(samples, config, eval_key, …)

Class that stores the results of a regression evaluation.

SegmentationResults(samples, config, …[, …])

Class that stores the results of a segmentation evaluation.
fiftyone.utils.eval.evaluate_classifications(samples, pred_field, gt_field='ground_truth', eval_key=None, classes=None, missing=None, method='simple', **kwargs)

Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.

By default, this method simply compares the ground truth and prediction for each sample, but other strategies such as binary evaluation and top-k matching can be configured via the method parameter.

You can customize the evaluation method by passing additional parameters for the method’s config class as kwargs.

The supported method values and their associated configs are:

  • "simple": SimpleEvaluationConfig

  • "top-k": TopKEvaluationConfig

  • "binary": BinaryEvaluationConfig

If an eval_key is specified, then this method will record some statistics on each sample:

  • When evaluating sample-level fields, an eval_key field will be populated on each sample recording whether that sample’s prediction is correct.

  • When evaluating frame-level fields, an eval_key field will be populated on each frame recording whether that frame’s prediction is correct. In addition, an eval_key field will be populated on each sample that records the average accuracy of the frame predictions of the sample.

Parameters

  • samples – a fiftyone.core.collections.SampleCollection

  • pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances

  • gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Classification instances

  • eval_key (None) – an evaluation key to use to refer to this evaluation

  • classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

  • missing (None) – a missing label string. Any None-valued labels are given this label for results purposes

  • method ("simple") – a string specifying the evaluation method to use. Supported values are ("simple", "binary", "top-k")

  • **kwargs – optional keyword arguments for the constructor of the ClassificationEvaluationConfig being used

Returns

a ClassificationResults

class fiftyone.utils.eval.ClassificationResults(samples, config, eval_key, ytrue, ypred, confs=None, weights=None, ytrue_ids=None, ypred_ids=None, classes=None, missing=None, backend=None)

Bases: fiftyone.utils.eval.base.BaseEvaluationResults

Class that stores the results of a classification evaluation.

Parameters

  • samples – the fiftyone.core.collections.SampleCollection used

  • config – the ClassificationEvaluationConfig used

  • eval_key – the evaluation key

  • ytrue – a list of ground truth labels

  • ypred – a list of predicted labels

  • confs (None) – an optional list of confidences for the predictions

  • weights (None) – an optional list of sample weights

  • ytrue_ids (None) – a list of IDs for the ground truth labels

  • ypred_ids (None) – a list of IDs for the predicted labels

  • classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

  • missing (None) – a missing label string. Any None-valued labels are given this label for evaluation purposes

  • backend (None) – a ClassificationEvaluation backend

Methods:

attributes()

Returns the list of class attributes that will be serialized by serialize().

confusion_matrix([classes, include_other])

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

copy()

Returns a deep copy of the object.

custom_attributes([dynamic, private])

Returns a customizable list of class attributes.

from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

get_class_name()

Returns the fully-qualified class name string of this object.

metrics([classes, average, beta])

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

plot_confusion_matrix([classes, …])

Plots a confusion matrix for the evaluation results.

print_report([classes, digits])

Prints a classification report for the results via sklearn.metrics.classification_report().

report([classes])

Generates a classification report for the results via sklearn.metrics.classification_report().

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize([reflective])

Serializes the object into a dictionary.

to_str([pretty_print])

Returns a string representation of this object.

write_json(path[, pretty_print])

Serializes the object and writes it to disk.

Attributes:

backend

The Run for these results.

cls

The fully-qualified name of this RunResults class.

config

The RunConfig for these results.

key

The run key for these results.

samples

The fiftyone.core.collections.SampleCollection associated with these results.
attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The Run for these results.

property cls

The fully-qualified name of this RunResults class.

property config

The RunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

  • include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

  • dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

  • private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

Parameters
Returns

a RunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

  • path – the path to the JSON file on disk

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

  • s – a JSON string representation of a Serializable object

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

  • classes (None) – an optional list of classes to include in the calculations

  • average ("micro") – the averaging strategy to use

  • beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix

  • include_other (None) –

    whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

    • None (default): only include a row/column for other labels if there are any

    • True: do include a row/column for other labels

    • False: do not include a row/column for other labels

  • include_missing (None) –

    whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

    • None (default): only include a row/column for missing labels if there are any

    • True: do include a row/column for missing labels

    • False: do not include a row/column for missing labels

  • other_label ("(other)") – the label to use for “other” predictions

  • backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

  • **kwargs –

    keyword arguments for the backend plotting method:

Returns

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

  • classes (None) – an optional list of classes to include in the report

  • digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

  • **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

  • path – the output path

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

  • **kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.BinaryClassificationResults(samples, config, eval_key, ytrue, ypred, confs, classes, weights=None, ytrue_ids=None, ypred_ids=None, backend=None)

Bases: fiftyone.utils.eval.classification.ClassificationResults

Class that stores the results of a binary classification evaluation.

Any missing ground truth or prediction labels are assumed to be examples of the negative class (with zero confidence, for predictions).

Parameters

  • samples – the fiftyone.core.collections.SampleCollection used

  • config – the ClassificationEvaluationConfig used

  • eval_key – the evaluation key

  • ytrue – a list of ground truth labels

  • ypred – a list of predicted labels

  • confs – a list of confidences for the predictions

  • classes – the (neg_label, pos_label) label strings for the task

  • weights (None) – an optional list of sample weights

  • ytrue_ids (None) – a list of IDs for the ground truth labels

  • ypred_ids (None) – a list of IDs for the predicted labels

  • backend (None) – a ClassificationEvaluation backend

Methods:

attributes()

Returns the list of class attributes that will be serialized by serialize().

average_precision([average])

Computes the average precision for the results via sklearn.metrics.average_precision_score().

confusion_matrix([classes, include_other])

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

copy()

Returns a deep copy of the object.

custom_attributes([dynamic, private])

Returns a customizable list of class attributes.

from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

get_class_name()

Returns the fully-qualified class name string of this object.

metrics([classes, average, beta])

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

plot_confusion_matrix([classes, …])

Plots a confusion matrix for the evaluation results.

plot_pr_curve([average, backend])

Plots a precision-recall (PR) curve for the results.

plot_roc_curve([backend])

Plots a receiver operating characteristic (ROC) curve for the results.

print_report([classes, digits])

Prints a classification report for the results via sklearn.metrics.classification_report().

report([classes])

Generates a classification report for the results via sklearn.metrics.classification_report().

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize([reflective])

Serializes the object into a dictionary.

to_str([pretty_print])

Returns a string representation of this object.

write_json(path[, pretty_print])

Serializes the object and writes it to disk.

Attributes:

backend

The Run for these results.

cls

The fully-qualified name of this RunResults class.

config

The RunConfig for these results.

key

The run key for these results.

samples

The fiftyone.core.collections.SampleCollection associated with these results.
average_precision(average='micro')

Computes the average precision for the results via sklearn.metrics.average_precision_score().

Parameters

average ("micro") – the averaging strategy to use

Returns

the average precision

plot_pr_curve(average='micro', backend='plotly', **kwargs)

Plots a precision-recall (PR) curve for the results.

Parameters
Returns

Return type

one of the following

plot_roc_curve(backend='plotly', **kwargs)

Plots a receiver operating characteristic (ROC) curve for the results.

Parameters
Returns

Return type

one of the following

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The Run for these results.

property cls

The fully-qualified name of this RunResults class.

property config

The RunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

  • include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

  • dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

  • private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

Parameters
Returns

a RunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

  • path – the path to the JSON file on disk

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

  • s – a JSON string representation of a Serializable object

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

  • classes (None) – an optional list of classes to include in the calculations

  • average ("micro") – the averaging strategy to use

  • beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix

  • include_other (None) –

    whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

    • None (default): only include a row/column for other labels if there are any

    • True: do include a row/column for other labels

    • False: do not include a row/column for other labels

  • include_missing (None) –

    whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

    • None (default): only include a row/column for missing labels if there are any

    • True: do include a row/column for missing labels

    • False: do not include a row/column for missing labels

  • other_label ("(other)") – the label to use for “other” predictions

  • backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

  • **kwargs –

    keyword arguments for the backend plotting method:

Returns

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

  • classes (None) – an optional list of classes to include in the report

  • digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

  • **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

  • path – the output path

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

  • **kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_detections(samples, pred_field, gt_field='ground_truth', eval_key=None, classes=None, missing=None, method=None, iou=0.5, use_masks=False, use_boxes=False, classwise=True, dynamic=True, **kwargs)

Evaluates the predicted detections in the given samples with respect to the specified ground truth detections.

This method supports evaluating the following spatial data types:

For spatial object detection evaluation, this method uses COCO-style evaluation by default.

When evaluating keypoints, “IoUs” are computed via object keypoint similarity.

For temporal segment detection, this method uses ActivityNet-style evaluation by default.

You can use the method parameter to select a different method, and you can optionally customize the method by passing additional parameters for the method’s config class as kwargs.

The supported method values and their associated configs are:

If an eval_key is provided, a number of fields are populated at the object- and sample-level recording the results of the evaluation:

  • True positive (TP), false positive (FP), and false negative (FN) counts for each sample are saved in top-level fields of each sample:

    TP: sample.<eval_key>_tp
FP: sample.<eval_key>_fp
FN: sample.<eval_key>_fn

    In addition, when evaluating frame-level objects, TP/FP/FN counts are recorded for each frame:

    TP: frame.<eval_key>_tp
FP: frame.<eval_key>_fp
FN: frame.<eval_key>_fn

  • The fields listed below are populated on each individual object; these fields tabulate the TP/FP/FN status of the object, the ID of the matching object (if any), and the matching IoU:

    TP/FP/FN: object.<eval_key>
      ID: object.<eval_key>_id
     IoU: object.<eval_key>_iou
Parameters

  • samples – a fiftyone.core.collections.SampleCollection

  • pred_field – the name of the field containing the predicted fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, fiftyone.core.labels.Keypoints, or fiftyone.core.labels.TemporalDetections

  • gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, fiftyone.core.labels.Keypoints, or fiftyone.core.labels.TemporalDetections

  • eval_key (None) – an evaluation key to use to refer to this evaluation

  • classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

  • missing (None) – a missing label string. Any unmatched objects are given this label for results purposes

  • method (None) – a string specifying the evaluation method to use. For spatial object detection, the supported values are ("coco", "open-images") and the default is "coco". For temporal segment detection, the supported values are ("activitynet") and the default is "activitynet"

  • iou (0.50) – the IoU threshold to use to determine matches

  • use_masks (False) – whether to compute IoUs using the instances masks in the mask attribute of the provided objects, which must be fiftyone.core.labels.Detection instances

  • use_boxes (False) – whether to compute IoUs using the bounding boxes of the provided fiftyone.core.labels.Polyline instances rather than using their actual geometries

  • classwise (True) – whether to only match objects with the same class label (True) or allow matches between classes (False)

  • dynamic (True) – whether to declare the dynamic object-level attributes that are populated on the dataset’s schema

  • **kwargs – optional keyword arguments for the constructor of the DetectionEvaluationConfig being used

Returns

a DetectionResults

class fiftyone.utils.eval.DetectionResults(samples, config, eval_key, matches, classes=None, missing=None, backend=None)

Bases: fiftyone.utils.eval.base.BaseEvaluationResults

Class that stores the results of a detection evaluation.

Parameters

  • samples – the fiftyone.core.collections.SampleCollection used

  • config – the DetectionEvaluationConfig used

  • eval_key – the evaluation key

  • matches – a list of (gt_label, pred_label, iou, pred_confidence, gt_id, pred_id) matches. Either label can be None to indicate an unmatched object

  • classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used

  • missing (None) – a missing label string. Any unmatched objects are given this label for evaluation purposes

  • backend (None) – a DetectionEvaluation backend

Methods:

attributes()

Returns the list of class attributes that will be serialized by serialize().

confusion_matrix([classes, include_other])

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

copy()

Returns a deep copy of the object.

custom_attributes([dynamic, private])

Returns a customizable list of class attributes.

from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

get_class_name()

Returns the fully-qualified class name string of this object.

metrics([classes, average, beta])

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

plot_confusion_matrix([classes, …])

Plots a confusion matrix for the evaluation results.

print_report([classes, digits])

Prints a classification report for the results via sklearn.metrics.classification_report().

report([classes])

Generates a classification report for the results via sklearn.metrics.classification_report().

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize([reflective])

Serializes the object into a dictionary.

to_str([pretty_print])

Returns a string representation of this object.

write_json(path[, pretty_print])

Serializes the object and writes it to disk.

Attributes:

backend

The Run for these results.

cls

The fully-qualified name of this RunResults class.

config

The RunConfig for these results.

key

The run key for these results.

samples

The fiftyone.core.collections.SampleCollection associated with these results.
attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The Run for these results.

property cls

The fully-qualified name of this RunResults class.

property config

The RunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

  • include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

  • dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

  • private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

Parameters
Returns

a RunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

  • path – the path to the JSON file on disk

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

  • s – a JSON string representation of a Serializable object

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

  • classes (None) – an optional list of classes to include in the calculations

  • average ("micro") – the averaging strategy to use

  • beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix

  • include_other (None) –

    whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

    • None (default): only include a row/column for other labels if there are any

    • True: do include a row/column for other labels

    • False: do not include a row/column for other labels

  • include_missing (None) –

    whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

    • None (default): only include a row/column for missing labels if there are any

    • True: do include a row/column for missing labels

    • False: do not include a row/column for missing labels

  • other_label ("(other)") – the label to use for “other” predictions

  • backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

  • **kwargs –

    keyword arguments for the backend plotting method:

Returns

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

  • classes (None) – an optional list of classes to include in the report

  • digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

  • **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

  • path – the output path

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

  • **kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_regressions(samples, pred_field, gt_field='ground_truth', eval_key=None, missing=None, method='simple', **kwargs)

Evaluates the regression predictions in the given collection with respect to the specified ground truth values.

You can customize the evaluation method by passing additional parameters for the method’s config class as kwargs.

The supported method values and their associated configs are:

  • "simple": SimpleEvaluationConfig

If an eval_key is specified, then this method will record some statistics on each sample:

  • When evaluating sample-level fields, an eval_key field will be populated on each sample recording the error of that sample’s prediction.

  • When evaluating frame-level fields, an eval_key field will be populated on each frame recording the error of that frame’s prediction. In addition, an eval_key field will be populated on each sample that records the average error of the frame predictions of the sample.

Parameters

  • samples – a fiftyone.core.collections.SampleCollection

  • pred_field – the name of the field containing the predicted fiftyone.core.labels.Regression instances

  • gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Regression instances

  • eval_key (None) – a string key to use to refer to this evaluation

  • missing (None) – a missing value. Any None-valued regressions are given this value for results purposes

  • method ("simple") – a string specifying the evaluation method to use. Supported values are ("simple")

  • **kwargs – optional keyword arguments for the constructor of the RegressionEvaluationConfig being used

Returns

a RegressionResults

class fiftyone.utils.eval.RegressionResults(samples, config, eval_key, ytrue, ypred, confs=None, ids=None, missing=None, backend=None)

Bases: fiftyone.core.evaluation.EvaluationResults

Class that stores the results of a regression evaluation.

Parameters

  • samples – the fiftyone.core.collections.SampleCollection used

  • config – the RegressionEvaluationConfig used

  • eval_key (None) – the evaluation key

  • ytrue – a list of ground truth values

  • ypred – a list of predicted values

  • confs (None) – an optional list of confidences for the predictions

  • eval_key – the evaluation key of the evaluation

  • gt_field (None) – the name of the ground truth field

  • pred_field (None) – the name of the predictions field

  • ids (None) – a list of sample or frame IDs corresponding to the regressions

  • missing (None) – a missing value. Any None-valued regressions are given this value for results purposes

  • backend (None) – a RegressionEvaluation backend

Methods:

attributes()

Returns the list of class attributes that will be serialized by serialize().

copy()

Returns a deep copy of the object.

custom_attributes([dynamic, private])

Returns a customizable list of class attributes.

from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

get_class_name()

Returns the fully-qualified class name string of this object.

metrics([weights])

Computes various popular regression metrics for the results.

plot_results([labels, sizes, backend])

Plots the regression results.

print_metrics([weights, digits])

Prints the regression metrics computed via metrics().

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize([reflective])

Serializes the object into a dictionary.

to_str([pretty_print])

Returns a string representation of this object.

write_json(path[, pretty_print])

Serializes the object and writes it to disk.

Attributes:

backend

The Run for these results.

cls

The fully-qualified name of this RunResults class.

config

The RunConfig for these results.

key

The run key for these results.

samples

The fiftyone.core.collections.SampleCollection associated with these results.
metrics(weights=None)

Computes various popular regression metrics for the results.

The computed metrics are:

Parameters

weights (None) – an optional list of weights for each example

Returns

a dict

print_metrics(weights=None, digits=2)

Prints the regression metrics computed via metrics().

Parameters

  • weights (None) – an optional list of weights for each example

  • digits (2) – the number of digits of precision to print

plot_results(labels=None, sizes=None, backend='plotly', **kwargs)

Plots the regression results.

You can use the labels parameters to define a coloring for the points, and you can use the sizes parameter to scale the sizes of the points.

You can attach plots generated by this method to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected points in the plot.

Parameters
Returns

an fiftyone.core.plots.base.InteractivePlot

attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

property backend

The Run for these results.

property cls

The fully-qualified name of this RunResults class.

property config

The RunConfig for these results.

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

  • dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

  • private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

Parameters
Returns

a RunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

  • path – the path to the JSON file on disk

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

  • s – a JSON string representation of a Serializable object

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

  • **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

  • path – the output path

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

  • **kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_segmentations(samples, pred_field, gt_field='ground_truth', eval_key=None, mask_targets=None, method='simple', **kwargs)

Evaluates the specified semantic segmentation masks in the given collection with respect to the specified ground truth masks.

If the size of a predicted mask does not match the ground truth mask, it is resized to match the ground truth.

By default, this method simply performs pixelwise evaluation of the full masks, but other strategies such as boundary-only evaluation can be configured by passing additional parameters for the method’s config class as kwargs.

The supported method values and their associated configs are:

  • "simple": SimpleEvaluationConfig

If an eval_key is provided, the accuracy, precision, and recall of each sample is recorded in top-level fields of each sample:

 Accuracy: sample.<eval_key>_accuracy
Precision: sample.<eval_key>_precision
   Recall: sample.<eval_key>_recall

In addition, when evaluating frame-level masks, the accuracy, precision, and recall of each frame if recorded in the following frame-level fields:

 Accuracy: frame.<eval_key>_accuracy
Precision: frame.<eval_key>_precision
   Recall: frame.<eval_key>_recall

Note

The mask values 0 and #000000 are treated as a background class for the purposes of computing evaluation metrics like precision and recall.

Parameters

  • samples – a fiftyone.core.collections.SampleCollection

  • pred_field – the name of the field containing the predicted fiftyone.core.labels.Segmentation instances

  • gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Segmentation instances

  • eval_key (None) – an evaluation key to use to refer to this evaluation

  • mask_targets (None) – a dict mapping pixel values or RGB hex strings to labels. If not provided, the observed values are used as labels

  • method ("simple") – a string specifying the evaluation method to use. Supported values are ("simple")

  • **kwargs – optional keyword arguments for the constructor of the SegmentationEvaluationConfig being used

Returns

a SegmentationResults

class fiftyone.utils.eval.SegmentationResults(samples, config, eval_key, pixel_confusion_matrix, classes, missing=None, backend=None)

Bases: fiftyone.utils.eval.base.BaseEvaluationResults

Class that stores the results of a segmentation evaluation.

Parameters

  • samples – the fiftyone.core.collections.SampleCollection used

  • config – the SegmentationEvaluationConfig used

  • eval_key – the evaluation key

  • pixel_confusion_matrix – a pixel value confusion matrix

  • classes – a list of class labels corresponding to the confusion matrix

  • missing (None) – a missing (background) class

  • backend (None) – a SegmentationEvaluation backend

Methods:

attributes()

Returns the list of class attributes that will be serialized by serialize().

confusion_matrix([classes, include_other])

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

copy()

Returns a deep copy of the object.

custom_attributes([dynamic, private])

Returns a customizable list of class attributes.

dice_score()

Computes the Dice score across all samples in the evaluation.

from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

get_class_name()

Returns the fully-qualified class name string of this object.

metrics([classes, average, beta])

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

plot_confusion_matrix([classes, …])

Plots a confusion matrix for the evaluation results.

print_report([classes, digits])

Prints a classification report for the results via sklearn.metrics.classification_report().

report([classes])

Generates a classification report for the results via sklearn.metrics.classification_report().

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize([reflective])

Serializes the object into a dictionary.

to_str([pretty_print])

Returns a string representation of this object.

write_json(path[, pretty_print])

Serializes the object and writes it to disk.

Attributes:

backend

The Run for these results.

cls

The fully-qualified name of this RunResults class.

config

The RunConfig for these results.

key

The run key for these results.

samples

The fiftyone.core.collections.SampleCollection associated with these results.
attributes()

Returns the list of class attributes that will be serialized by serialize().

Returns

a list of attributes

dice_score()

Computes the Dice score across all samples in the evaluation.

Returns

the Dice score in [0, 1]

property backend

The Run for these results.

property cls

The fully-qualified name of this RunResults class.

property config

The RunConfig for these results.

confusion_matrix(classes=None, include_other=False)

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix. Include self.missing in this list if you would like to include a row/column for unmatched examples

  • include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided

Returns

a num_classes x num_classes confusion matrix

copy()

Returns a deep copy of the object.

Returns

a Serializable instance

custom_attributes(dynamic=False, private=False)

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

  • dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False

  • private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)

Builds a RunResults from a JSON dict representation of it.

Parameters
Returns

a RunResults

classmethod from_json(path, *args, **kwargs)

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

  • path – the path to the JSON file on disk

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

  • s – a JSON string representation of a Serializable object

  • *args – optional positional arguments for self.from_dict()

  • **kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()

Returns the fully-qualified class name string of this object.

property key

The run key for these results.

metrics(classes=None, average='micro', beta=1.0)

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Parameters

  • classes (None) – an optional list of classes to include in the calculations

  • average ("micro") – the averaging strategy to use

  • beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

  • classes (None) – an optional list of classes to include in the confusion matrix

  • include_other (None) –

    whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:

    • None (default): only include a row/column for other labels if there are any

    • True: do include a row/column for other labels

    • False: do not include a row/column for other labels

  • include_missing (None) –

    whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:

    • None (default): only include a row/column for missing labels if there are any

    • True: do include a row/column for missing labels

    • False: do not include a row/column for missing labels

  • other_label ("(other)") – the label to use for “other” predictions

  • backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")

  • **kwargs –

    keyword arguments for the backend plotting method:

Returns

Return type

one of the following

print_report(classes=None, digits=2)

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

  • classes (None) – an optional list of classes to include in the report

  • digits (2) – the number of digits of precision to print

report(classes=None)

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report

Returns

a dict

property samples

The fiftyone.core.collections.SampleCollection associated with these results.

save()

Saves the results to the database.

save_config()

Saves these results config to the database.

serialize(reflective=False)

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters

reflective – whether to include reflective attributes when serializing the object. By default, this is False

Returns

a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)

Returns a string representation of this object.

Parameters

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True

  • **kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)

Serializes the object and writes it to disk.

Parameters

  • path – the output path

  • pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False

  • **kwargs – optional keyword arguments for self.serialize()