fiftyone.utils.eval¶

Module contents¶

Evaluation utilities.

Copyright 2017-2025, Voxel51, Inc.
voxel51.com

Functions:

`evaluate_classifications`(samples, pred_field)	Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.
`evaluate_detections`(samples, pred_field[, …])	Evaluates the predicted detections in the given samples with respect to the specified ground truth detections.
`evaluate_regressions`(samples, pred_field[, …])	Evaluates the regression predictions in the given collection with respect to the specified ground truth values.
`evaluate_segmentations`(samples, pred_field)	Evaluates the specified semantic segmentation masks in the given collection with respect to the specified ground truth masks.

Classes:

`ClassificationEvaluationConfig`(pred_field, …)	Base class for configuring `ClassificationEvaluation` instances.
`ClassificationResults`(samples, config, …)	Class that stores the results of a classification evaluation.
`DetectionEvaluationConfig`(pred_field, gt_field)	Base class for configuring `DetectionEvaluation` instances.
`DetectionResults`(samples, config, eval_key, …)	Class that stores the results of a detection evaluation.
`RegressionEvaluationConfig`(pred_field, gt_field)	Base class for configuring `RegressionEvaluation` instances.
`RegressionResults`(samples, config, eval_key, …)	Class that stores the results of a regression evaluation.
`SegmentationEvaluationConfig`(pred_field, …)	Base class for configuring `SegmentationEvaluation` instances.
`SegmentationResults`(samples, config, …[, …])	Class that stores the results of a segmentation evaluation.

fiftyone.utils.eval.evaluate_classifications(samples, pred_field, gt_field='ground_truth', eval_key=None, classes=None, missing=None, method=None, custom_metrics=None, progress=None, **kwargs)¶

Evaluates the classification predictions in the given collection with respect to the specified ground truth labels.

By default, this method simply compares the ground truth and prediction for each sample, but other strategies such as binary evaluation and top-k matching can be configured via the method parameter.

You can customize the evaluation method by passing additional parameters for the method’s config class as kwargs.

The natively provided method values and their associated configs are:

"simple": SimpleEvaluationConfig
"top-k": TopKEvaluationConfig
"binary": BinaryEvaluationConfig

If an eval_key is specified, then this method will record some statistics on each sample:

When evaluating sample-level fields, an eval_key field will be populated on each sample recording whether that sample’s prediction is correct.
When evaluating frame-level fields, an eval_key field will be populated on each frame recording whether that frame’s prediction is correct. In addition, an eval_key field will be populated on each sample that records the average accuracy of the frame predictions of the sample.

Parameters

samples – a fiftyone.core.collections.SampleCollection
pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances
gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Classification instances
eval_key (None) – an evaluation key to use to refer to this evaluation
classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used
missing (None) – a missing label string. Any None-valued labels are given this label for results purposes
method (None) – a string specifying the evaluation method to use. The supported values are fo.evaluation_config.classification_backends.keys() and the default is fo.evaluation_config.default_classification_backend
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional keyword arguments for the constructor of the ClassificationEvaluationConfig being used

Returns

a ClassificationResults

class fiftyone.utils.eval.ClassificationEvaluationConfig(pred_field, gt_field, custom_metrics=None, **kwargs)¶

Bases: fiftyone.utils.eval.base.BaseEvaluationMethodConfig

Base class for configuring ClassificationEvaluation instances.

Parameters

pred_field – the name of the field containing the predicted fiftyone.core.labels.Classification instances
gt_field – the name of the field containing the ground truth fiftyone.core.labels.Classification instances
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts

Attributes:

`type`	The type of run.
`cls`	The fully-qualified name of this `BaseRunConfig` class.
`method`	The name of the method.
`run_cls`	The `BaseRun` class associated with this config.

Methods:

`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_config_cls`(type)	Returns the config class for the given run type.
`build`()	Builds the `BaseRun` instance associated with this config.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a `BaseRunConfig` from a serialized JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_credentials`(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

property type¶: The type of run.

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

static base_config_cls(type)¶

Returns the config class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunConfig subclass

build()¶

Builds the BaseRun instance associated with this config.

Returns: a BaseRun instance

classmethod builder()¶: Returns a ConfigBuilder instance for this class.

property cls¶: The fully-qualified name of this BaseRunConfig class.

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()¶

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)¶

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters: d – a JSON dict
Returns: a BaseRunConfig

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)¶

Constructs a Config object from keyword arguments.

Parameters: **kwargs – keyword arguments that define the fields expected by cls
Returns: an instance of cls

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)¶

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters: **kwargs – subclass-specific credentials

classmethod load_default()¶

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

property method¶: The name of the method.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw array attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a boolean value.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)¶

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)¶

Parses an integer attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)¶

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Returns: the (field, value) that was set
Raises: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a number attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw (arbitrary) JSON field.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a string attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls¶: The BaseRun class associated with this config.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

static validate_all_or_nothing_fields(fields)¶

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Raises: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.ClassificationResults(samples, config, eval_key, ytrue, ypred, confs=None, weights=None, ytrue_ids=None, ypred_ids=None, classes=None, missing=None, custom_metrics=None, backend=None)¶

Bases: fiftyone.utils.eval.base.BaseClassificationResults

Class that stores the results of a classification evaluation.

Parameters

samples – the fiftyone.core.collections.SampleCollection used
config – the ClassificationEvaluationConfig used
eval_key – the evaluation key
ytrue – a list of ground truth labels
ypred – a list of predicted labels
confs (None) – an optional list of confidences for the predictions
weights (None) – an optional list of sample weights
ytrue_ids (None) – a list of IDs for the ground truth labels
ypred_ids (None) – a list of IDs for the predicted labels
classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used
missing (None) – a missing label string. Any None-valued labels are given this label for evaluation purposes
custom_metrics (None) – an optional dict of custom metrics
backend (None) – a ClassificationEvaluation backend

Methods:

`add_custom_metrics`(custom_metrics[, overwrite])	Computes the given custom metrics and adds them to these results.
`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_results_cls`(type)	Returns the results class for the given run type.
`confusion_matrix`([classes, include_other, …])	Generates a confusion matrix for the results via `sklearn.metrics.confusion_matrix()`.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`from_dict`(d, samples, config, key)	Builds a `BaseRunResults` from a JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`metrics`([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
`plot_confusion_matrix`([classes, …])	Plots a confusion matrix for the evaluation results.
`print_metrics`([classes, average, beta, digits])	Prints the metrics computed via `metrics()`.
`print_report`([classes, digits])	Prints a classification report for the results via `sklearn.metrics.classification_report()`.
`report`([classes])	Generates a classification report for the results via `sklearn.metrics.classification_report()`.
`save`()	Saves the results to the database.
`save_config`()	Saves these results config to the database.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

`backend`	The `BaseRun` for these results.
`cls`	The fully-qualified name of this `BaseRunResults` class.
`config`	The `BaseRunConfig` for these results.
`key`	The run key for these results.
`samples`	The `fiftyone.core.collections.SampleCollection` associated with these results.

add_custom_metrics(custom_metrics, overwrite=True)¶

Computes the given custom metrics and adds them to these results.

Parameters

custom_metrics – a list of custom metrics to compute or a dict mapping metric names to kwargs dicts
overwrite (True) – whether to recompute any custom metrics that have already been applied

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

property backend¶: The BaseRun for these results.

static base_results_cls(type)¶

Returns the results class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunResults subclass

property cls¶: The fully-qualified name of this BaseRunResults class.

property config¶: The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False, include_missing=False)¶

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided
include_missing (False) – whether to include a row/column at the end of the matrix for unmatched labels. Only applicable if self.missing does not already appear in classes. If both “other” and “missing” rows/columns are requested, this one is last

Returns

a num_classes x num_classes confusion matrix

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)¶

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

d – a JSON dict
samples – the fiftyone.core.collections.SampleCollection for the run
config – the BaseRunConfig for the run
key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

property key¶: The run key for these results.

metrics(classes=None, average='micro', beta=1.0)¶

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Also includes any custom metrics from custom_metrics.

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)¶

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (None) –
whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:
- None (default): only include a row/column for other labels if there are any
- True: do include a row/column for other labels
- False: do not include a row/column for other labels
include_missing (None) –
whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:
- None (default): only include a row/column for missing labels if there are any
- True: do include a row/column for missing labels
- False: do not include a row/column for missing labels
other_label ("(other)") – the label to use for “other” predictions
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used
a matplotlib figure, otherwise

Return type

one of the following

print_metrics(classes=None, average='micro', beta=1.0, digits=2)¶

Prints the metrics computed via metrics().

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use
digits (2) – the number of digits of precision to print

print_report(classes=None, digits=2)¶

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report
digits (2) – the number of digits of precision to print

report(classes=None)¶

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters: classes (None) – an optional list of classes to include in the report
Returns: a dict

property samples¶: The fiftyone.core.collections.SampleCollection associated with these results.

save()¶: Saves the results to the database.

save_config()¶: Saves these results config to the database.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_detections(samples, pred_field, gt_field='ground_truth', eval_key=None, classes=None, missing=None, method=None, iou=0.5, use_masks=False, use_boxes=False, classwise=True, dynamic=True, custom_metrics=None, progress=None, **kwargs)¶

Evaluates the predicted detections in the given samples with respect to the specified ground truth detections.

This method supports evaluating the following spatial data types:

Object detections in fiftyone.core.labels.Detections format
Instance segmentations in fiftyone.core.labels.Detections format with their mask attributes populated
Polygons in fiftyone.core.labels.Polylines format
Keypoints in fiftyone.core.labels.Keypoints format
Temporal detections in fiftyone.core.labels.TemporalDetections format

For spatial object detection evaluation, this method uses COCO-style evaluation by default.

When evaluating keypoints, “IoUs” are computed via object keypoint similarity.

For temporal segment detection, this method uses ActivityNet-style evaluation by default.

You can use the method parameter to select a different method, and you can optionally customize the method by passing additional parameters for the method’s config class as kwargs.

The natively provided method values and their associated configs are:

"coco": fiftyone.utils.eval.coco.COCOEvaluationConfig
"open-images": fiftyone.utils.eval.openimages.OpenImagesEvaluationConfig
"activitynet": fiftyone.utils.eval.activitynet.ActivityNetEvaluationConfig

If an eval_key is provided, a number of fields are populated at the object- and sample-level recording the results of the evaluation:

True positive (TP), false positive (FP), and false negative (FN) counts for each sample are saved in top-level fields of each sample:
```
TP: sample.<eval_key>_tp
FP: sample.<eval_key>_fp
FN: sample.<eval_key>_fn
```
In addition, when evaluating frame-level objects, TP/FP/FN counts are recorded for each frame:
```
TP: frame.<eval_key>_tp
FP: frame.<eval_key>_fp
FN: frame.<eval_key>_fn
```
The fields listed below are populated on each individual object; these fields tabulate the TP/FP/FN status of the object, the ID of the matching object (if any), and the matching IoU:
```
TP/FP/FN: object.<eval_key>
      ID: object.<eval_key>_id
     IoU: object.<eval_key>_iou
```

Parameters

samples – a fiftyone.core.collections.SampleCollection
pred_field – the name of the field containing the predicted fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, fiftyone.core.labels.Keypoints, or fiftyone.core.labels.TemporalDetections
gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, fiftyone.core.labels.Keypoints, or fiftyone.core.labels.TemporalDetections
eval_key (None) – an evaluation key to use to refer to this evaluation
classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used
missing (None) – a missing label string. Any unmatched objects are given this label for results purposes
method (None) – a string specifying the evaluation method to use. The supported values are fo.evaluation_config.detection_backends.keys() and the default is fo.evaluation_config.default_detection_backend
iou (0.50) – the IoU threshold to use to determine matches
use_masks (False) – whether to compute IoUs using the instances masks in the mask attribute of the provided objects, which must be fiftyone.core.labels.Detection instances
use_boxes (False) – whether to compute IoUs using the bounding boxes of the provided fiftyone.core.labels.Polyline instances rather than using their actual geometries
classwise (True) – whether to only match objects with the same class label (True) or allow matches between classes (False)
dynamic (True) – whether to declare the dynamic object-level attributes that are populated on the dataset’s schema
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional keyword arguments for the constructor of the DetectionEvaluationConfig being used

Returns

a DetectionResults

class fiftyone.utils.eval.DetectionEvaluationConfig(pred_field, gt_field, iou=None, classwise=None, custom_metrics=None, **kwargs)¶

Bases: fiftyone.utils.eval.base.BaseEvaluationMethodConfig

Base class for configuring DetectionEvaluation instances.

Parameters

pred_field – the name of the field containing the predicted fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, or fiftyone.core.labels.Keypoints
gt_field – the name of the field containing the ground truth fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, or fiftyone.core.labels.Keypoints
iou (None) – the IoU threshold to use to determine matches
classwise (None) – whether to only match objects with the same class label (True) or allow matches between classes (False)
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts

Attributes:

`type`	The type of run.
`requires_additional_fields`	Whether fields besides `pred_field` and `gt_field` are required in order to perform evaluation.
`cls`	The fully-qualified name of this `BaseRunConfig` class.
`method`	The name of the method.
`run_cls`	The `BaseRun` class associated with this config.

Methods:

`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_config_cls`(type)	Returns the config class for the given run type.
`build`()	Builds the `BaseRun` instance associated with this config.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a `BaseRunConfig` from a serialized JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_credentials`(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

property type¶: The type of run.

property requires_additional_fields¶

Whether fields besides pred_field and gt_field are required in order to perform evaluation.

If True then the entire samples will be loaded rather than using select_fields() to optimize.

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

static base_config_cls(type)¶

Returns the config class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunConfig subclass

build()¶

Builds the BaseRun instance associated with this config.

Returns: a BaseRun instance

classmethod builder()¶: Returns a ConfigBuilder instance for this class.

property cls¶: The fully-qualified name of this BaseRunConfig class.

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()¶

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)¶

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters: d – a JSON dict
Returns: a BaseRunConfig

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)¶

Constructs a Config object from keyword arguments.

Parameters: **kwargs – keyword arguments that define the fields expected by cls
Returns: an instance of cls

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)¶

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters: **kwargs – subclass-specific credentials

classmethod load_default()¶

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

property method¶: The name of the method.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw array attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a boolean value.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)¶

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)¶

Parses an integer attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)¶

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Returns: the (field, value) that was set
Raises: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a number attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw (arbitrary) JSON field.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a string attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls¶: The BaseRun class associated with this config.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

static validate_all_or_nothing_fields(fields)¶

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Raises: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.DetectionResults(samples, config, eval_key, matches, classes=None, missing=None, custom_metrics=None, backend=None)¶

Bases: fiftyone.utils.eval.base.BaseClassificationResults

Class that stores the results of a detection evaluation.

Parameters

samples – the fiftyone.core.collections.SampleCollection used
config – the DetectionEvaluationConfig used
eval_key – the evaluation key
matches – a list of (gt_label, pred_label, iou, pred_confidence, gt_id, pred_id) matches. Either label can be None to indicate an unmatched object
classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used
missing (None) – a missing label string. Any unmatched objects are given this label for evaluation purposes
custom_metrics (None) – an optional dict of custom metrics
backend (None) – a DetectionEvaluation backend

Methods:

`add_custom_metrics`(custom_metrics[, overwrite])	Computes the given custom metrics and adds them to these results.
`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_results_cls`(type)	Returns the results class for the given run type.
`confusion_matrix`([classes, include_other, …])	Generates a confusion matrix for the results via `sklearn.metrics.confusion_matrix()`.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`from_dict`(d, samples, config, key)	Builds a `BaseRunResults` from a JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`metrics`([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
`plot_confusion_matrix`([classes, …])	Plots a confusion matrix for the evaluation results.
`print_metrics`([classes, average, beta, digits])	Prints the metrics computed via `metrics()`.
`print_report`([classes, digits])	Prints a classification report for the results via `sklearn.metrics.classification_report()`.
`report`([classes])	Generates a classification report for the results via `sklearn.metrics.classification_report()`.
`save`()	Saves the results to the database.
`save_config`()	Saves these results config to the database.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

`backend`	The `BaseRun` for these results.
`cls`	The fully-qualified name of this `BaseRunResults` class.
`config`	The `BaseRunConfig` for these results.
`key`	The run key for these results.
`samples`	The `fiftyone.core.collections.SampleCollection` associated with these results.

add_custom_metrics(custom_metrics, overwrite=True)¶

Computes the given custom metrics and adds them to these results.

Parameters

custom_metrics – a list of custom metrics to compute or a dict mapping metric names to kwargs dicts
overwrite (True) – whether to recompute any custom metrics that have already been applied

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

property backend¶: The BaseRun for these results.

static base_results_cls(type)¶

Returns the results class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunResults subclass

property cls¶: The fully-qualified name of this BaseRunResults class.

property config¶: The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False, include_missing=False)¶

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided
include_missing (False) – whether to include a row/column at the end of the matrix for unmatched labels. Only applicable if self.missing does not already appear in classes. If both “other” and “missing” rows/columns are requested, this one is last

Returns

a num_classes x num_classes confusion matrix

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)¶

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

d – a JSON dict
samples – the fiftyone.core.collections.SampleCollection for the run
config – the BaseRunConfig for the run
key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

property key¶: The run key for these results.

metrics(classes=None, average='micro', beta=1.0)¶

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Also includes any custom metrics from custom_metrics.

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)¶

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (None) –
whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:
- None (default): only include a row/column for other labels if there are any
- True: do include a row/column for other labels
- False: do not include a row/column for other labels
include_missing (None) –
whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:
- None (default): only include a row/column for missing labels if there are any
- True: do include a row/column for missing labels
- False: do not include a row/column for missing labels
other_label ("(other)") – the label to use for “other” predictions
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used
a matplotlib figure, otherwise

Return type

one of the following

print_metrics(classes=None, average='micro', beta=1.0, digits=2)¶

Prints the metrics computed via metrics().

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use
digits (2) – the number of digits of precision to print

print_report(classes=None, digits=2)¶

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report
digits (2) – the number of digits of precision to print

report(classes=None)¶

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters: classes (None) – an optional list of classes to include in the report
Returns: a dict

property samples¶: The fiftyone.core.collections.SampleCollection associated with these results.

save()¶: Saves the results to the database.

save_config()¶: Saves these results config to the database.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_regressions(samples, pred_field, gt_field='ground_truth', eval_key=None, missing=None, method=None, custom_metrics=None, progress=None, **kwargs)¶

Evaluates the regression predictions in the given collection with respect to the specified ground truth values.

You can customize the evaluation method by passing additional parameters for the method’s config class as kwargs.

The natively provided method values and their associated configs are:

"simple": SimpleEvaluationConfig

If an eval_key is specified, then this method will record some statistics on each sample:

When evaluating sample-level fields, an eval_key field will be populated on each sample recording the error of that sample’s prediction.
When evaluating frame-level fields, an eval_key field will be populated on each frame recording the error of that frame’s prediction. In addition, an eval_key field will be populated on each sample that records the average error of the frame predictions of the sample.

Parameters

samples – a fiftyone.core.collections.SampleCollection
pred_field – the name of the field containing the predicted fiftyone.core.labels.Regression instances
gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Regression instances
eval_key (None) – a string key to use to refer to this evaluation
missing (None) – a missing value. Any None-valued regressions are given this value for results purposes
method (None) – a string specifying the evaluation method to use. The supported values are fo.evaluation_config.regression_backends.keys() and the default is fo.evaluation_config.default_regression_backend
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional keyword arguments for the constructor of the RegressionEvaluationConfig being used

Returns

a RegressionResults

class fiftyone.utils.eval.RegressionEvaluationConfig(pred_field, gt_field, custom_metrics=None, **kwargs)¶

Bases: fiftyone.utils.eval.base.BaseEvaluationMethodConfig

Base class for configuring RegressionEvaluation instances.

Parameters

pred_field – the name of the field containing the predicted fiftyone.core.labels.Regression instances
gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Regression instances
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts

Attributes:

`type`	The type of run.
`cls`	The fully-qualified name of this `BaseRunConfig` class.
`method`	The name of the method.
`run_cls`	The `BaseRun` class associated with this config.

Methods:

`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_config_cls`(type)	Returns the config class for the given run type.
`build`()	Builds the `BaseRun` instance associated with this config.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a `BaseRunConfig` from a serialized JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_credentials`(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

property type¶: The type of run.

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

static base_config_cls(type)¶

Returns the config class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunConfig subclass

build()¶

Builds the BaseRun instance associated with this config.

Returns: a BaseRun instance

classmethod builder()¶: Returns a ConfigBuilder instance for this class.

property cls¶: The fully-qualified name of this BaseRunConfig class.

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()¶

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)¶

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters: d – a JSON dict
Returns: a BaseRunConfig

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)¶

Constructs a Config object from keyword arguments.

Parameters: **kwargs – keyword arguments that define the fields expected by cls
Returns: an instance of cls

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)¶

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters: **kwargs – subclass-specific credentials

classmethod load_default()¶

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

property method¶: The name of the method.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw array attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a boolean value.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)¶

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)¶

Parses an integer attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)¶

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Returns: the (field, value) that was set
Raises: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a number attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw (arbitrary) JSON field.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a string attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls¶: The BaseRun class associated with this config.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

static validate_all_or_nothing_fields(fields)¶

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Raises: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.RegressionResults(samples, config, eval_key, ytrue, ypred, confs=None, ids=None, missing=None, custom_metrics=None, backend=None)¶

Bases: fiftyone.utils.eval.base.BaseEvaluationResults

Class that stores the results of a regression evaluation.

Parameters

samples – the fiftyone.core.collections.SampleCollection used
config – the RegressionEvaluationConfig used
eval_key (None) – the evaluation key
ytrue – a list of ground truth values
ypred – a list of predicted values
confs (None) – an optional list of confidences for the predictions
eval_key – the evaluation key of the evaluation
gt_field (None) – the name of the ground truth field
pred_field (None) – the name of the predictions field
ids (None) – a list of sample or frame IDs corresponding to the regressions
missing (None) – a missing value. Any None-valued regressions are given this value for results purposes
custom_metrics (None) – an optional dict of custom metrics
backend (None) – a RegressionEvaluation backend

Methods:

`metrics`([weights])	Computes various popular regression metrics for the results.
`print_metrics`([weights, digits])	Prints the metrics computed via `metrics()`.
`plot_results`([labels, sizes, backend])	Plots the regression results.
`add_custom_metrics`(custom_metrics[, overwrite])	Computes the given custom metrics and adds them to these results.
`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_results_cls`(type)	Returns the results class for the given run type.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`from_dict`(d, samples, config, key)	Builds a `BaseRunResults` from a JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`save`()	Saves the results to the database.
`save_config`()	Saves these results config to the database.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

`backend`	The `BaseRun` for these results.
`cls`	The fully-qualified name of this `BaseRunResults` class.
`config`	The `BaseRunConfig` for these results.
`key`	The run key for these results.
`samples`	The `fiftyone.core.collections.SampleCollection` associated with these results.

metrics(weights=None)¶

Computes various popular regression metrics for the results.

The computed metrics are:

Mean squared error: sklearn.metrics.mean_squared_error()
Root mean squared error: sklearn.metrics.mean_squared_error()
Mean absolute error: sklearn.metrics.mean_absolute_error()
Median absolute error: sklearn.metrics.median_absolute_error()
R^2 score: sklearn.metrics.r2_score()
Explained variance score: sklearn.metrics.explained_variance_score()
Max error: sklearn.metrics.max_error()
Support: the number of examples

Also includes any custom metrics from custom_metrics.

Parameters: weights (None) – an optional list of weights for each example
Returns: a dict

print_metrics(weights=None, digits=2)¶

Prints the metrics computed via metrics().

Parameters

weights (None) – an optional list of weights for each example
digits (2) – the number of digits of precision to print

plot_results(labels=None, sizes=None, backend='plotly', **kwargs)¶

Plots the regression results.

You can use the labels parameters to define a coloring for the points, and you can use the sizes parameter to scale the sizes of the points.

You can attach plots generated by this method to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected points in the plot.

Parameters

labels (None) –
data to use to color the points. Can be any of the following:
- the name of a sample field or embedded.field.name of from which to extract numeric or string values
- a fiftyone.core.expressions.ViewExpression defining numeric or string values to extract via fiftyone.core.collections.SampleCollection.values()
- a list or array-like of numeric or string values (or lists of lists for frame-level regressions)
sizes (None) –
data to use to scale the sizes of the points. Can be any of the following:
- the name of a sample field or embedded.field.name from which to extract numeric values
- a fiftyone.core.expressions.ViewExpression defining numeric values to extract via fiftyone.core.collections.SampleCollection.values()
- a list or array-like of numeric values (or lists of lists for frame-level regressions)
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_regressions()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_regressions()

Returns

an fiftyone.core.plots.base.InteractivePlot

add_custom_metrics(custom_metrics, overwrite=True)¶

Computes the given custom metrics and adds them to these results.

Parameters

custom_metrics – a list of custom metrics to compute or a dict mapping metric names to kwargs dicts
overwrite (True) – whether to recompute any custom metrics that have already been applied

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

property backend¶: The BaseRun for these results.

static base_results_cls(type)¶

Returns the results class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunResults subclass

property cls¶: The fully-qualified name of this BaseRunResults class.

property config¶: The BaseRunConfig for these results.

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)¶

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

d – a JSON dict
samples – the fiftyone.core.collections.SampleCollection for the run
config – the BaseRunConfig for the run
key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

property key¶: The run key for these results.

property samples¶: The fiftyone.core.collections.SampleCollection associated with these results.

save()¶: Saves the results to the database.

save_config()¶: Saves these results config to the database.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

fiftyone.utils.eval.evaluate_segmentations(samples, pred_field, gt_field='ground_truth', eval_key=None, mask_targets=None, method=None, custom_metrics=None, progress=None, **kwargs)¶

Evaluates the specified semantic segmentation masks in the given collection with respect to the specified ground truth masks.

If the size of a predicted mask does not match the ground truth mask, it is resized to match the ground truth.

By default, this method simply performs pixelwise evaluation of the full masks, but other strategies such as boundary-only evaluation can be configured by passing additional parameters for the method’s config class as kwargs.

The natively provided method values and their associated configs are:

"simple": SimpleEvaluationConfig

If an eval_key is provided, the accuracy, precision, and recall of each sample is recorded in top-level fields of each sample:

 Accuracy: sample.<eval_key>_accuracy
Precision: sample.<eval_key>_precision
   Recall: sample.<eval_key>_recall

In addition, when evaluating frame-level masks, the accuracy, precision, and recall of each frame if recorded in the following frame-level fields:

 Accuracy: frame.<eval_key>_accuracy
Precision: frame.<eval_key>_precision
   Recall: frame.<eval_key>_recall

Note

The mask values 0 and #000000 are treated as a background class for the purposes of computing evaluation metrics like precision and recall.

Parameters

samples – a fiftyone.core.collections.SampleCollection
pred_field – the name of the field containing the predicted fiftyone.core.labels.Segmentation instances
gt_field ("ground_truth") – the name of the field containing the ground truth fiftyone.core.labels.Segmentation instances
eval_key (None) – an evaluation key to use to refer to this evaluation
mask_targets (None) – a dict mapping pixel values or RGB hex strings to labels. If not provided, the observed values are used as labels
method (None) – a string specifying the evaluation method to use. The supported values are fo.evaluation_config.segmentation_backends.keys() and the default is fo.evaluation_config.default_segmentation_backend
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional keyword arguments for the constructor of the SegmentationEvaluationConfig being used

Returns

a SegmentationResults

class fiftyone.utils.eval.SegmentationEvaluationConfig(pred_field, gt_field, compute_dice=False, custom_metrics=None, **kwargs)¶

Bases: fiftyone.utils.eval.base.BaseEvaluationMethodConfig

Base class for configuring SegmentationEvaluation instances.

Parameters

pred_field – the name of the field containing the predicted fiftyone.core.labels.Segmentation instances
gt_field – the name of the field containing the ground truth fiftyone.core.labels.Segmentation instances
compute_dice (False) – whether to compute the Dice coefficient for each sample
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts

Attributes:

`type`	The type of run.
`cls`	The fully-qualified name of this `BaseRunConfig` class.
`method`	The name of the method.
`run_cls`	The `BaseRun` class associated with this config.

Methods:

`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_config_cls`(type)	Returns the config class for the given run type.
`build`()	Builds the `BaseRun` instance associated with this config.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a `BaseRunConfig` from a serialized JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_credentials`(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

property type¶: The type of run.

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

static base_config_cls(type)¶

Returns the config class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunConfig subclass

build()¶

Builds the BaseRun instance associated with this config.

Returns: a BaseRun instance

classmethod builder()¶: Returns a ConfigBuilder instance for this class.

property cls¶: The fully-qualified name of this BaseRunConfig class.

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod default()¶

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)¶

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters: d – a JSON dict
Returns: a BaseRunConfig

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_kwargs(**kwargs)¶

Constructs a Config object from keyword arguments.

Parameters: **kwargs – keyword arguments that define the fields expected by cls
Returns: an instance of cls

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)¶

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters: **kwargs – subclass-specific credentials

classmethod load_default()¶

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

property method¶: The name of the method.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw array attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns

a list of raw (untouched) values

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a boolean value.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns

True/False

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)¶

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field, which is equal to a value from choices

Raises

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns

a dictionary

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)¶

Parses an integer attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns

an int

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)¶

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Returns: the (field, value) that was set
Raises: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a number attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns

a number (e.g. int, float)

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns

an instance of cls

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns

a list of cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)¶

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns

a dictionary whose values are cls instances

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a path string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a raw (arbitrary) JSON field.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns

the raw (untouched) value of the given field

Raises

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)¶

Parses a string attribute.

Parameters

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns

a string

Raises

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls¶: The BaseRun class associated with this config.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

static validate_all_or_nothing_fields(fields)¶

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters: fields – a dictionary of pre-parsed fields
Raises: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.SegmentationResults(samples, config, eval_key, pixel_confusion_matrix, classes, matches=None, missing=None, custom_metrics=None, backend=None)¶

Bases: fiftyone.utils.eval.base.BaseClassificationResults

Class that stores the results of a segmentation evaluation.

Parameters

samples – the fiftyone.core.collections.SampleCollection used
config – the SegmentationEvaluationConfig used
eval_key – the evaluation key
pixel_confusion_matrix – a pixel value confusion matrix
classes – a list of class labels corresponding to the confusion matrix
matches (None) – a list of (gt_label, pred_label, pixel_count, gt_id, pred_id) matches
missing (None) – a missing (background) class
custom_metrics (None) – an optional dict of custom metrics
backend (None) – a SegmentationEvaluation backend

Methods:

`dice_score`()	Computes the Dice score across all samples in the evaluation.
`add_custom_metrics`(custom_metrics[, overwrite])	Computes the given custom metrics and adds them to these results.
`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_results_cls`(type)	Returns the results class for the given run type.
`confusion_matrix`([classes, include_other, …])	Generates a confusion matrix for the results via `sklearn.metrics.confusion_matrix()`.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`from_dict`(d, samples, config, key)	Builds a `BaseRunResults` from a JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`metrics`([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
`plot_confusion_matrix`([classes, …])	Plots a confusion matrix for the evaluation results.
`print_metrics`([classes, average, beta, digits])	Prints the metrics computed via `metrics()`.
`print_report`([classes, digits])	Prints a classification report for the results via `sklearn.metrics.classification_report()`.
`report`([classes])	Generates a classification report for the results via `sklearn.metrics.classification_report()`.
`save`()	Saves the results to the database.
`save_config`()	Saves these results config to the database.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

`backend`	The `BaseRun` for these results.
`cls`	The fully-qualified name of this `BaseRunResults` class.
`config`	The `BaseRunConfig` for these results.
`key`	The run key for these results.
`samples`	The `fiftyone.core.collections.SampleCollection` associated with these results.

dice_score()¶

Computes the Dice score across all samples in the evaluation.

Returns: the Dice score in [0, 1]

add_custom_metrics(custom_metrics, overwrite=True)¶

Computes the given custom metrics and adds them to these results.

Parameters

custom_metrics – a list of custom metrics to compute or a dict mapping metric names to kwargs dicts
overwrite (True) – whether to recompute any custom metrics that have already been applied

attributes()¶

Returns the list of class attributes that will be serialized by serialize().

Returns: a list of attributes

property backend¶: The BaseRun for these results.

static base_results_cls(type)¶

Returns the results class for the given run type.

Parameters: type – a BaseRunConfig.type
Returns: a BaseRunResults subclass

property cls¶: The fully-qualified name of this BaseRunResults class.

property config¶: The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False, include_missing=False)¶

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided
include_missing (False) – whether to include a row/column at the end of the matrix for unmatched labels. Only applicable if self.missing does not already appear in classes. If both “other” and “missing” rows/columns are requested, this one is last

Returns

a num_classes x num_classes confusion matrix

copy()¶

Returns a deep copy of the object.

Returns: a Serializable instance

custom_attributes(dynamic=False, private=False)¶

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns

a list of class attributes

classmethod from_dict(d, samples, config, key)¶

Builds a BaseRunResults from a JSON dict representation of it.

Parameters

d – a JSON dict
samples – the fiftyone.core.collections.SampleCollection for the run
config – the BaseRunConfig for the run
key – the run key

Returns

a BaseRunResults

classmethod from_json(path, *args, **kwargs)¶

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)¶

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns

an instance of the Serializable class

classmethod get_class_name()¶: Returns the fully-qualified class name string of this object.

property key¶: The run key for these results.

metrics(classes=None, average='micro', beta=1.0)¶

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Also includes any custom metrics from custom_metrics.

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use

Returns

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)¶

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters

classes (None) – an optional list of classes to include in the confusion matrix
include_other (None) –
whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:
- None (default): only include a row/column for other labels if there are any
- True: do include a row/column for other labels
- False: do not include a row/column for other labels
include_missing (None) –
whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:
- None (default): only include a row/column for missing labels if there are any
- True: do include a row/column for missing labels
- False: do not include a row/column for missing labels
other_label ("(other)") – the label to use for “other” predictions
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns

a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used
a matplotlib figure, otherwise

Return type

one of the following

print_metrics(classes=None, average='micro', beta=1.0, digits=2)¶

Prints the metrics computed via metrics().

Parameters

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use
digits (2) – the number of digits of precision to print

print_report(classes=None, digits=2)¶

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters

classes (None) – an optional list of classes to include in the report
digits (2) – the number of digits of precision to print

report(classes=None)¶

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters: classes (None) – an optional list of classes to include in the report
Returns: a dict

property samples¶: The fiftyone.core.collections.SampleCollection associated with these results.

save()¶: Saves the results to the database.

save_config()¶: Saves these results config to the database.

serialize(reflective=False)¶

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)¶

Returns a string representation of this object.

Parameters

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns

a string representation of the object

write_json(path, pretty_print=False, **kwargs)¶

Serializes the object and writes it to disk.

Parameters

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()