fiftyone.utils.eval.openimages#

Open Images-style detection evaluation.

Copyright 2017-2025, Voxel51, Inc.
voxel51.com

Classes:

`OpenImagesEvaluationConfig`(pred_field, gt_field)	Open Images-style evaluation config.
`OpenImagesEvaluation`(config)	Open Images-style evaluation.
`OpenImagesDetectionResults`(samples, config, ...)	Class that stores the results of an Open Images detection evaluation.

class fiftyone.utils.eval.openimages.OpenImagesEvaluationConfig(pred_field, gt_field, iou=None, classwise=None, iscrowd='IsGroupOf', use_masks=False, use_boxes=False, tolerance=None, max_preds=None, error_level=1, hierarchy=None, pos_label_field=None, neg_label_field=None, expand_gt_hierarchy=True, expand_pred_hierarchy=False, custom_metrics=None, **kwargs)#

Bases: DetectionEvaluationConfig

Open Images-style evaluation config.

Parameters:

pred_field – the name of the field containing the predicted fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, or fiftyone.core.labels.Keypoints
gt_field – the name of the field containing the ground truth fiftyone.core.labels.Detections, fiftyone.core.labels.Polylines, or fiftyone.core.labels.Keypoints
iou (None) – the IoU threshold to use to determine matches
classwise (None) – whether to only match objects with the same class label (True) or allow matches between classes (False)
iscrowd ("IsGroupOf") – the name of the crowd attribute
use_masks (False) – whether to compute IoUs using the instances masks in the mask attribute of the provided objects, which must be fiftyone.core.labels.Detection instances
use_boxes (False) – whether to compute IoUs using the bounding boxes of the provided fiftyone.core.labels.Polyline instances rather than using their actual geometries
tolerance (None) – a tolerance, in pixels, when generating approximate polylines for instance masks. Typical values are 1-3 pixels. By default, IoUs are computed directly on the dense pixel masks
max_preds (None) – the maximum number of predicted objects to evaluate when computing mAP and PR curves
error_level (1) –
the error level to use when manipulating instance masks or polylines. Valid values are:
- 0: raise geometric errors that are encountered
- 1: log warnings if geometric errors are encountered
- 2: ignore geometric errors
If error_level > 0, any calculation that raises a geometric error will default to an IoU of 0
hierarchy (None) – an optional dict containing a hierarchy of classes for evaluation following the structure {"LabelName": label, "Subcategory": [{...}, ...]}
pos_label_field (None) – the name of a field containing image-level fiftyone.core.labels.Classifications that specify which classes should be evaluated in the image
neg_label_field (None) – the name of a field containing image-level fiftyone.core.labels.Classifications that specify which classes should not be evaluated in the image
expand_gt_hierarchy (True) – whether to expand ground truth objects and labels according to the provided hierarchy
expand_pred_hierarchy (False) – whether to expand predicted objects and labels according to the provided hierarchy
custom_metrics (None) – an optional list of custom metrics to compute or dict mapping metric names to kwargs dicts

Attributes:

`method`	The name of the method.
`requires_additional_fields`	Whether fields besides `pred_field` and `gt_field` are required in order to perform evaluation.
`cls`	The fully-qualified name of this `BaseRunConfig` class.
`run_cls`	The `BaseRun` class associated with this config.
`type`	The type of run.

Methods:

`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_config_cls`(type)	Returns the config class for the given run type.
`build`()	Builds the `BaseRun` instance associated with this config.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a `BaseRunConfig` from a serialized JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_credentials`(**kwargs)	Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

property method#: The name of the method.

property requires_additional_fields#

Whether fields besides pred_field and gt_field are required in order to perform evaluation.

If True then the entire samples will be loaded rather than using select_fields() to optimize.

attributes()#

Returns the list of class attributes that will be serialized by serialize().

Returns:: a list of attributes

static base_config_cls(type)#

Returns the config class for the given run type.

Parameters:: type – a BaseRunConfig.type
Returns:: a BaseRunConfig subclass

build()#

Builds the BaseRun instance associated with this config.

Returns:: a BaseRun instance

classmethod builder()#: Returns a ConfigBuilder instance for this class.

property cls#: The fully-qualified name of this BaseRunConfig class.

copy()#

Returns a deep copy of the object.

Returns:: a Serializable instance

custom_attributes(dynamic=False, private=False)#

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters:

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns:

a list of class attributes

classmethod default()#

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)#

Constructs a BaseRunConfig from a serialized JSON dict representation of it.

Parameters:: d – a JSON dict
Returns:: a BaseRunConfig

classmethod from_json(path, *args, **kwargs)#

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters:

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod from_kwargs(**kwargs)#

Constructs a Config object from keyword arguments.

Parameters:: **kwargs – keyword arguments that define the fields expected by cls
Returns:: an instance of cls

classmethod from_str(s, *args, **kwargs)#

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters:

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod get_class_name()#: Returns the fully-qualified class name string of this object.

load_credentials(**kwargs)#

Loads any necessary credentials from the given keyword arguments or the relevant FiftyOne config.

Parameters:: **kwargs – subclass-specific credentials

classmethod load_default()#

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw array attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns:

a list of raw (untouched) values

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)#

Parses a boolean value.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns:

True/False

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)#

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters:

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field, which is equal to a value from choices

Raises:

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)#

Parses a dictionary attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns:

a dictionary

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)#

Parses an integer attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns:

an int

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)#

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Returns:: the (field, value) that was set
Raises:: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)#

Parses a number attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns:

a number (e.g. int, float)

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns:

an instance of cls

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns:

a list of cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns:

a dictionary whose values are cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)#

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a path string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw (arbitrary) JSON field.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field

Raises:

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)#

Parses a string attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

property run_cls#: The BaseRun class associated with this config.

serialize(reflective=False)#

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters:: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns:: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)#

Returns a string representation of this object.

Parameters:

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns:

a string representation of the object

property type#: The type of run.

static validate_all_or_nothing_fields(fields)#

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Raises:: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)#

Serializes the object and writes it to disk.

Parameters:

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.utils.eval.openimages.OpenImagesEvaluation(config)#

Bases: DetectionEvaluation

Open Images-style evaluation.

Parameters:: config – a OpenImagesEvaluationConfig

Methods:

`evaluate`(sample_or_frame[, eval_key])	Performs Open Images-style evaluation on the given image.
`generate_results`(samples, matches[, ...])	Generates aggregate evaluation results for the samples.
`add_fields_to_sidebar_group`(samples, eval_key)
`cleanup`(samples, eval_key)	Cleans up the results of the run with the given key from the collection.
`cleanup_custom_metrics`(samples, eval_key[, ...])
`compute_custom_metrics`(samples, eval_key, ...)
`delete_run`(samples, key[, cleanup])	Deletes the results associated with the given run key from the collection.
`delete_runs`(samples[, cleanup])	Deletes all runs from the collection.
`ensure_requirements`()	Ensures that any necessary packages to execute this run are installed.
`ensure_usage_requirements`()	Ensures that any necessary packages to use existing results for this run are installed.
`from_config`(config)	Instantiates a Configurable class from a <cls>Config instance.
`from_dict`(d)	Instantiates a Configurable class from a <cls>Config dict.
`from_json`(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
`from_kwargs`(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
`get_custom_metric_fields`(samples, eval_key)
`get_fields`(samples, eval_key[, ...])	Gets the fields that were involved in the given run.
`get_run_info`(samples, key)	Gets the `BaseRunInfo` for the given key on the collection.
`has_cached_run_results`(samples, key)	Determines whether `BaseRunResults` for the given key are cached on the collection.
`list_runs`(samples[, type, method])	Returns the list of run keys on the given collection.
`load_run_results`(samples, key[, cache, ...])	Loads the `BaseRunResults` for the given key on the collection.
`load_run_view`(samples, key[, select_fields])	Loads the view on which the specified run was performed.
`parse`(class_name[, module_name])	Parses a Configurable subclass name string.
`register_run`(samples, key[, overwrite, cleanup])	Registers a run of this method under the given key on the given collection.
`register_samples`(samples, eval_key[, dynamic])	Registers the collection on which evaluation will be performed.
`rename`(samples, eval_key, new_eval_key)	Performs any necessary operations required to rename this run's key.
`rename_custom_metrics`(samples, eval_key, ...)
`run_info_cls`()	The `BaseRunInfo` class associated with this class.
`save_run_info`(samples, run_info[, ...])	Saves the run information on the collection.
`save_run_results`(samples, key, run_results)	Saves the run results on the collection.
`update_run_config`(samples, key, config)	Updates the `BaseRunConfig` for the given run on the collection.
`update_run_key`(samples, key, new_key)	Replaces the key for the given run with a new key.
`validate`(config)	Validates that the given config is an instance of <cls>Config.
`validate_run`(samples, key[, overwrite])	Validates that the collection can accept this run.

evaluate(sample_or_frame, eval_key=None)#

Performs Open Images-style evaluation on the given image.

Predicted objects are matched to ground truth objects in descending order of confidence, with matches requiring a minimum IoU of self.config.iou.

The self.config.classwise parameter controls whether to only match objects with the same class label (True) or allow matches between classes (False).

If a ground truth object has its self.config.iscrowd attribute set, then the object can have multiple true positive predictions matched to it.

Parameters:

sample_or_frame – a fiftyone.core.sample.Sample or fiftyone.core.frame.Frame
eval_key (None) – the evaluation key for this evaluation

Returns:

a list of matched (gt_label, pred_label, iou, pred_confidence, gt_id, pred_id) tuples

generate_results(samples, matches, eval_key=None, classes=None, missing=None, progress=None)#

Generates aggregate evaluation results for the samples.

This method generates precision and recall curves for the configured IoU at self.config.iou.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
matches – a list of (gt_label, pred_label, iou, pred_confidence, gt_id, pred_id) matches. Either label can be None to indicate an unmatched object
eval_key (None) – the evaluation key for this evaluation
classes (None) – the list of possible classes. If not provided, the observed ground truth/predicted labels are used for results purposes
missing (None) – a missing label string. Any unmatched objects are given this label for results purposes
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns:

a OpenImagesDetectionResults

add_fields_to_sidebar_group(samples, eval_key, omit_fields=None)#

cleanup(samples, eval_key)#

Cleans up the results of the run with the given key from the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key

cleanup_custom_metrics(samples, eval_key, metric_uris=None)#

compute_custom_metrics(samples, eval_key, results, metric_uris=None)#

classmethod delete_run(samples, key, cleanup=True)#

Deletes the results associated with the given run key from the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
cleanup (True) – whether to execute the run’s BaseRun.cleanup() method

classmethod delete_runs(samples, cleanup=True)#

Deletes all runs from the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
cleanup (True) – whether to execute the run’s BaseRun.cleanup() methods

ensure_requirements()#

Ensures that any necessary packages to execute this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

ensure_usage_requirements()#

Ensures that any necessary packages to use existing results for this run are installed.

Runs should respect fiftyone.config.requirement_error_level when handling errors.

classmethod from_config(config)#: Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)#

Instantiates a Configurable class from a <cls>Config dict.

Parameters:: d – a dict to construct a <cls>Config
Returns:: an instance of cls

classmethod from_json(json_path)#

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters:: json_path – path to a JSON file for type <cls>Config
Returns:: an instance of cls

classmethod from_kwargs(**kwargs)#

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters:: **kwargs – keyword arguments that define the fields of a <cls>Config dict
Returns:: an instance of cls

get_custom_metric_fields(samples, eval_key, metric_uris=None)#

get_fields(samples, eval_key, include_custom_metrics=True)#

Gets the fields that were involved in the given run.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key

Returns:

a list of fields

classmethod get_run_info(samples, key)#

Gets the BaseRunInfo for the given key on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key

Returns:

a BaseRunInfo

classmethod has_cached_run_results(samples, key)#

Determines whether BaseRunResults for the given key are cached on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key

Returns:

True/False

classmethod list_runs(samples, type=None, method=None, **kwargs)#

Returns the list of run keys on the given collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
type (None) –
a specific run type to match, which can be:
- a string fiftyone.core.runs.BaseRunConfig.type
- a fiftyone.core.runs.BaseRun class or its fully-qualified class name string
method (None) – a specific fiftyone.core.runs.BaseRunConfig.method string to match
**kwargs – optional config parameters to match

Returns:

a list of run keys

classmethod load_run_results(samples, key, cache=True, load_view=True, **kwargs)#

Loads the BaseRunResults for the given key on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
cache (True) – whether to cache the results on the collection
load_view (True) – whether to load the run view in the results (True) or the full dataset (False)
**kwargs – keyword arguments for the run’s BaseRunConfig.load_credentials() method

Returns:

a BaseRunResults, or None if the run did not save results

classmethod load_run_view(samples, key, select_fields=False)#

Loads the view on which the specified run was performed.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
select_fields (False) – whether to exclude fields involved in other runs of the same type

Returns:

a fiftyone.core.collections.SampleCollection

static parse(class_name, module_name=None)#

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters:

class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”
module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns:

the Configurable class config_cls: the Config class associated with cls

Return type:

cls

register_run(samples, key, overwrite=True, cleanup=True)#

Registers a run of this method under the given key on the given collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
overwrite (True) – whether to allow overwriting an existing run of the same type
cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

register_samples(samples, eval_key, dynamic=True)#

Registers the collection on which evaluation will be performed.

This method will be called before the first call to evaluate(). Subclasses can extend this method to perform any setup required for an evaluation run.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
eval_key – the evaluation key for this evaluation
dynamic (True) – whether to declare the dynamic object-level attributes that are populated on the dataset’s schema

rename(samples, eval_key, new_eval_key)#

Performs any necessary operations required to rename this run’s key.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
new_key – a new run key

rename_custom_metrics(samples, eval_key, new_eval_key, metric_uris=None)#

classmethod run_info_cls()#: The BaseRunInfo class associated with this class.

classmethod save_run_info(samples, run_info, overwrite=True, cleanup=True)#

Saves the run information on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
run_info – a BaseRunInfo
overwrite (True) – whether to overwrite an existing run with the same key
cleanup (True) – whether to execute an existing run’s BaseRun.cleanup() method when overwriting it

classmethod save_run_results(samples, key, run_results, overwrite=True, cache=True)#

Saves the run results on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
run_results – a BaseRunResults, or None
overwrite (True) – whether to overwrite an existing result with the same key
cache (True) – whether to cache the results on the collection

classmethod update_run_config(samples, key, config)#

Updates the BaseRunConfig for the given run on the collection.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
config – a BaseRunConfig

classmethod update_run_key(samples, key, new_key)#

Replaces the key for the given run with a new key.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
new_key – a new run key

classmethod validate(config)#

Validates that the given config is an instance of <cls>Config.

Raises:: ConfigurableError – if config is not an instance of <cls>Config

validate_run(samples, key, overwrite=True)#

Validates that the collection can accept this run.

The run may be invalid if, for example, a run of a different type has already been run under the same key and thus overwriting it would cause ambiguity on how to cleanup the results.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
key – a run key
overwrite (True) – whether to allow overwriting an existing run of the same type

Raises:

ValueError – if the run is invalid

class fiftyone.utils.eval.openimages.OpenImagesDetectionResults(samples, config, eval_key, matches, precision, recall, classes, thresholds=None, missing=None, custom_metrics=None, backend=None)#

Bases: DetectionResults

Class that stores the results of an Open Images detection evaluation.

Parameters:

samples – the fiftyone.core.collections.SampleCollection used
config – the OpenImagesEvaluationConfig used
eval_key – the evaluation key
matches – a list of (gt_label, pred_label, iou, pred_confidence, gt_id, pred_id) matches. Either label can be None to indicate an unmatched object
precision – a dict of per-class precision values
recall – a dict of per-class recall values
classes – the list of possible classes
thresholds (None) – an optional dict of per-class decision thresholds
missing (None) – a missing label string. Any unmatched objects are given this label for evaluation purposes
custom_metrics (None) – an optional dict of custom metrics
backend (None) – a OpenImagesEvaluation backend

Methods:

`plot_pr_curves`([classes, num_points, backend])	Plots precision-recall (PR) curves for the detection results.
`mAP`([classes])	Computes Open Images-style mean average precision (mAP) for the specified classes.
`add_custom_metrics`(custom_metrics[, overwrite])	Computes the given custom metrics and adds them to these results.
`attributes`()	Returns the list of class attributes that will be serialized by `serialize()`.
`base_results_cls`(type)	Returns the results class for the given run type.
`clear_subset`()	Clears the subset set by `use_subset()`, if any.
`confusion_matrix`([classes, include_other, ...])	Generates a confusion matrix for the results via `sklearn.metrics.confusion_matrix()`.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`from_dict`(d, samples, config, key)	Builds a `BaseRunResults` from a JSON dict representation of it.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`metrics`([classes, average, beta])	Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.
`plot_confusion_matrix`([classes, ...])	Plots a confusion matrix for the evaluation results.
`print_metrics`([classes, average, beta, digits])	Prints the metrics computed via `metrics()`.
`print_report`([classes, digits])	Prints a classification report for the results via `sklearn.metrics.classification_report()`.
`report`([classes])	Generates a classification report for the results via `sklearn.metrics.classification_report()`.
`save`()	Saves the results to the database.
`save_config`()	Saves these results config to the database.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`use_subset`(subset_def)	Restricts the evaluation results to the specified subset.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

Attributes:

`backend`	The `BaseRun` for these results.
`cls`	The fully-qualified name of this `BaseRunResults` class.
`config`	The `BaseRunConfig` for these results.
`has_subset`	Whether these results are currently restricted to a subset via `use_subset()`.
`key`	The run key for these results.
`samples`	The `fiftyone.core.collections.SampleCollection` associated with these results.

plot_pr_curves(classes=None, num_points=101, backend='plotly', **kwargs)#

Plots precision-recall (PR) curves for the detection results.

Parameters:

classes (None) – a list of classes to generate curves for. By default, the top 3 AP classes will be plotted
num_points (101) – the number of linearly spaced recall values to plot
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_pr_curves()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_pr_curves()

Returns:

a fiftyone.core.plots.plotly.PlotlyNotebookPlot, if you are working in a notebook context and the plotly backend is used
a plotly or matplotlib figure, otherwise

Return type:

one of the following

mAP(classes=None)#

Computes Open Images-style mean average precision (mAP) for the specified classes.

See this page for more details about Open Images-style mAP.

Parameters:: classes (None) – a list of classes for which to compute mAP
Returns:: the mAP in [0, 1]

add_custom_metrics(custom_metrics, overwrite=True)#

Computes the given custom metrics and adds them to these results.

Parameters:

custom_metrics – a list of custom metrics to compute or a dict mapping metric names to kwargs dicts
overwrite (True) – whether to recompute any custom metrics that have already been applied

attributes()#

Returns the list of class attributes that will be serialized by serialize().

Returns:: a list of attributes

property backend#: The BaseRun for these results.

static base_results_cls(type)#

Returns the results class for the given run type.

Parameters:: type – a BaseRunConfig.type
Returns:: a BaseRunResults subclass

clear_subset()#

Clears the subset set by use_subset(), if any.

Subsequent operations will be performed on the full results.

property cls#: The fully-qualified name of this BaseRunResults class.

property config#: The BaseRunConfig for these results.

confusion_matrix(classes=None, include_other=False, include_missing=False)#

Generates a confusion matrix for the results via sklearn.metrics.confusion_matrix().

The rows of the confusion matrix represent ground truth and the columns represent predictions.

Parameters:

classes (None) – an optional list of classes to include in the confusion matrix
include_other (False) – whether to include an extra row/column at the end of the matrix for labels that do not appear in classes. Only applicable if classes are provided
include_missing (False) – whether to include a row/column at the end of the matrix for unmatched labels. Only applicable if self.missing does not already appear in classes. If both “other” and “missing” rows/columns are requested, this one is last

Returns:

a num_classes x num_classes confusion matrix

copy()#

Returns a deep copy of the object.

Returns:: a Serializable instance

custom_attributes(dynamic=False, private=False)#

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters:

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns:

a list of class attributes

classmethod from_dict(d, samples, config, key)#

Builds a BaseRunResults from a JSON dict representation of it.

Parameters:

d – a JSON dict
samples – the fiftyone.core.collections.SampleCollection for the run
config – the BaseRunConfig for the run
key – the run key

Returns:

a BaseRunResults

classmethod from_json(path, *args, **kwargs)#

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters:

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod from_str(s, *args, **kwargs)#

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters:

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod get_class_name()#: Returns the fully-qualified class name string of this object.

property has_subset#: Whether these results are currently restricted to a subset via use_subset().

property key#: The run key for these results.

metrics(classes=None, average='micro', beta=1.0)#

Computes classification metrics for the results, including accuracy, precision, recall, and F-beta score.

See sklearn.metrics.precision_recall_fscore_support() for details.

Also includes any custom metrics from custom_metrics.

Parameters:

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use

Returns:

a dict

plot_confusion_matrix(classes=None, include_other=None, include_missing=None, other_label='(other)', backend='plotly', **kwargs)#

Plots a confusion matrix for the evaluation results.

If you are working in a notebook environment with the default plotly backend, this method returns an interactive fiftyone.core.plots.plotly.InteractiveHeatmap that you can attach to an App session via its fiftyone.core.session.Session.plots attribute, which will automatically sync the session’s view with the currently selected cells in the confusion matrix.

Parameters:

classes (None) – an optional list of classes to include in the confusion matrix
include_other (None) –
whether to include a row/column for examples whose label is in classes but are matched to labels that do not appear in classes. Only applicable if classes are provided. The supported values are:
- None (default): only include a row/column for other labels if there are any
- True: do include a row/column for other labels
- False: do not include a row/column for other labels
include_missing (None) –
whether to include a row/column for missing ground truth/predictions in the confusion matrix. The supported values are:
- None (default): only include a row/column for missing labels if there are any
- True: do include a row/column for missing labels
- False: do not include a row/column for missing labels
other_label ("(other)") – the label to use for “other” predictions
backend ("plotly") – the plotting backend to use. Supported values are ("plotly", "matplotlib")
**kwargs –
keyword arguments for the backend plotting method:
- ”plotly” backend: fiftyone.core.plots.plotly.plot_confusion_matrix()
- ”matplotlib” backend: fiftyone.core.plots.matplotlib.plot_confusion_matrix()

Returns:

a fiftyone.core.plots.plotly.InteractiveHeatmap, if the plotly backend is used
a matplotlib figure, otherwise

Return type:

one of the following

print_metrics(classes=None, average='micro', beta=1.0, digits=2)#

Prints the metrics computed via metrics().

Parameters:

classes (None) – an optional list of classes to include in the calculations
average ("micro") – the averaging strategy to use
beta (1.0) – the F-beta value to use
digits (2) – the number of digits of precision to print

print_report(classes=None, digits=2)#

Prints a classification report for the results via sklearn.metrics.classification_report().

Parameters:

classes (None) – an optional list of classes to include in the report
digits (2) – the number of digits of precision to print

report(classes=None)#

Generates a classification report for the results via sklearn.metrics.classification_report().

Parameters:: classes (None) – an optional list of classes to include in the report
Returns:: a dict

property samples#: The fiftyone.core.collections.SampleCollection associated with these results.

save()#: Saves the results to the database.

save_config()#: Saves these results config to the database.

serialize(reflective=False)#

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters:: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns:: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)#

Returns a string representation of this object.

Parameters:

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns:

a string representation of the object

use_subset(subset_def)#

Restricts the evaluation results to the specified subset.

Subsequent calls to supported methods on this instance will only contain results from the specified subset rather than the full results.

Use clear_subset() to reset to the full results. Or, equivalently, use the context manager interface as demonstrated below to automatically reset the results when the context exits.

Example usage:

import fiftyone as fo
import fiftyone.zoo as foz
import fiftyone.utils.random as four
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")
four.random_split(dataset, {"sunny": 0.7, "cloudy": 0.2, "rainy": 0.1})

results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
)

counts = dataset.count_values("ground_truth.detections.label")
classes = sorted(counts, key=counts.get, reverse=True)[:5]

# Full results
results.print_report(classes=classes)

# Sunny samples
subset_def = dict(type="field", field="tags", value="sunny")
with results.use_subset(subset_def):
    results.print_report(classes=classes)

# Small objects
bbox_area = F("bounding_box")[2] * F("bounding_box")[3]
small_objects = bbox_area <= 0.05
subset_def = dict(type="attribute", expr=small_objects)
with results.use_subset(subset_def):
    results.print_report(classes=classes)

Parameters:

subset_def –

the subset definition, which can be:

a dict or list of dicts defining the subset. See above for examples and see get_subset_view() for full syntax
a fiftyone.core.view.DatasetView defining the subset

Returns:

self

write_json(path, pretty_print=False, **kwargs)#

Serializes the object and writes it to disk.

Parameters:

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()