Advanced Evaluation Usage

Evaluating views into your dataset

All evaluation methods are exposed on DatasetView objects, which means that you can define arbitrarily complex views into your datasets and run evaluation on those.

For example, the snippet below evaluates only the medium-sized objects in a dataset:

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart", dataset_name="eval-demo")
dataset.compute_metadata()

# Create an expression that will match objects whose bounding boxes have
# areas between 32^2 and 96^2 pixels
bbox_area = (
    F("$metadata.width") * F("bounding_box")[2] *
    F("$metadata.height") * F("bounding_box")[3]
)
medium_boxes = (32 ** 2 < bbox_area) & (bbox_area < 96 ** 2)

# Create a view that contains only medium-sized objects
medium_view = (
    dataset
    .filter_labels("ground_truth", medium_boxes)
    .filter_labels("predictions", medium_boxes)
)

print(medium_view)

# Evaluate the medium-sized objects
results = medium_view.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval_medium",
)

# Print some aggregate metrics
print(results.metrics())

# View results in the App
session = fo.launch_app(view=medium_view)

Note

If you run evaluation on a complex view, don’t worry, you can always load the view later!

Loading a previous evaluation result

You can view a list of evaluation keys for evaluations that you have previously run on a dataset via list_evaluations().

Evaluation keys are stored at the dataset-level, but if a particular evaluation was run on a view into your dataset, you can use load_evaluation_view() to retrieve the exact view on which you evaluated:

import fiftyone as fo

dataset = fo.load_dataset(...)

# List available evaluations
dataset.list_evaluations()
# ["my_eval1", "my_eval2", ...]

# Load the view into the dataset on which `my_eval1` was run
eval1_view = dataset.load_evaluation_view("my_eval1")

Note

If you have run multiple evaluations on a dataset, you can use the select_fields parameter of the load_evaluation_view() method to hide any fields that were populated by other evaluation runs, allowing you to, for example, focus on a specific set of evaluation results in the App:

import fiftyone as fo

dataset = fo.load_dataset(...)

# Load a view that contains the results of evaluation `my_eval1` and
# hides all other evaluation data
eval1_view = dataset.load_evaluation_view("my_eval1", select_fields=True)

session = fo.launch_app(view=eval1_view)

Evaluating videos

Available in: Open SourceEnterprise

Introduced in FiftyOne 0.11.2 · FiftyOne Enterprise 1.0

All evaluation methods can be applied to frame-level labels in addition to sample-level labels.

You can evaluate frame-level labels of a video dataset by adding the frames prefix to the relevant prediction and ground truth frame fields.

Note

When evaluating frame-level labels, helpful statistics are tabulated at both the sample- and frame-levels of your dataset. Refer to the documentation of the relevant evaluation method for more details.

The example below demonstrates evaluating (mocked) frame-level detections on the quickstart-video dataset from the Dataset Zoo:

import random

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset(
    "quickstart-video", dataset_name="video-eval-demo"
)

#
# Create some test predictions by copying the ground truth objects into a
# new `predictions` field of the frames with 10% of the labels perturbed at
# random
#

classes = dataset.distinct("frames.detections.detections.label")

def jitter(val):
    if random.random() < 0.10:
        return random.choice(classes)

    return val

predictions = []
for sample_gts in dataset.values("frames.detections"):
    sample_predictions = []
    for frame_gts in sample_gts:
        sample_predictions.append(
            fo.Detections(
                detections=[
                    fo.Detection(
                        label=jitter(gt.label),
                        bounding_box=gt.bounding_box,
                        confidence=random.random(),
                    )
                    for gt in frame_gts.detections
                ]
            )
        )

    predictions.append(sample_predictions)

dataset.set_values("frames.predictions", predictions)

print(dataset)

# Evaluate the frame-level `predictions` against the frame-level
# `detections` objects
results = dataset.evaluate_detections(
    "frames.predictions",
    gt_field="frames.detections",
    eval_key="eval",
)

# Print a classification report
results.print_report()

              precision    recall  f1-score   support

      person       0.76      0.93      0.84      1108
   road sign       0.90      0.94      0.92      2726
     vehicle       0.98      0.94      0.96      7511

   micro avg       0.94      0.94      0.94     11345
   macro avg       0.88      0.94      0.91     11345
weighted avg       0.94      0.94      0.94     11345

You can also view frame-level evaluation results as evaluation patches by first converting to frames and then to patches!

# Convert to frame evaluation patches
frames = dataset.to_frames(sample_frames=True)
frame_eval_patches = frames.to_evaluation_patches("eval")
print(frame_eval_patches)

print(frame_eval_patches.count_values("type"))
# {'tp': 10578, 'fn': 767, 'fp': 767}

session = fo.launch_app(view=frame_eval_patches)

Dataset:     video-eval-demo
Media type:  image
Num patches: 12112
Patch fields:
    id:               fiftyone.core.fields.ObjectIdField
    sample_id:        fiftyone.core.fields.ObjectIdField
    frame_id:         fiftyone.core.fields.ObjectIdField
    filepath:         fiftyone.core.fields.StringField
    frame_number:     fiftyone.core.fields.FrameNumberField
    tags:             fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:         fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.ImageMetadata)
    created_at:       fiftyone.core.fields.DateTimeField
    last_modified_at: fiftyone.core.fields.DateTimeField
    predictions:      fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    detections:       fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
    type:             fiftyone.core.fields.StringField
    iou:              fiftyone.core.fields.FloatField
    crowd:            fiftyone.core.fields.BooleanField
View stages:
    1. ToFrames(config=None)
    2. ToEvaluationPatches(eval_key='eval', config=None)

Custom evaluation metrics

Available in: Open SourceEnterprise

Introduced in FiftyOne 1.3.0 · FiftyOne Enterprise 2.5.0

You can add custom metrics to your evaluation runs in FiftyOne.

Custom metrics are supported by all FiftyOne evaluation methods, and you can compute them via the SDK, or directly from the App if you’re running FiftyOne Enterprise.

Using custom metrics

The example below shows how to compute a custom metric from the metric-examples plugin when evaluating object detections:

# Install the example metrics plugin
fiftyone plugins download \
    https://github.com/voxel51/fiftyone-plugins \
    --plugin-names @voxel51/metric-examples

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Custom metrics are specified via their operator URI
metric_uri = "@voxel51/metric-examples/example_metric"

# Custom metrics can optionally accept kwargs that configure their behavior
metric_kwargs = dict(value="spam")

results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
    custom_metrics={metric_uri: metric_kwargs},
)

# Custom metrics may populate new fields on each sample
dataset.count_values("eval_example_metric")
# {'spam': 200}

# Custom metrics may also compute an aggregate value, which is included in
# the run's metrics report
results.print_metrics()
"""
accuracy   0.25
precision  0.26
recall     0.86
fscore     0.40
support    1735
example    spam  # the custom metric
"""

#
# Launch the app
#
# Open the Model Evaluation panel and you'll see the "Example metric" in
# the Summary table
#
session = fo.launch_app(dataset)

# Deleting an evaluation automatically deletes any custom metrics
# associated with it
dataset.delete_evaluation("eval")
assert not dataset.has_field("eval_example_metric")

When using metric operators without custom parameters, you can also pass a list of operator URI’s to the custom_metrics parameter:

# Apply two custom metrics to a regression evaluation
results = dataset.evaluate_regressions(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
    custom_metrics=[
        "@voxel51/metric-examples/absolute_error",
        "@voxel51/metric-examples/squared_error",
    ],
)

You can also add custom metrics to an existing evaluation at any time via add_custom_metrics():

# Load an existing evaluation run
results = dataset.load_evaluation_results("eval")

# Add some custom metrics
results.add_custom_metrics(
    [
        "@voxel51/metric-examples/absolute_error",
        "@voxel51/metric-examples/squared_error",
    ]
)

Developing custom metrics

Each custom metric is implemented as an operator that implements the EvaluationMetric interface.

Let’s look at an example evaluation metric operator:

import fiftyone.operators as foo
from fiftyone.operators import types

class ExampleMetric(foo.EvaluationMetric):
    @property
    def config(self):
        return foo.EvaluationMetricConfig(
            # The metric's URI: f"{plugin_name}/{name}"
            name="example_metric",  # required

            # The display name of the metric in the Summary table of the
            # Model Evaluation panel
            label="Example metric",  # required

            # A description for the operator
            description="An example evaluation metric",  # optional

            # List of evaluation types that the metrics supports
            # EG: ["regression", "classification", "detection", ...]
            # If omitted, the metric may be applied to any evaluation
            eval_types=None,  # optional

            # An optional custom key under which the metric's aggregate
            # value is stored and returned in methods like `metrics()`
            # If omitted, the metric's `name` is used
            aggregate_key="example",  # optional

            # Metrics are generally not designed to be directly invoked
            # via the Operator browser, so they should be unlisted
            unlisted=True,  # required
        )

    def resolve_input(self, ctx, inputs):
        """You can optionally implement this method to collect user input
        for the metric's parameters in the App.

        Returns:
            a :class:`fiftyone.operators.types.Property`, or None
        """
        inputs = types.Object()
        inputs.str(
            "value",
            label="Example value",
            description="The example value to store/return",
            default="foo",
            required=True,
        )
        return types.Property(inputs)

    def compute(self, samples, results, value="foo"):
        """All metric operators must implement this method. It defines the
        computation done by the metric and which per-frame and/or
        per-sample fields store the computed value.

        This method can return None or the aggregate metric value. The
        aggregrate metric value is included in the result's `metrics()`
        and displayed in the Summary table of the Model Evaluation panel.
        """
        dataset = samples._dataset
        eval_key = results.key
        metric_field = f"{eval_key}_{self.config.name}"
        dataset.add_sample_field(metric_field, fo.StringField)
        samples.set_field(metric_field, value).save()

        return value

    def get_fields(self, samples, config, eval_key):
        """Lists the fields that were populated by the evaluation metric
        with the given key, if any.
        """
        return [f"{eval_key}_{self.config.name}"]

Note

By convention, evaluation metrics should include f"{eval_key}" in any sample fields that they populate. If your metric populates fields whose names do not contain the evaluation key, then you must also implement rename() and cleanup() so that they are properly handled when renaming/deleting evaluation runs.

Custom evaluation backends

Available in: Open SourceEnterprise

Introduced in FiftyOne 0.23.0 · FiftyOne Enterprise 1.5.0

If you would like to use an evaluation protocol that is not natively supported by FiftyOne, you can follow the instructions below to implement an interface for your protocol and then configure your environment so that FiftyOne’s evaluation methods will use it.

RegressionClassificationDetectionSegmentation

You can define custom regression evaluation backends that can be used by passing the method parameter to evaluate_regressions():

view.evaluate_regressions(..., method="<backend>", ...)

Regression evaluation backends are defined by writing subclasses of the following two classes:

• RegressionEvaluation: this class implements the evaluation protocol itself. Specifically you should implement evaluate_samples(), which accepts a sample collection to evaluate as input and returns a RegressionResults instance that contains the results of the evaluation

• RegressionEvaluationConfig: this class defines the available parameters that users can pass as keyword arguments to evaluate_regressions() to customize the behavior of the evaluation run

If desired, you can also implement and return a custom RegressionResults subclass. This is useful if you want to expose custom methods that users can call to view and/or interact with the evaluation results programmatically.

The recommended way to expose a custom regression evaluation method is to add it to your evaluation config at ~/.fiftyone/evaluation_config.json as follows:

{
    "default_regression_backend": "<backend>",
    "regression_backends": {
        "<backend>": {
            "config_cls": "your.custom.RegressionEvaluationConfig"
        }
    },
    ...
}

In the above, <backend> defines the name of your custom backend, which you can henceforward pass as the method parameter to evaluate_regressions(), and the config_cls parameter specifies the fully-qualified name of the RegressionEvaluationConfig subclass for your evaluation backend.

With the optional default_regression_backend parameter set to your custom backend as shown above, calling evaluate_regressions() will automatically use your backend.

Evaluation config

FiftyOne provides an evaluation config that you can use to either temporarily or permanently configure the behavior of the evaluation API.

Viewing your config

You can print your current evaluation config at any time via the Python library and the CLI:

PythonCLI

import fiftyone as fo

# Print your current evaluation config
print(fo.evaluation_config)

{
    "default_regresion_backend": "simple",
    "default_classification_backend": "simple",
    "default_detection_backend": "coco",
    "default_segmentation_backend": "simple",
    "regression_backends": {
        "simple": {
            "config_cls": "fiftyone.utils.eval.regression.SimpleEvaluationConfig"
        }
    },
    "classification_backends": {
        "binary": {
            "config_cls": "fiftyone.utils.eval.classification.BinaryEvaluationConfig"
        },
        "simple": {
            "config_cls": "fiftyone.utils.eval.classification.SimpleEvaluationConfig"
        },
        "top-k": {
            "config_cls": "fiftyone.utils.eval.classification.TopKEvaluationConfig"
        }
    },
    "detection_backends": {
        "activitynet": {
            "config_cls": "fiftyone.utils.eval.activitynet.ActivityNetEvaluationConfig"
        },
        "coco": {
            "config_cls": "fiftyone.utils.eval.coco.COCOEvaluationConfig"
        },
        "open-images": {
            "config_cls": "fiftyone.utils.eval.openimages.OpenImagesEvaluationConfig"
        }
    },
    "segmentation_backends": {
        "simple": {
            "config_cls": "fiftyone.utils.eval.segmentation.SimpleEvaluationConfig"
        }
    }
}

Note

If you have customized your evaluation config via any of the methods described below, printing your config is a convenient way to ensure that the changes you made have taken effect as you expected.

Modifying your config

You can modify your evaluation config in a variety of ways. The following sections describe these options in detail.

Order of precedence

The following order of precedence is used to assign values to your evaluation config settings as runtime:

1. Config settings applied at runtime by directly editing fiftyone.evaluation_config

2. FIFTYONE_XXX environment variables

3. Settings in your JSON config (~/.fiftyone/evaluation_config.json)

4. The default config values

Editing your JSON config

You can permanently customize your evaluation config by creating a ~/.fiftyone/evaluation_config.json file on your machine. The JSON file may contain any desired subset of config fields that you wish to customize.

For example, the following config JSON file declares a new custom detection evaluation backend without changing any other default config settings:

{
    "default_detection_backend": "custom",
    "detection_backends": {
        "custom": {
            "config_cls": "path.to.your.CustomDetectionEvaluationConfig"
        }
    }
}

When fiftyone is imported, any options from your JSON config are merged into the default config, as per the order of precedence described above.

Note

You can customize the location from which your JSON config is read by setting the FIFTYONE_EVALUATION_CONFIG_PATH environment variable.

Setting environment variables

Evaluation config settings may be customized on a per-session basis by setting the FIFTYONE_<TYPE>_XXX environment variable(s) for the desired config settings, where <TYPE> can be REGRESSION, CLASSIFICATION, DETECTION, or SEGMENTATION.

The FIFTYONE_DEFAULT_<TYPE>_BACKEND environment variables allows you to configure your default backend:

export FIFTYONE_DEFAULT_DETECTION_BACKEND=coco

You can declare parameters for specific evaluation backends by setting environment variables of the form FIFTYONE_<TYPE>_<BACKEND>_<PARAMETER>. Any settings that you declare in this way will be passed as keyword arguments to methods like evaluate_detections() whenever the corresponding backend is in use:

export FIFTYONE_DETECTION_COCO_ISCROWD=is_crowd

The FIFTYONE_<TYPE>_BACKENDS environment variables can be set to a list,of,backends that you want to expose in your session, which may exclude native backends and/or declare additional custom backends whose parameters are defined via additional config modifications of any kind:

export FIFTYONE_DETECTION_BACKENDS=custom,coco,open-images

When declaring new backends, you can include * to append new backend(s) without omitting or explicitly enumerating the builtin backends. For example, you can add a custom detection evaluation backend as follows:

export FIFTYONE_DETECTION_BACKENDS=*,custom
export FIFTYONE_DETECTION_CUSTOM_CONFIG_CLS=your.custom.DetectionEvaluationConfig

Modifying your config in code

You can dynamically modify your evaluation config at runtime by directly editing the fiftyone.evaluation_config object.

Any changes to your evaluation config applied via this manner will immediately take effect in all subsequent calls to fiftyone.evaluation_config during your current session.

import fiftyone as fo

fo.evaluation_config.default_detection_backend = "custom"