Evaluating Models

Available in: Open SourceEnterprise

Introduced in FiftyOne 0.7.3 · FiftyOne Enterprise 1.0

FiftyOne provides a variety of builtin methods for evaluating your model predictions, including regressions, classifications, detections, polygons, instance and semantic segmentations, on both image and video datasets.

When you evaluate a model in FiftyOne, you get access to the standard aggregate metrics such as classification reports, confusion matrices, and PR curves for your model. In addition, FiftyOne can also record fine-grained statistics like accuracy and false positive counts at the sample-level, which you can interactively explore in the App to diagnose the strengths and weaknesses of your models on individual data samples.

Sample-level analysis often leads to critical insights that will help you improve your datasets and models. For example, viewing the samples with the most false positive predictions can reveal errors in your annotation schema. Or, viewing the cluster of samples with the lowest accuracy can reveal gaps in your training dataset that you need to address in order to improve your model’s performance. A key goal of FiftyOne is to help you uncover these insights on your data!

Note

Check out the tutorials page for in-depth walkthroughs of evaluating various types of models with FiftyOne, or the Model Evaluation Guide for a complete step-by-step walkthrough.

Overview

FiftyOne’s evaluation methods are conveniently exposed as methods on all Dataset and DatasetView objects, which means that you can evaluate entire datasets or specific views into them via the same syntax.

Let’s illustrate the basic workflow by loading the quickstart dataset and analyzing the object detections in its predictions field using the evaluate_detections() method:

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

# Evaluate the objects in the `predictions` field with respect to the
# objects in the `ground_truth` field
results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
)

session = fo.launch_app(dataset)

Per-class metrics

You can also retrieve and interact with evaluation results via the SDK.

Running an evaluation returns an instance of a task-specific subclass of EvaluationResults that provides a handful of methods for generating aggregate statistics about your dataset.

# Get the 10 most common classes in the dataset
counts = dataset.count_values("ground_truth.detections.label")
classes = sorted(counts, key=counts.get, reverse=True)[:10]

# Print a classification report for the top-10 classes
results.print_report(classes=classes)

               precision    recall  f1-score   support

       person       0.45      0.74      0.56       783
         kite       0.55      0.72      0.62       156
          car       0.12      0.54      0.20        61
         bird       0.63      0.67      0.65       126
       carrot       0.06      0.49      0.11        47
         boat       0.05      0.24      0.08        37
    surfboard       0.10      0.43      0.17        30
traffic light       0.22      0.54      0.31        24
     airplane       0.29      0.67      0.40        24
      giraffe       0.26      0.65      0.37        23

    micro avg       0.32      0.68      0.44      1311
    macro avg       0.27      0.57      0.35      1311
 weighted avg       0.42      0.68      0.51      1311

Note

For details on micro, macro, and weighted averaging, see the sklearn.metrics documentation.

Per-sample metrics

In addition to standard aggregate metrics, when you pass an eval_key parameter to the evaluation routine, FiftyOne will populate helpful task-specific information about your model’s predictions on each sample, such as false negative/positive counts and per-sample accuracies.

Continuing with our example, let’s use dataset views and the FiftyOne App to leverage these sample metrics to investigate the samples with the most false positive predictions in the dataset:

import fiftyone as fo
from fiftyone import ViewField as F

# Create a view that has samples with the most false positives first, and
# only includes false positive boxes in the `predictions` field
view = (
    dataset
    .sort_by("eval_fp", reverse=True)
    .filter_labels("predictions", F("eval") == "fp")
)

# Visualize results in the App
session = fo.launch_app(view=view)

Notice anything wrong? The sample with the most false positives is a plate of carrots where the entire plate has been boxed as a single example in the ground truth while the model is generating predictions for individual carrots!

If you’re familiar with COCO format (which is recognized by evaluate_detections() by default), you’ll notice that the issue here is that the iscrowd attribute of this ground truth annotation has been incorrectly set to 0. Resolving mistakes like these will provide a much more accurate picture of the real performance of a model.

Confusion matrices

Note

The easiest way to work with confusion matrices in FiftyOne is via the Model Evaluation panel!

When you use evaluation methods such as evaluate_detections() that support confusion matrices, you can use the plot_confusion_matrix() method to render responsive plots that can be attached to App instances to interactively explore specific cases of your model’s performance:

# Plot confusion matrix
plot = results.plot_confusion_matrix(classes=classes)
plot.show()

# Connect to session
session.plots.attach(plot)

In this setup, you can click on individual cells of the confusion matrix to select the corresponding ground truth and/or predicted objects in the App. For example, if you click on a diagonal cell of the confusion matrix, you will see the true positive examples of that class in the App.

Likewise, whenever you modify the Session’s view, either in the App or by programmatically setting session.view, the confusion matrix is automatically updated to show the cell counts for only those objects that are included in the current view.

Analyzing scenarios

Available in: Open SourceEnterprise

Introduced in FiftyOne 1.6.0 · FiftyOne Enterprise 2.9.1

Note

Did you know? You can create and analyze model evaluation scenarios in the App via the Scenario Analysis tab.

The use_subset() method allows you to evaluate the performance of your model under specific scenarios, i.e., subsets of the overall dataset on which evaluation was performed.

Consider the following example:

import fiftyone as fo
import fiftyone.zoo as foz
import fiftyone.utils.random as four
from fiftyone import ViewField as F

dataset = foz.load_zoo_dataset("quickstart")

four.random_split(dataset, {"sunny": 0.7, "cloudy": 0.2, "rainy": 0.1})

counts = dataset.count_values("ground_truth.detections.label")
classes = sorted(counts, key=counts.get, reverse=True)[:5]

dataset.save_view("take100", dataset.take(100))

results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
)

By default, invoking methods on an EvaluationResults instance reports statistics across the entire evaluation:

# Full results
results.print_report(classes=classes)

              precision    recall  f1-score   support

      person       0.52      0.94      0.67       716
        kite       0.59      0.88      0.71       140
         car       0.18      0.80      0.29        61
        bird       0.65      0.78      0.71       110
      carrot       0.09      0.74      0.16        47

   micro avg       0.42      0.90      0.57      1074
   macro avg       0.41      0.83      0.51      1074
weighted avg       0.51      0.90      0.64      1074

However, you can use use_subset() to analyze the performance of the model on specific subsets of interest:

Sunny samplesSmall objectsSaved viewCompound subset

# Sunny samples
subset_def = dict(type="field", field="tags", value="sunny")
with results.use_subset(subset_def):
    results.print_report(classes=classes)

              precision    recall  f1-score   support

      person       1.00      0.93      0.96       495
        kite       1.00      0.90      0.95        62
         car       1.00      0.69      0.81        35
        bird       1.00      0.78      0.88       104
      carrot       1.00      0.69      0.82        36

   micro avg       1.00      0.88      0.94       732
   macro avg       1.00      0.80      0.88       732
weighted avg       1.00      0.88      0.94       732

Refer to use_subset() and get_subset_view() for a complete description of the supported syntax for defining subsets to analyze.

Managing evaluations

When you run an evaluation with an eval_key argument, the evaluation is recorded on the dataset and you can retrieve information about it later, rename it, delete it (along with any modifications to your dataset that were performed by it), and retrieve the view that you evaluated on using the following methods on your dataset:

• list_evaluations()

• get_evaluation_info()

• load_evaluation_results()

• load_evaluation_view()

• rename_evaluation()

• delete_evaluation()

The example below demonstrates the basic interface:

# List evaluations you've run on a dataset
dataset.list_evaluations()
# ['eval']

# Print information about an evaluation
print(dataset.get_evaluation_info("eval"))

# Load existing evaluation results and use them
results = dataset.load_evaluation_results("eval")
results.print_report()

# Rename the evaluation
# This will automatically rename any evaluation fields on your dataset
dataset.rename_evaluation("eval", "still_eval")

# Delete the evaluation
# This will remove any evaluation data that was populated on your dataset
dataset.delete_evaluation("still_eval")

Model Evaluation panel

Available in: Open SourceEnterprise

Introduced in FiftyOne 1.1.0 · FiftyOne Enterprise 2.2.0

When you load a dataset in the App that contains one or more evaluations, you can open the Model Evaluation panel to visualize and interactively explore the evaluation results in the App:

Note

Did you know? With FiftyOne Enterprise you can execute model evaluations natively from the App in the background while you work.