Run in Google Colab

View source on GitHub

Download notebook

# Evaluating Object Detections with FiftyOne This walkthrough demonstrates how to use FiftyOne to perform hands-on evaluation of your detection model. It covers the following concepts: - Loading a dataset with ground truth labels [into FiftyOne](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html) - [Adding model predictions](https://voxel51.com/docs/fiftyone/recipes/adding_detections.html) to your dataset - [Evaluating your model](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections) using FiftyOne’s evaluation API - Viewing the best and worst performing samples in your dataset **So, what’s the takeaway?** Aggregate measures of performance like mAP don’t give you the full picture of your detection model. In practice, the limiting factor on your model’s performance is often data quality issues that you need to **see** to address. FiftyOne is designed to make it easy to do just that. Running the workflow presented here on your ML projects will help you to understand the current failure modes (edge cases) of your model and how to fix them, including: - Identifying scenarios that require additional training samples in order to boost your model’s performance - Deciding whether your ground truth annotations have errors/weaknesses that need to be corrected before any subsequent model training will be profitable ## Setup If you haven’t already, install FiftyOne: In this tutorial, we’ll use an off-the-shelf [Faster R-CNN detection model](https://pytorch.org/docs/stable/torchvision/models.html#faster-r-cnn) provided by PyTorch. To use it, you’ll need to install `torch` and `torchvision`, if necessary. If you wanted to, you could download the pretrained model from the web and load it with `torchvision`. However, this model is also available via the [FiftyOne Model Zoo](https://docs.voxel51.com/user_guide/model_zoo/index.html), which makes our lives much easier! We’ll perform our analysis on the validation split of the [COCO dataset](https://cocodataset.org/#home), which is conveniently available for download via the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html#coco-2017). The snippet below will download the validation split and load it into FiftyOne. Let’s inspect the dataset to see what we downloaded: Note that the ground truth detections are stored in the `ground_truth` field of the samples. Before we go further, let’s launch the [FiftyOne App](https://voxel51.com/docs/fiftyone/user_guide/app.html) and use the GUI to explore the dataset visually:  ## Add predictions to dataset Now let’s generate some predictions to analyze. Because we loaded the model from the FiftyOne Model Zoo, it is a FiftyOne model object, which means we can apply it directly to our dataset (or any subset thereof) for inference using the sample collection’s `apply_model()` method. The code below performs inference with the Faster R-CNN model on a randomly chosen subset of 100 samples from the dataset and stores the resulting predictions in a `faster_rcnn` field of the samples. Let’s load `predictions_view` in the App to visualize the predictions that we added:  ## Analyzing detections Let’s analyze the raw predictions we’ve added to our dataset in more detail. ### Visualizing bounding boxes Let’s start by loading the full dataset in the App:  Only the 100 samples in `predictions_view` have predictions in their `faster_rcnn` field, so some of the samples we see above do not have predicted boxes. If we want to recover our predictions view, we can do this programmatically via `session.view = predictions_view`, or we can use the [view bar in the App](https://voxel51.com/docs/fiftyone/user_guide/app.html#using-the-view-bar) to accomplish the same thing:  Each field of the samples are shown as togglable checkboxes on the left sidebar which can be used to control whether ground truth detections or predictions are rendered on the images. You can also click on an image to view the sample in more detail:  ### Selecting samples of interest You can select images in the App by clicking on the checkbox when hovering over an image. Then, you can create a view that contains only those samples by clicking the orange checkmark with the number of selected samples in the top left corner of the sample grid and clicking `Only show selected samples`.  Let’s reset our session to show our `predictions_view`: ### Confidence thresholding in the App From the App instance above, it looks like our detector is generating some spurious low-quality detections. Let’s use the App to interactively filter the predictions by `confidence` to identify a reasonable confidence threshold for our model:  It looks like a confidence threshold of 0.75 is a good choice for our model, but we’ll confirm that quantitatively later. ### Confidence thresholding in Python FiftyOne also provides the ability to [write expressions](https://voxel51.com/docs/fiftyone/user_guide/using_views.html#filtering) that match, filter, and sort detections based on their attributes. See [using DatasetViews](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) for full details. For example, we can programmatically generate a view that contains only detections whose `confidence` is at least `0.75` as follows: Note the `only_matches=False` argument. When filtering labels, any samples that no longer contain labels would normally be removed from the view. However, this is not desired when performing evaluations since it can skew your results between views. We set `only_matches=False` so that all samples will be retained, even if some no longer contain labels. Now let’s load our view in the App to view the predictions that we programmatically selected:  ### Viewing object patches There are multiple situations where it can be useful to visualize each object separately. For example, if a sample contains dozens of objects overlapping one another or if you want to look specifically for instances of a class of objects. In any case, the FiftyOne App provides a [patches view button](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-object-patches) that allows you to take any `Detections` field in your dataset and visualize each object as an individual patch in the image grid.  ## Evaluate detections Now that we have samples with ground truth and predicted objects, let’s use FiftyOne to evaluate the quality of the detections. FiftyOne provides a powerful [evaluation API](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html) that contains a collection of methods for performing evaluation of model predictions. Since we’re working with object detections here, we’ll use [detection evaluation](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections). ### Running evaluation We can run evaluation on our samples via [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections). Note that this method is available on both the `Dataset` and `DatasetView` classes, which means that we can run evaluation on our `high_conf_view` to assess the quality of only the high confidence predictions in our dataset. By default, this method will use the [COCO evaluation protocol](https://cocodataset.org/#detection-eval), plus some extra goodies that we will use later. ### Aggregate results The `results` object returned by the evaluation routine provides a number of convenient methods for analyzing our predictions. For example, let’s print a classification report for the top-10 most common classes in the dataset: We can also compute the mean average-precision (mAP) of our detector: Since [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections) uses the official [COCO evaluation protocol](https://cocodataset.org/#detection-eval), this mAP value will match what `pycocotools` would report. We can also view some precision-recall (PR) curves for specific classes of our model: Install ipywidgets to view the PR curves: ### Sample-level analysis The evaluation routine also populated some new fields on our dataset that contain helpful information that we can use to evaluate our predictions at the sample-level. In particular, each sample now contains new fields: - `eval_tp`: the number of true positive (TP) predictions in the sample - `eval_fp`: the number of false positive (FP) predictions in the sample - `eval_fn`: the number of false negative (FN) predictions in the sample The individual predicted and ground truth objects also have fields populated on them describing the results of the matching process: - `eval`: whether the object is a TP/FP/FN - `eval_id`: the ID of the matching ground truth/predicted object, if any - `eval_iou`: the IoU between the matching objects, if any These extra fields were added because we provided the `eval_key` parameter to [evaluate_detections()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.evaluate_detections). If we had omitted this parameter, then no information would have been recorded on our samples. Don’t worry, if you forget what evaluations you’ve run, you can retrieve information about the evaluation later: You can even load the view on which you ran an evaluation by calling the [load_evaluation_view()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html?highlight=load_evaluation_view#fiftyone.core.collections.SampleCollection.load_evaluation_view) method on the parent dataset: Finally, you can delete an evaluation from a dataset, including any information that was added to your samples, by calling [delete_evaluation()](https://voxel51.com/docs/fiftyone/api/fiftyone.core.collections.html#fiftyone.core.collections.SampleCollection.delete_evaluation). ### Evaluation views So, now that we have a sense for the aggregate performance of our model, let’s dive into sample-level analysis by creating an [evaluation view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-evaluation-patches). Any evaluation that you stored on your dataset can be used to generate an [evaluation view](https://voxel51.com/docs/fiftyone/user_guide/app.html#viewing-evaluation-patches) that is a patches view creating a sample for every true positive, false positive, and false negative in your dataset. Through this view, you can quickly filter and sort evaluated detections by their type (TP/FP/FN), evaluated IoU, and if they are matched to a crowd object. These evaluation views can be created through Python or directly in the App as shown below.  Let’s use this evaluation view to find individual false positive detections with a confidence of 0.85 or greater.  ### View the best-performing samples To dig in further, let’s create a view that sorts by `eval_tp` so we can see the best-performing cases of our model (i.e., the samples with the most correct predictions):  ### View the worst-performing samples Similarly, we can sort by the `eval_fp` field to see the worst-performing cases of our model (i.e., the samples with the most false positive predictions):  ### Filtering by bounding box area [Dataset views](https://voxel51.com/docs/fiftyone/user_guide/using_views.html) are extremely powerful. For example, let’s look at how our model performed on small objects by creating a view that contains only predictions whose bounding box area is less than `32^2` pixels:  We can always re-run evaluation to see how our detector fairs on only small boxes: ### Viewing detections in a crowd If you’re familiar with the [COCO data format](https://cocodataset.org/#format-data), you’ll know that the ground truth annotations have an `iscrowd = 0/1` attribute that indicates whether a box contains multiple instances of the same object. Let’s create a view that contains only samples with at least one detection for which `iscrowd` is 1:  ### More complex insights Let’s combine our previous operations to form more complex queries that provide deeper insight into the quality of our detections. For example, let’s sort our view of crowded images from the previous section in decreasing order of false positive counts, so that we can see samples that have many (allegedly) spurious predictions in images that are known to contain crowds of objects:  Let’s compare the above view to another view that just sorts by false positive count, regardless of whether the image is crowded:  This was one of the first images in the view. As we can see, while the evaluation is detecting $7$ false positives, all of the model’s predictions seem accurate. It is just that the ground truth labels lumped a bunch of orange slices together into one box. **See anything interesting?** What you find will likely be different because a random subset of samples were chosen. In our case, we find missing ground truth boxes for two of the laptop keyboards, a bottle, and even perhaps a cell phone. The model did not confidently predict many of the boxes in this image, but from a high-level, an example like this makes us consider the consequences of including complex or dense images in datasets. It will likely mean incorrect or incomplete ground truth annotations the annotators are not diligent! And that ultimately leads to confused models, and misinformed evaluations. This conclusion would have been nearly impossible to achieve without visually inspecting the individual samples in the dataset according to the variety of criteria that we considered in this tutorial. ### Tagging and next steps In practice, the next step is to take action on the issues that we identified above. A natural first step is to *tag* the issues so they can be retrieved and dealt with later. FiftyOne provides support for [tagging samples and labels](https://voxel51.com/docs/fiftyone/user_guide/app.html#tags-and-tagging), both programmatically and via the App. In your App instance, try tagging the predictions with missing ground truth detections. You can do this by clicking on the boxes of the predictions of interest and using the tagging element in the top-right corner to assign a `possibly-missing` tag. Alternatively, we can programmatically tag a batch of labels by creating a view that contains the objects of interest and then applying [tag_labels()](https://voxel51.com/docs/fiftyone/user_guide/fiftyone.core.collections.html?highlight=tag_labels#fiftyone.core.collections.SampleCollection.tag_labels): These tagged labels could then be sent off to our annotation provider of choice for review and addition to the ground truth labels. FiftyOne currently offers integrations for [Scale AI](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.scale.html), [Labelbox](https://voxel51.com/docs/fiftyone/api/fiftyone.utils.labelbox.html), and [CVAT](https://voxel51.com/docs/fiftyone/api/fiftyone.types.dataset_types.html?highlight=cvat#fiftyone.types.dataset_types.CVATImageDataset). For example, the snippet below exports the tagged labels and their source media to disk in CVAT format: ## Summary In this tutorial, we covered loading a dataset into FiftyOne and analyzing the performance of an out-of-the-box object detection model on the dataset. **So, what’s the takeaway?** Aggregate evaluation results for an object detector are important, but they alone don’t tell the whole story of a model’s performance. It’s critical to study the failure modes of your model so you can take the right actions to improve them. In this tutorial, we covered two types of analysis: - Analyzing the performance of your detector across different strata, like high confidence, small objects in crowded scenes - Inspecting the hardest samples in your dataset to diagnose the underlying issue, whether it be your detector or the ground truth annotations