Run in Google Colab

View source on GitHub

Download notebook

# Training and Evaluating FiftyOne Datasets with Detectron2 FiftyOne has all of the building blocks necessary to develop high-quality datasets to train your models, as well as advanced model evaluation capabilities. To make use of these, FiftyOne easily integrates with your existing model training and inference pipelines. In this walktrhough we’ll cover how you can use your FiftyOne datasets to train a model with [Detectron2](https://github.com/facebookresearch/detectron2), Facebook AI Reasearch’s library for detection and segmentation algorithms. This walkthrough is based off of the [official Detectron2 tutorial](https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5), augmented to load data to and from FiftyOne. Specifically, this walkthrough covers: - Loading a dataset from the FiftyOne Zoo, and splitting it into training/validation - Initializing a segmentation model from the detectron2 model zoo - Loading ground truth annotations from a FiftyOne dataset into a detectron2 model training pipeline and training the model - Loading predictions from a detectron2 model into a FiftyOne dataset - Evaluating model predictions in FiftyOne **So, what’s the takeaway?** By writing two simple functions, you can integrate FiftyOne into your Detectron2 model training and inference pipelines. ## Setup To get started, you need to install [FiftyOne](https://voxel51.com/docs/fiftyone/getting_started/install.html) and [detectron2](https://detectron2.readthedocs.io/en/latest/tutorials/install.html): ## Train on a FiftyOne dataset In this section, we show how to use a custom FiftyOne Dataset to train a detectron2 model. We’ll train a license plate segmentation model from an existing model pre-trained on COCO dataset, available in detectron2’s model zoo. Since the COCO dataset doesn’t have a “Vehicle registration plates” category, we will be using segmentations of license plates from the Open Images v6 dataset in the [FiftyOne Dataset Zoo](https://voxel51.com/docs/fiftyone/user_guide/dataset_zoo/datasets.html#open-images-v6) to train the model to recognize this new category. ## Prepare the dataset For this example, we will just use some of the samples from the official “validation” split of the dataset. To improve model performance, we could always add in more data from the official “train” split as well but that will take longer to train so we’ll just stick to the “validation” split for this walkthrough. Specifying a `classes` when downloading a dataset from the zoo will ensure that only samples with one of the given classes will be present. However, these samples may still contain other labels, so we can use the powerful [filtering capability](https://voxel51.com/docs/fiftyone/user_guide/using_views.html#filtering) of FiftyOne to easily keep only the “Vehicle registration plate” labels. We will also untag these samples as “validation” and create our own split out of them. Next we will register the FiftyOne dataset to detectron2, following the [detectron2 custom dataset tutorial](https://detectron2.readthedocs.io/tutorials/datasets.html). Here, the dataset is in its custom format, therefore we write a function to parse it and prepare it into [detectron2’s standard format](https://detectron2.readthedocs.io/en/latest/tutorials/datasets.html#standard-dataset-dicts). Note: In this example, we are specifically parsing the segmentations into bounding boxes and polylines. This function may require tweaks depending on the model being trained and the data it expects. To verify the dataset is in correct format, let’s visualize the annotations of the training set: ## Load the model and train! Now, let’s fine-tune a COCO-pretrained R50-FPN Mask R-CNN model on the FiftyOne dataset. It takes ~2 minutes to train 300 iterations on a P100 GPU.  ## Inference & evaluation using the trained model Now, let’s run inference with the trained model on the license plate validation dataset. First, let’s create a predictor using the model we just trained: Then, we generate predictions on each sample in the validation set, and convert the outputs from detectron2 to FiftyOne format, then add them to our FiftyOne dataset. Let’s visualize the predictions and take a look at how the model did. We can click the eye icon next to the “val” tag to view all of the validation samples that we ran inference on. From here, we can use the built-in [evaluation methods](https://voxel51.com/docs/fiftyone/user_guide/evaluation.html#detections) provided by FiftyOne. The `evaluate_detections()` method can be used to evaluate the instance segmentations using the `use_masks=True` parameter. We can also use this to compute mAP with the options being the [COCO-style](https://voxel51.com/docs/fiftyone/integrations/coco.html#map-protocol) (default) or [Open Images-style](https://voxel51.com/docs/fiftyone/integrations/open_images.html#map-protocol) mAP protocol. We can use this results object to view the mAP, print an evaluation report, plot PR curves, plot confusion matrices, and more.  From the PR curve we can see that the model is not generating many predictions—resulting in many false negatives—but the predictions that are generated are often fairly accurate. We can also create a view into the dataset looking at high-confidence false positive predictions to understand where the model is going wrong and how to potentially improve it in the future. There are a few samples with false positives like this one that contain plates with characters not from the Latin alphabet indicating we may want to introduce images from a wider range of countries into the training set.