Run in Google Colab   View source on GitHub   Download notebook
# Convert Dataset Formats This recipe demonstrates how to use FiftyOne to convert datasets on disk between common formats. ## Setup If you haven’t already, install FiftyOne: This notebook contains bash commands. To run it as a notebook, you must install the [Jupyter bash kernel](https://github.com/takluyver/bash_kernel) via the command below. Alternatively, you can just copy + paste the code blocks into your shell. In this recipe we’ll use the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) to download some open source datasets to work with. Specifically, we’ll need [TensorFlow](https://www.tensorflow.org/) and [TensorFlow Datasets](https://www.tensorflow.org/datasets) installed to [access the datasets](https://docs.voxel51.com/dataset_zoo/api.html#customizing-your-ml-backend): ## Download datasets Download the test split of the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html) from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) using the command below: Download the validation split of the [KITTI dataset](http://www.cvlibs.net/datasets/kitti) from the [FiftyOne Dataset Zoo](https://docs.voxel51.com/dataset_zoo/index.html) using the command below: ## The fiftyone convert command The [FiftyOne CLI](https://voxel51.com/docs/fiftyone/cli/index.html) provides a number of utilities for importing and exporting datasets in a variety of common (or custom) formats. Specifically, the `fiftyone convert` command provides a convenient way to convert datasets on disk between formats by specifying the [fiftyone.types.Dataset](https://voxel51.com/docs/fiftyone/api/fiftyone.types.html#fiftyone.types.dataset_types.Dataset) type of the input and desired output. FiftyOne provides a collection of [builtin types](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#supported-import-formats) that you can use to read/write datasets in common formats out-of-the-box:
| Dataset format | Import Supported? | Export Supported? | Conversion Supported? | |--------------------------------------------------------------------------------------------------------------------------------------------------|---------------------|---------------------|-------------------------| | [Image Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageDirectory-import) | ✓ | ✓ | ✓ | | [Video Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#VideoDirectory-import) | ✓ | ✓ | ✓ | | [Media Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#MediaDirectory-import) | ✓ | ✓ | ✓ | | [Image Classification Directory Tree](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageClassificationDirectoryTree-import) | ✓ | ✓ | ✓ | | [FiftyOne Image Classification](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageClassificationDataset-import) | ✓ | ✓ | ✓ | | [TF Image Classification](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#TFImageClassificationDataset-import) | ✓ | ✓ | ✓ | | [COCO](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#COCODetectionDataset-import) | ✓ | ✓ | ✓ | | [VOC](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#VOCDetectionDataset-import) | ✓ | ✓ | ✓ | | [KITTI](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#KITTIDetectionDataset-import) | ✓ | ✓ | ✓ | | [YOLOv4](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#YOLOv4Dataset-import) | ✓ | ✓ | ✓ | | [YOLOv5](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#YOLOv5Dataset-import) | ✓ | ✓ | ✓ | | [FiftyOne Object Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageDetectionDataset-import) | ✓ | ✓ | ✓ | | [FiftyOne Temporal Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneTemporalDetectionDataset-import) | ✓ | ✓ | ✓ | | [TF Object Detection](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#TFObjectDetectionDataset-import) | ✓ | ✓ | ✓ | | [Image Segmentation Directory](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#ImageSegmentationDirectory-import) | ✓ | ✓ | ✓ | | [CVAT Image](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CVATImageDataset-import) | ✓ | ✓ | ✓ | | [CVAT Video](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CVATVideoDataset-import) | ✓ | ✓ | ✓ | | [OpenLABEL Image](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#OpenLABELImageDataset-import) | ✓ | | | | [OpenLABEL Video](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#OpenLABELVideoDataset-import) | ✓ | | | | [BDD](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#BDDDataset-import) | ✓ | ✓ | ✓ | | [CSV](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#CSVDataset-import) | ✓ | ✓ | ✓ | | [DICOM](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#DICOMDataset-import) | ✓ | ✓ | ✓ | | [GeoJSON](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#GeoJSONDataset-import) | ✓ | ✓ | ✓ | | [GeoTIFF](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#GeoTIFFDataset-import) | ✓ | | | | [FiftyOne Dataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneDataset-import) | ✓ | ✓ | ✓ | | [FiftyOne Image Labels](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageLabelsDataset-import) | ✓ | ✓ | ✓ | | [FiftyOne Video Labels](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneVideoLabelsDataset-import) | ✓ | ✓ | ✓ |
In addition, you can define your own [custom dataset types](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#custom-dataset-importer) to read/write datasets in your own formats. The usage of the `fiftyone convert` command is as follows: ## Convert CIFAR-10 dataset When you downloaded the test split of the CIFAR-10 dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageClassificationDataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageClassificationDataset-import) format. You can verify this by printing information about the downloaded dataset: The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.ImageClassificationDirectoryTree](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#imageclassificationdirectorytree) format, which stores classification datasets on disk in a directory tree structure with images organized per-class: ```none ├── / │ ├── . │ ├── . │ └── ... ├── / │ ├── . │ ├── . │ └── ... └── ... ``` Let’s verify that the conversion happened as expected: Now let’s convert the classification directory tree to [TFRecords](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#tfimageclassificationdataset) format! Let’s verify that the conversion happened as expected: ## Convert KITTI dataset When you downloaded the validation split of the KITTI dataset above, it was written to disk as a dataset in [fiftyone.types.FiftyOneImageDetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/import_datasets.html#FiftyOneImageDetectionDataset-import) format. You can verify this by printing information about the downloaded dataset: The snippet below uses `fiftyone convert` to convert the test split of the CIFAR-10 dataset to [fiftyone.types.COCODetectionDataset](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cocodetectiondataset) format, which writes the dataset to disk with annotations in [COCO format](https://cocodataset.org/#format-data). Let’s verify that the conversion happened as expected: Now let’s convert from COCO format to [CVAT Image format](https://voxel51.com/docs/fiftyone/user_guide/export_datasets.html#cvatimageformat) format! Let’s verify that the conversion happened as expected: ## Cleanup You can cleanup the files generated by this recipe by running the command below: