fiftyone.utils.bdd

Utilities for working with datasets in Berkeley DeepDrive (BDD) format.

Copyright 2017-2024, Voxel51, Inc.

voxel51.com

Classes:

BDDDatasetImporter([dataset_dir, data_path, …])	Importer for BDD datasets stored on disk.
BDDDatasetExporter([export_dir, data_path, …])	Exporter that writes BDD datasets to disk.

Functions:

load_bdd_annotations(json_path)	Loads the BDD annotations from the given JSON file.
parse_bdd100k_dataset(source_dir, dataset_dir)	Parses the raw BDD100K download files in the specified directory into per-split directories in BDD format.

class fiftyone.utils.bdd.BDDDatasetImporter(dataset_dir=None, data_path=None, labels_path=None, include_all_data=False, extra_attrs=True, shuffle=False, seed=None, max_samples=None)

Bases: fiftyone.utils.data.importers.LabeledImageDatasetImporter, fiftyone.utils.data.importers.ImportPathsMixin

Importer for BDD datasets stored on disk.

See this page for format details.

Parameters

• dataset_dir (None) – the dataset directory. If omitted, data_path and/or labels_path must be provided

• data_path (None) –

  an optional parameter that enables explicit control over the location of the media. Can be any of the following:

  • a folder name like "data" or "data/" specifying a subfolder of dataset_dir where the media files reside

  • an absolute directory path where the media files reside. In this case, the dataset_dir has no effect on the location of the data

  • a filename like "data.json" specifying the filename of the JSON data manifest file in dataset_dir

  • an absolute filepath specifying the location of the JSON data manifest. In this case, dataset_dir has no effect on the location of the data

  • a dict mapping filenames to absolute filepaths

  If None, this parameter will default to whichever of data/ or data.json exists in the dataset directory

• labels_path (None) –

  an optional parameter that enables explicit control over the location of the labels. Can be any of the following:

  • a filename like "labels.json" specifying the location of the labels in dataset_dir

  • an absolute filepath to the labels. In this case, dataset_dir has no effect on the location of the labels

  If None, the parameter will default to labels.json

• include_all_data (False) – whether to generate samples for all images in the data directory (True) rather than only creating samples for images with label entries (False)

• extra_attrs (True) –

  whether to load extra annotation attributes onto the imported labels. Supported values are:

  • True: load all extra attributes found

  • False: do not load extra attributes

  • a name or list of names of specific attributes to load

• shuffle (False) – whether to randomly shuffle the order in which the samples are imported

• seed (None) – a random seed to use when shuffling

• max_samples (None) – a maximum number of samples to import. By default, all samples are imported

Attributes:

has_dataset_info	Whether this importer produces a dataset info dictionary.
has_image_metadata	Whether this importer produces fiftyone.core.metadata.ImageMetadata instances for each image.
label_cls	The fiftyone.core.labels.Label class(es) returned by this importer.

Methods:

setup()	Performs any necessary setup before importing the first sample in the dataset.
close(*args)	Performs any necessary actions after the last sample has been imported.
get_dataset_info()	Returns the dataset info for the dataset.

property has_dataset_info

Whether this importer produces a dataset info dictionary.

property has_image_metadata

Whether this importer produces fiftyone.core.metadata.ImageMetadata instances for each image.

property label_cls

The fiftyone.core.labels.Label class(es) returned by this importer.

This can be any of the following:

• a fiftyone.core.labels.Label class. In this case, the importer is guaranteed to return labels of this type

• a list or tuple of fiftyone.core.labels.Label classes. In this case, the importer can produce a single label field of any of these types

• a dict mapping keys to fiftyone.core.labels.Label classes. In this case, the importer will return label dictionaries with keys and value-types specified by this dictionary. Not all keys need be present in the imported labels

• None. In this case, the importer makes no guarantees about the labels that it may return

setup()

Performs any necessary setup before importing the first sample in the dataset.

This method is called when the importer’s context manager interface is entered, DatasetImporter.__enter__().

close(*args)

Performs any necessary actions after the last sample has been imported.

This method is called when the importer’s context manager interface is exited, DatasetImporter.__exit__().

Parameters

*args – the arguments to DatasetImporter.__exit__()

get_dataset_info()

Returns the dataset info for the dataset.

By convention, this method should be called after all samples in the dataset have been imported.

Returns

a dict of dataset info

class fiftyone.utils.bdd.BDDDatasetExporter(export_dir=None, data_path=None, labels_path=None, export_media=None, rel_dir=None, abs_paths=False, image_format=None, extra_attrs=True)

Bases: fiftyone.utils.data.exporters.LabeledImageDatasetExporter, fiftyone.utils.data.exporters.ExportPathsMixin

Exporter that writes BDD datasets to disk.

See this page for format details.

Parameters

• export_dir (None) – the directory to write the export. This has no effect if data_path and labels_path are absolute paths

• data_path (None) –

  an optional parameter that enables explicit control over the location of the exported media. Can be any of the following:

  • a folder name like "data" or "data/" specifying a subfolder of export_dir in which to export the media

  • an absolute directory path in which to export the media. In this case, the export_dir has no effect on the location of the data

  • a JSON filename like "data.json" specifying the filename of the manifest file in export_dir generated when export_media is "manifest"

  • an absolute filepath specifying the location to write the JSON manifest file when export_media is "manifest". In this case, export_dir has no effect on the location of the data

  If None, the default value of this parameter will be chosen based on the value of the export_media parameter

• labels_path (None) –

  an optional parameter that enables explicit control over the location of the exported labels. Can be any of the following:

  • a filename like "labels.json" specifying the location in export_dir in which to export the labels

  • an absolute filepath to which to export the labels. In this case, the export_dir has no effect on the location of the labels

  If None, the labels will be exported into export_dir using the default filename

• export_media (None) –

  controls how to export the raw media. The supported values are:

  • True: copy all media files into the output directory

  • False: don’t export media

  • "move": move all media files into the output directory

  • "symlink": create symlinks to the media files in the output directory

  • "manifest": create a data.json in the output directory that maps UUIDs used in the labels files to the filepaths of the source media, rather than exporting the actual media

  If None, the default value of this parameter will be chosen based on the value of the data_path parameter

• rel_dir (None) – an optional relative directory to strip from each input filepath to generate a unique identifier for each image. When exporting media, this identifier is joined with data_path to generate an output path for each exported image. This argument allows for populating nested subdirectories that match the shape of the input paths. The path is converted to an absolute path (if necessary) via fiftyone.core.storage.normalize_path()

• abs_paths (False) – whether to store absolute paths to the images in the exported labels

• image_format (None) – the image format to use when writing in-memory images to disk. By default, fiftyone.config.default_image_ext is used

• extra_attrs (True) –

  whether to include extra object attributes in the exported labels. Supported values are:

  • True: export all extra attributes found

  • False: do not export extra attributes

  • a name or list of names of specific attributes to export

Attributes:

requires_image_metadata	Whether this exporter requires fiftyone.core.metadata.ImageMetadata instances for each sample being exported.
label_cls	The fiftyone.core.labels.Label class(es) exported by this exporter.

Methods:

setup()	Performs any necessary setup before exporting the first sample in the dataset.
export_sample(image_or_path, labels[, metadata])	Exports the given sample to the dataset.
close(*args)	Performs any necessary actions after the last sample has been exported.
log_collection(sample_collection)	Logs any relevant information about the fiftyone.core.collections.SampleCollection whose samples will be exported.

property requires_image_metadata

Whether this exporter requires fiftyone.core.metadata.ImageMetadata instances for each sample being exported.

property label_cls

The fiftyone.core.labels.Label class(es) exported by this exporter.

This can be any of the following:

• a fiftyone.core.labels.Label class. In this case, the exporter directly exports labels of this type

• a list or tuple of fiftyone.core.labels.Label classes. In this case, the exporter can export a single label field of any of these types

• a dict mapping keys to fiftyone.core.labels.Label classes. In this case, the exporter can handle label dictionaries with value-types specified by this dictionary. Not all keys need be present in the exported label dicts

• None. In this case, the exporter makes no guarantees about the labels that it can export

setup()

Performs any necessary setup before exporting the first sample in the dataset.

This method is called when the exporter’s context manager interface is entered, DatasetExporter.__enter__().

export_sample(image_or_path, labels, metadata=None)

Exports the given sample to the dataset.

Parameters

• image_or_path – an image or the path to the image on disk

• label – an instance of label_cls(), or a dictionary mapping field names to fiftyone.core.labels.Label instances, or None if the sample is unlabeled

• metadata (None) – a fiftyone.core.metadata.ImageMetadata instance for the sample. Only required when requires_image_metadata() is True

close(*args)

Performs any necessary actions after the last sample has been exported.

This method is called when the exporter’s context manager interface is exited, DatasetExporter.__exit__().

Parameters

*args – the arguments to DatasetExporter.__exit__()

log_collection(sample_collection)

Logs any relevant information about the fiftyone.core.collections.SampleCollection whose samples will be exported.

Subclasses can optionally implement this method if their export format can record information such as the fiftyone.core.collections.SampleCollection.info() of the collection being exported.

By convention, this method must be optional; i.e., if it is not called before the first call to export_sample(), then the exporter must make do without any information about the fiftyone.core.collections.SampleCollection (which may not be available, for example, if the samples being exported are not stored in a collection).

Parameters

sample_collection – the fiftyone.core.collections.SampleCollection whose samples will be exported

fiftyone.utils.bdd.load_bdd_annotations(json_path)

Loads the BDD annotations from the given JSON file.

See this page for format details.

Parameters

json_path – the path to the annotations JSON file

Returns

a dict mapping filenames to BDD annotation dicts

fiftyone.utils.bdd.parse_bdd100k_dataset(source_dir, dataset_dir, copy_files=True, overwrite=False)

Parses the raw BDD100K download files in the specified directory into per-split directories in BDD format.

This function assumes that the input source_dir contains the following contents:

source_dir/
    labels/
        bdd100k_labels_images_train.json
        bdd100k_labels_images_val.json
    images/
        100k/
            train/
            test/
            val/
    ...

and will populate dataset_dir as follows:

dataset_dir/
    train/
        data/
        labels.json
    validation/
        data/
        labels.json
    test/
        data/

Parameters

• source_dir – the source directory containing the manually downloaded BDD100K files

• dataset_dir – the directory to construct the output split directories

• copy_files (True) – whether to move (False) or create copies (True) of the source files when populating dataset_dir

• overwrite (False) – whether to overwrite existing files/directories in the output location, if they exist

Raises

OSError – if any required source files are not present