fiftyone.utils.kinetics¶
Utilities for working with the Kinetics dataset <https://deepmind.com/research/open-source/kinetics>.
Classes:
|
|
|
Kinetics 400-specific dataset info. |
|
Kinetics 600-specific dataset info. |
|
Kinetics 700-2020-specific dataset info. |
|
Kinetics 700-specific dataset info. |
|
Clas that downloads and extracts Kinetics tars from AWS. |
|
Class that contains information such as paths, labels, and sample IDs for a Kinetics download. |
|
Class that manages the sample IDs and labels that need to be downloaded as well as performing the actual downloading. |
|
Config class for a Kinetics download run. |
Functions:
|
Utility that downloads full or partial splits of the Kinetics dataset. |
-
fiftyone.utils.kinetics.
download_kinetics_split
(dataset_dir, split, classes=None, num_workers=None, shuffle=None, seed=None, max_samples=None, retry_errors=False, scratch_dir=None, version='700-2020')¶ Utility that downloads full or partial splits of the Kinetics dataset.
The downloaded splits are stored on disk in VideoClassificationDirectoryTree format.
- Parameters
dataset_dir – the directory to download the dataset
split – the split to download. Supported values are
("train", "validation", "test")
classes (None) – a string or list of strings specifying required classes to load. If provided, only samples containing at least one instance of a specified class will be loaded
num_workers (None) – the number of processes to use when downloading individual video. By default,
multiprocessing.cpu_count()
is usedshuffle (False) – whether to randomly shuffle the order in which samples are chosen for partial downloads
seed (None) – a random seed to use when shuffling
max_samples (None) – a maximum number of samples to load per split. If
classes
are also specified, only up to the number of samples that contain at least one specified class will be loaded. By default, all matching samples are loadedretry_errors (False) – whether to retry downloading samples from YouTube that have previously raised an error
scratch_dir (None) – a scratch directory to use to store temporary files
version ("700-2020") – the version of the Kinetics dataset to download (“400”, “600”, “700”, or “700-2020”)
- Returns
a tuple of
num_samples: the total number of downloaded videos, or
None
if everything was already downloadedclasses: the list of all classes, or
None
if everything was already downloadeddid_download: whether any content was downloaded (True) or if all necessary files were already downloaded (False)
-
class
fiftyone.utils.kinetics.
KineticsDatasetManager
(info)¶ Bases:
object
Class that manages the sample IDs and labels that need to be downloaded as well as performing the actual downloading.
Methods:
download
(config, downloader)download_partial_split
(config, downloader)-
download
(config, downloader)¶
-
download_partial_split
(config, downloader)¶
-
-
class
fiftyone.utils.kinetics.
KineticsDatasetDownloader
(num_workers=None)¶ Bases:
object
Clas that downloads and extracts Kinetics tars from AWS.
Methods:
download_classes
(info, classes)download_entire_split
(info)-
download_entire_split
(info)¶
-
download_classes
(info, classes)¶
-
-
class
fiftyone.utils.kinetics.
KineticsDownloadConfig
(split, classes=None, num_workers=None, shuffle=None, seed=None, max_samples=None, retry_errors=False)¶ Bases:
object
Config class for a Kinetics download run.
Attributes:
Methods:
validate
()-
property
load_entire_split
¶
-
validate
()¶
-
validate_split
()¶
-
property
-
class
fiftyone.utils.kinetics.
KineticsDatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
object
Class that contains information such as paths, labels, and sample IDs for a Kinetics download.
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
splits
¶
-
property
version
¶
-
property
supports_classwise_s3_downloads
¶
-
property
raw_dir
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
urls_s3_file
¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
property
error_path
¶
-
property
prev_errors
¶
-
property
loaded_tar_path
¶
-
property
prev_loaded_tars
¶
-
property
multisplit_urls
¶
-
property
split_dir
¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
id_from_filename
(video_fn)¶
-
filename_from_id
(video_id)¶
-
segment_from_id
(video_id)¶
-
url_from_id
(video_id)¶
-
id_from_url
(video_url)¶
-
get_video_class
(video_id)¶
-
cleanup_partial_downloads
()¶
-
cleanup_excess_videos
()¶
-
update_existing_sample_ids
()¶
-
get_incomplete_classes
()¶
-
validate_classes
(classes)¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
property
-
class
fiftyone.utils.kinetics.
Kinetics400DatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
fiftyone.utils.kinetics.KineticsDatasetInfo
Kinetics 400-specific dataset info.
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
supports_classwise_s3_downloads
¶
-
property
version
¶
-
property
multisplit_urls
¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
cleanup_excess_videos
()¶
-
cleanup_partial_downloads
()¶
-
property
error_path
¶
-
filename_from_id
(video_id)¶
-
get_incomplete_classes
()¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
get_video_class
(video_id)¶
-
id_from_filename
(video_fn)¶
-
id_from_url
(video_url)¶
-
property
loaded_tar_path
¶
-
property
prev_errors
¶
-
property
prev_loaded_tars
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
raw_dir
¶
-
segment_from_id
(video_id)¶
-
property
split_dir
¶
-
property
splits
¶
-
update_existing_sample_ids
()¶
-
url_from_id
(video_id)¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
property
urls_s3_file
¶
-
validate_classes
(classes)¶
-
property
-
class
fiftyone.utils.kinetics.
ClasswiseS3KineticsDatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
fiftyone.utils.kinetics.KineticsDatasetInfo
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)class_url
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)unloaded_class_urls
(classes)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
supports_classwise_s3_downloads
¶
-
class_url
(c)¶
-
unloaded_class_urls
(classes)¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
cleanup_excess_videos
()¶
-
cleanup_partial_downloads
()¶
-
property
error_path
¶
-
filename_from_id
(video_id)¶
-
get_incomplete_classes
()¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
get_video_class
(video_id)¶
-
id_from_filename
(video_fn)¶
-
id_from_url
(video_url)¶
-
property
loaded_tar_path
¶
-
property
multisplit_urls
¶
-
property
prev_errors
¶
-
property
prev_loaded_tars
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
raw_dir
¶
-
segment_from_id
(video_id)¶
-
property
split_dir
¶
-
property
splits
¶
-
update_existing_sample_ids
()¶
-
url_from_id
(video_id)¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
property
urls_s3_file
¶
-
validate_classes
(classes)¶
-
property
version
¶
-
property
-
class
fiftyone.utils.kinetics.
Kinetics600DatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
fiftyone.utils.kinetics.ClasswiseS3KineticsDatasetInfo
Kinetics 600-specific dataset info.
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)class_url
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)unloaded_class_urls
(classes)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
version
¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
class_url
(c)¶
-
cleanup_excess_videos
()¶
-
cleanup_partial_downloads
()¶
-
property
error_path
¶
-
filename_from_id
(video_id)¶
-
get_incomplete_classes
()¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
get_video_class
(video_id)¶
-
id_from_filename
(video_fn)¶
-
id_from_url
(video_url)¶
-
property
loaded_tar_path
¶
-
property
multisplit_urls
¶
-
property
prev_errors
¶
-
property
prev_loaded_tars
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
raw_dir
¶
-
segment_from_id
(video_id)¶
-
property
split_dir
¶
-
property
splits
¶
-
property
supports_classwise_s3_downloads
¶
-
unloaded_class_urls
(classes)¶
-
update_existing_sample_ids
()¶
-
url_from_id
(video_id)¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
property
urls_s3_file
¶
-
validate_classes
(classes)¶
-
property
-
class
fiftyone.utils.kinetics.
Kinetics7002020DatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
fiftyone.utils.kinetics.ClasswiseS3KineticsDatasetInfo
Kinetics 700-2020-specific dataset info.
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)class_url
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)unloaded_class_urls
(classes)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
version
¶
-
class_url
(c)¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
cleanup_excess_videos
()¶
-
cleanup_partial_downloads
()¶
-
property
error_path
¶
-
filename_from_id
(video_id)¶
-
get_incomplete_classes
()¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
get_video_class
(video_id)¶
-
id_from_filename
(video_fn)¶
-
id_from_url
(video_url)¶
-
property
loaded_tar_path
¶
-
property
multisplit_urls
¶
-
property
prev_errors
¶
-
property
prev_loaded_tars
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
raw_dir
¶
-
segment_from_id
(video_id)¶
-
property
split_dir
¶
-
property
splits
¶
-
property
supports_classwise_s3_downloads
¶
-
unloaded_class_urls
(classes)¶
-
update_existing_sample_ids
()¶
-
url_from_id
(video_id)¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
property
urls_s3_file
¶
-
validate_classes
(classes)¶
-
property
-
class
fiftyone.utils.kinetics.
Kinetics700DatasetInfo
(kinetics_dir, scratch_dir, split)¶ Bases:
fiftyone.utils.kinetics.Kinetics7002020DatasetInfo
Kinetics 700-specific dataset info.
Methods:
build_for_version
(version, dataset_dir, …)class_dir
(c)class_url
(c)filename_from_id
(video_id)get_kinetics_dir
(dataset_dir)get_video_class
(video_id)id_from_filename
(video_fn)id_from_url
(video_url)raw_anno_path_split
(split)segment_from_id
(video_id)unloaded_class_urls
(classes)url_from_id
(video_id)validate_classes
(classes)Attributes:
-
property
version
¶
-
property
urls_s3_file
¶
-
classmethod
build_for_version
(version, dataset_dir, scratch_dir, split)¶
-
class_dir
(c)¶
-
class_existing_sample_ids
(c)¶
-
class_sample_ids
(c)¶
-
class_url
(c)¶
-
cleanup_excess_videos
()¶
-
cleanup_partial_downloads
()¶
-
property
error_path
¶
-
filename_from_id
(video_id)¶
-
get_incomplete_classes
()¶
-
classmethod
get_kinetics_dir
(dataset_dir)¶
-
get_video_class
(video_id)¶
-
id_from_filename
(video_fn)¶
-
id_from_url
(video_url)¶
-
property
loaded_tar_path
¶
-
property
multisplit_urls
¶
-
property
prev_errors
¶
-
property
prev_loaded_tars
¶
-
property
raw_anno_path
¶
-
raw_anno_path_split
(split)¶
-
property
raw_dir
¶
-
segment_from_id
(video_id)¶
-
property
split_dir
¶
-
property
splits
¶
-
property
supports_classwise_s3_downloads
¶
-
unloaded_class_urls
(classes)¶
-
update_existing_sample_ids
()¶
-
url_from_id
(video_id)¶
-
property
urls_filename
¶
-
property
urls_path
¶
-
validate_classes
(classes)¶
-
property