fiftyone.core.storage¶

File storage utilities.

Copyright 2017-2023, Voxel51, Inc.
voxel51.com

Classes:

`FileSystem`	Enumeration of the available file systems.
`TempDir`([basedir])	Context manager that creates and destroys a temporary directory.

Functions:

`abspath`(path)	Converts the given path to an absolute path.
`copy_dir`(indir, outdir[, overwrite, …])	Copies the input directory to the output directory.
`copy_file`(inpath, outpath)	Copies the input file to the output location.
`copy_files`(inpaths, outpaths[, …])	Copies the files to the given locations.
`delete_dir`(dirpath)	Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.
`delete_file`(path)	Deletes the file at the given path.
`delete_files`(paths[, skip_failures, progress])	Deletes the files from the given locations.
`ensure_basedir`(path)	Makes the base directory of the given path, if necessary.
`ensure_dir`(dirpath)	Makes the given directory, if necessary.
`ensure_empty_dir`(dirpath[, cleanup])	Ensures that the given directory exists and is empty.
`ensure_local`(path)	Ensures that the given path is local.
`exists`(path)	Determines whether the given file or directory exists.
`extract_archive`(archive_path[, outdir, cleanup])	Extracts the contents of an archive.
`get_bucket_name`(path)	Gets the bucket name from the given path.
`get_file_system`(path)	Returns the file system enum for the given path.
`get_glob_matches`(glob_patt)	Returns a list of file paths matching the given glob pattern.
`get_glob_root`(glob_patt)	Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.
`is_local`(path)	Determines whether the given path is local.
`isabs`(path)	Determines whether the given path is absolute.
`isdir`(dirpath)	Determines whether the given directory exists.
`isfile`(path)	Determines whether the given file exists.
`join`(a, *p)	Joins the given path components into a single path.
`list_available_file_systems`()	Lists the file systems that are currently available for use with methods like `list_files()` and `list_buckets()`.
`list_buckets`(fs[, abs_paths])	Lists the available buckets in the given file system.
`list_files`(dirpath[, abs_paths, recursive, …])	Lists the files in the given directory.
`list_subdirs`(dirpath[, abs_paths, recursive])	Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.
`load_json`(path_or_str)	Loads JSON from the input argument.
`load_ndjson`(path_or_str)	Loads NDJSON from the input argument.
`make_archive`(dirpath, archive_path[, cleanup])	Makes an archive containing the given directory.
`make_temp_dir`([basedir])	Makes a temporary directory.
`move_dir`(indir, outdir[, overwrite, …])	Moves the contents of the given directory into the given output directory.
`move_file`(inpath, outpath)	Moves the given file to a new location.
`move_files`(inpaths, outpaths[, …])	Moves the files to the given locations.
`normalize_path`(path)	Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.
`normpath`(path)	Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.
`open_file`(path[, mode])	Opens the given file for reading or writing.
`read_file`(path[, binary])	Reads the file.
`read_json`(path)	Reads a JSON file.
`read_ndjson`(path)	Reads an NDJSON file.
`read_yaml`(path)	Reads a YAML file.
`run`(fcn, tasks[, num_workers, progress])	Applies the given function to each element of the given tasks.
`sep`(path)	Returns the path separator for the given path.
`split_prefix`(path)	Splits the file system prefix from the given path.
`write_file`(str_or_bytes, path)	Writes the given string/bytes to a file.
`write_json`(d, path[, pretty_print])	Writes JSON object to file.
`write_ndjson`(obj, path)	Writes the list of JSON dicts in NDJSON format.
`write_yaml`(obj, path, **kwargs)	Writes the object to a YAML file.

class fiftyone.core.storage.FileSystem¶

Bases: enum.Enum

Enumeration of the available file systems.

Attributes:

LOCAL

LOCAL = 'local'¶

fiftyone.core.storage.get_file_system(path)¶

Returns the file system enum for the given path.

Parameters: path – a path
Returns: a FileSystem value

fiftyone.core.storage.split_prefix(path)¶

Splits the file system prefix from the given path.

The prefix for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.split_prefix("/path/to/file")       # ('', '/path/to/file')
fos.split_prefix("a/file")              # ('', 'a/file')

Parameters: path – a path
Returns: a (prefix, path) tuple

fiftyone.core.storage.get_bucket_name(path)¶

Gets the bucket name from the given path.

The bucket name for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.get_bucket_name("/path/to/file")       # ''
fos.get_bucket_name("a/file")              # ''

Parameters: path – a path
Returns: the bucket name string

fiftyone.core.storage.is_local(path)¶

Determines whether the given path is local.

Parameters: path – a path
Returns: True/False

fiftyone.core.storage.ensure_local(path)¶

Ensures that the given path is local.

Parameters: path – a path

fiftyone.core.storage.normalize_path(path)¶

Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.

Parameters: path – a path
Returns: the normalized path

fiftyone.core.storage.make_temp_dir(basedir=None)¶

Makes a temporary directory.

Parameters: basedir (None) – an optional directory in which to create the new directory. The default is fiftyone.config.default_dataset_dir
Returns: the temporary directory path

class fiftyone.core.storage.TempDir(basedir=None)¶

Bases: object

Context manager that creates and destroys a temporary directory.

Parameters: basedir (None) – an optional directory in which to create the new directory. The default is fiftyone.config.default_dataset_dir

fiftyone.core.storage.open_file(path, mode='r')¶

Opens the given file for reading or writing.

This function must be used as a context manager.

Example usage:

import fiftyone.core.storage as fos

with fos.open_file("/tmp/file.txt", "w") as f:
    f.write("Hello, world!")

with fos.open_file("/tmp/file.txt", "r") as f:
    print(f.read())

Parameters

path – the path
mode ("r") – the mode. Supported values are ("r", "rb", "w", "wb")

fiftyone.core.storage.read_file(path, binary=False)¶

Reads the file.

Parameters

path – the filepath
binary (False) – whether to read the file in binary mode

Returns

the file contents

fiftyone.core.storage.write_file(str_or_bytes, path)¶

Writes the given string/bytes to a file.

If a string is provided, it is encoded via .encode().

Parameters

str_or_bytes – the string or bytes
path – the filepath

fiftyone.core.storage.sep(path)¶

Returns the path separator for the given path.

For local paths, os.path.sep is returned.

For remote paths, "/" is returned.

Parameters: path – the filepath
Returns: the path separator

fiftyone.core.storage.join(a, *p)¶

Joins the given path components into a single path.

Parameters

a – the root
*p – additional path components

Returns

the joined path

fiftyone.core.storage.isabs(path)¶

Determines whether the given path is absolute.

Remote paths are always considered absolute.

Parameters: path – the filepath
Returns: True/False

fiftyone.core.storage.abspath(path)¶

Converts the given path to an absolute path.

Remote paths are returned unchanged.

Parameters: path – the filepath
Returns: the absolute path

fiftyone.core.storage.normpath(path)¶

Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.

Use this function when you need a version of os.path.normpath that converts \ to / on Unix.

Parameters: path – a path
Returns: the normalized path

fiftyone.core.storage.exists(path)¶

Determines whether the given file or directory exists.

Parameters: path – the file or directory path
Returns: True/False

fiftyone.core.storage.isfile(path)¶

Determines whether the given file exists.

Parameters: path – the filepath
Returns: True/False

fiftyone.core.storage.isdir(dirpath)¶

Determines whether the given directory exists.

Cloud “folders” are deemed to exist only if they are non-empty.

Parameters: dirpath – the directory path
Returns: True/False

fiftyone.core.storage.make_archive(dirpath, archive_path, cleanup=False)¶

Makes an archive containing the given directory.

Supported formats include .zip, .tar, .tar.gz, .tgz, .tar.bz and .tbz.

Parameters

dirpath – the directory to archive
archive_path – the archive path to write
cleanup (False) – whether to delete the directory after archiving it

fiftyone.core.storage.extract_archive(archive_path, outdir=None, cleanup=False)¶

Extracts the contents of an archive.

The following formats are guaranteed to work: .zip, .tar, .tar.gz, .tgz, .tar.bz, .tbz.

If an archive not in the above list is found, extraction will be attempted via the patool package, which supports many formats but may require that additional system packages be installed.

Parameters

archive_path – the archive path
outdir (None) – the directory into which to extract the archive. By default, the directory containing the archive is used
cleanup (False) – whether to delete the archive after extraction

fiftyone.core.storage.ensure_empty_dir(dirpath, cleanup=False)¶

Ensures that the given directory exists and is empty.

Parameters

dirpath – the directory path
cleanup (False) – whether to delete any existing directory contents

Raises

ValueError – if the directory is not empty and cleanup is False

fiftyone.core.storage.ensure_basedir(path)¶

Makes the base directory of the given path, if necessary.

Parameters: path – the filepath

fiftyone.core.storage.ensure_dir(dirpath)¶

Makes the given directory, if necessary.

Parameters: dirpath – the directory path

fiftyone.core.storage.load_json(path_or_str)¶

Loads JSON from the input argument.

Parameters: path_or_str – the filepath or JSON string
Returns: the loaded JSON

fiftyone.core.storage.read_json(path)¶

Reads a JSON file.

Parameters: path – the filepath
Returns: the JSON data

fiftyone.core.storage.write_json(d, path, pretty_print=False)¶

Writes JSON object to file.

Parameters

d – JSON data
path – the filepath
pretty_print (False) – whether to render the JSON in human readable format with newlines and indentations

fiftyone.core.storage.load_ndjson(path_or_str)¶

Loads NDJSON from the input argument.

Parameters: path_or_str – the filepath or NDJSON string
Returns: a list of JSON dicts

fiftyone.core.storage.read_ndjson(path)¶

Reads an NDJSON file.

Parameters: path – the filepath
Returns: a list of JSON dicts

fiftyone.core.storage.write_ndjson(obj, path)¶

Writes the list of JSON dicts in NDJSON format.

Parameters

obj – a list of JSON dicts
path – the filepath

fiftyone.core.storage.read_yaml(path)¶

Reads a YAML file.

Parameters: path – the filepath
Returns: a list of JSON dicts

fiftyone.core.storage.write_yaml(obj, path, **kwargs)¶

Writes the object to a YAML file.

Parameters

obj – a Python object
path – the filepath
**kwargs – optional arguments for yaml.dump(..., **kwargs)

fiftyone.core.storage.list_files(dirpath, abs_paths=False, recursive=False, include_hidden_files=False, return_metadata=False, sort=True)¶

Lists the files in the given directory.

If the directory does not exist, an empty list is returned.

Parameters

dirpath – the path to the directory to list
abs_paths (False) – whether to return the absolute paths to the files
recursive (False) – whether to recursively traverse subdirectories
include_hidden_files (False) – whether to include dot files
return_metadata (False) – whether to return metadata dicts for each file instead of filepaths
sort (True) – whether to sort the list of files

Returns

a list of filepaths or metadata dicts

fiftyone.core.storage.list_subdirs(dirpath, abs_paths=False, recursive=False)¶

Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.

Parameters

dirpath – the path to the directory to list
abs_paths (False) – whether to return absolute paths
recursive (False) – whether to recursively traverse subdirectories

Returns

a list of subdirectories

fiftyone.core.storage.list_buckets(fs, abs_paths=False)¶

Lists the available buckets in the given file system.

This method returns subdirectories of / (or the current drive on Windows).

Parameters

fs – a FileSystem value
abs_paths (False) – whether to return absolute paths

Returns

a list of buckets

fiftyone.core.storage.list_available_file_systems()¶

Lists the file systems that are currently available for use with methods like list_files() and list_buckets().

Returns: a list of FileSystem values

fiftyone.core.storage.get_glob_matches(glob_patt)¶

Returns a list of file paths matching the given glob pattern.

The matches are returned in sorted order.

Parameters: glob_patt – a glob pattern like /path/to/files-*.jpg
Returns: a list of file paths

fiftyone.core.storage.get_glob_root(glob_patt)¶

Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.

Parameters

glob_patt – a glob pattern like /path/to/files-*.jpg

Returns

the root
True/False whether the pattern contains any special characters

Return type

a tuple of

fiftyone.core.storage.copy_file(inpath, outpath)¶

Copies the input file to the output location.

Parameters

inpath – the input path
outpath – the output path

fiftyone.core.storage.copy_files(inpaths, outpaths, skip_failures=False, progress=False)¶

Copies the files to the given locations.

Parameters

inpaths – a list of input paths
outpaths – a list of output paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation

fiftyone.core.storage.copy_dir(indir, outdir, overwrite=True, skip_failures=False, progress=False)¶

Copies the input directory to the output directory.

Parameters

indir – the input directory
outdir – the output directory
overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation

fiftyone.core.storage.move_file(inpath, outpath)¶

Moves the given file to a new location.

Parameters

inpath – the input path
outpath – the output path

fiftyone.core.storage.move_files(inpaths, outpaths, skip_failures=False, progress=False)¶

Moves the files to the given locations.

Parameters

inpaths – a list of input paths
outpaths – a list of output paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation

fiftyone.core.storage.move_dir(indir, outdir, overwrite=True, skip_failures=False, progress=False)¶

Moves the contents of the given directory into the given output directory.

Parameters

indir – the input directory
outdir – the output directory
overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation

fiftyone.core.storage.delete_file(path)¶

Deletes the file at the given path.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters: path – the filepath

fiftyone.core.storage.delete_files(paths, skip_failures=False, progress=False)¶

Deletes the files from the given locations.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters

paths – a list of paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation

fiftyone.core.storage.delete_dir(dirpath)¶

Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.

Parameters: dirpath – the directory path

fiftyone.core.storage.run(fcn, tasks, num_workers=None, progress=False)¶

Applies the given function to each element of the given tasks.

Parameters

fcn – a function that accepts a single argument
tasks – an iterable of function aguments
num_workers (None) – a suggested number of threads to use
progress (False) – whether to render a progress bar tracking the status of the operation

Returns

the list of function outputs