fiftyone.core.storage

File storage utilities.

Copyright 2017-2024, Voxel51, Inc.

voxel51.com

Classes:

FileSystem(value)	Enumeration of the available file systems.
TempDir([basedir])	Context manager that creates and destroys a temporary directory.

Functions:

get_file_system(path)	Returns the file system enum for the given path.
split_prefix(path)	Splits the file system prefix from the given path.
get_bucket_name(path)	Gets the bucket name from the given path.
is_local(path)	Determines whether the given path is local.
ensure_local(path)	Ensures that the given path is local.
normalize_path(path)	Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.
make_temp_dir([basedir])	Makes a temporary directory.
open_file(path[, mode])	Opens the given file for reading or writing.
open_files(paths[, mode, skip_failures, …])	Opens the given files for reading or writing.
read_file(path[, binary])	Reads the file.
read_files(paths[, binary, skip_failures, …])	Reads the specified files into memory.
write_file(str_or_bytes, path)	Writes the given string/bytes to a file.
sep(path)	Returns the path separator for the given path.
join(a, *p)	Joins the given path components into a single path.
realpath(path)	Converts the given path to absolute, resolving symlinks and relative path indicators such as . and ...
isabs(path)	Determines whether the given path is absolute.
abspath(path)	Converts the given path to an absolute path, resolving relative path indicators such as . and ...
normpath(path)	Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.
exists(path)	Determines whether the given file or directory exists.
isfile(path)	Determines whether the given file exists.
isdir(dirpath)	Determines whether the given directory exists.
make_archive(dirpath, archive_path[, cleanup])	Makes an archive containing the given directory.
extract_archive(archive_path[, outdir, cleanup])	Extracts the contents of an archive.
ensure_empty_dir(dirpath[, cleanup])	Ensures that the given directory exists and is empty.
ensure_basedir(path)	Makes the base directory of the given path, if necessary.
ensure_dir(dirpath)	Makes the given directory, if necessary.
load_json(path_or_str)	Loads JSON from the input argument.
read_json(path)	Reads a JSON file.
write_json(d, path[, pretty_print])	Writes JSON object to file.
load_ndjson(path_or_str)	Loads NDJSON from the input argument.
read_ndjson(path)	Reads an NDJSON file.
write_ndjson(obj, path)	Writes the list of JSON dicts in NDJSON format.
read_yaml(path)	Reads a YAML file.
write_yaml(obj, path, **kwargs)	Writes the object to a YAML file.
list_files(dirpath[, abs_paths, recursive, …])	Lists the files in the given directory.
list_subdirs(dirpath[, abs_paths, recursive])	Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.
list_buckets(fs[, abs_paths])	Lists the available buckets in the given file system.
list_available_file_systems()	Lists the file systems that are currently available for use with methods like list_files() and list_buckets().
get_glob_matches(glob_patt)	Returns a list of file paths matching the given glob pattern.
get_glob_root(glob_patt)	Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.
copy_file(inpath, outpath)	Copies the input file to the output location.
copy_files(inpaths, outpaths[, …])	Copies the files to the given locations.
copy_dir(indir, outdir[, overwrite, …])	Copies the input directory to the output directory.
move_file(inpath, outpath)	Moves the given file to a new location.
move_files(inpaths, outpaths[, …])	Moves the files to the given locations.
move_dir(indir, outdir[, overwrite, …])	Moves the contents of the given directory into the given output directory.
delete_file(path)	Deletes the file at the given path.
delete_files(paths[, skip_failures, progress])	Deletes the files from the given locations.
delete_dir(dirpath)	Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.
run(fcn, tasks[, return_results, …])	Applies the given function to each element of the given tasks.

class fiftyone.core.storage.FileSystem(value)

Bases: enum.Enum

Enumeration of the available file systems.

Attributes:

LOCAL

LOCAL = 'local'

fiftyone.core.storage.get_file_system(path)

Returns the file system enum for the given path.

Parameters

path – a path

Returns

a FileSystem value

fiftyone.core.storage.split_prefix(path)

Splits the file system prefix from the given path.

The prefix for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.split_prefix("/path/to/file")       # ('', '/path/to/file')
fos.split_prefix("a/file")              # ('', 'a/file')

Parameters

path – a path

Returns

a (prefix, path) tuple

fiftyone.core.storage.get_bucket_name(path)

Gets the bucket name from the given path.

The bucket name for local paths is "".

Example usages:

import fiftyone.core.storage as fos

fos.get_bucket_name("/path/to/file")       # ''
fos.get_bucket_name("a/file")              # ''

Parameters

path – a path

Returns

the bucket name string

fiftyone.core.storage.is_local(path)

Determines whether the given path is local.

Parameters

path – a path

Returns

True/False

fiftyone.core.storage.ensure_local(path)

Ensures that the given path is local.

Parameters

path – a path

fiftyone.core.storage.normalize_path(path)

Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.

Parameters

path – a path

Returns

the normalized path

fiftyone.core.storage.make_temp_dir(basedir=None)

Makes a temporary directory.

Parameters

basedir (None) – an optional directory in which to create the new directory. The default is fiftyone.config.default_dataset_dir

Returns

the temporary directory path

class fiftyone.core.storage.TempDir(basedir=None)

Bases: object

Context manager that creates and destroys a temporary directory.

Parameters

basedir (None) – an optional directory in which to create the new directory. The default is fiftyone.config.default_dataset_dir

fiftyone.core.storage.open_file(path, mode='r')

Opens the given file for reading or writing.

Example usage:

import fiftyone.core.storage as fos

with fos.open_file("/tmp/file.txt", "w") as f:
    f.write("Hello, world!")

with fos.open_file("/tmp/file.txt", "r") as f:
    print(f.read())

Parameters

• path – the path

• mode ("r") – the mode. Supported values are ("r", "rb", "w", "wb")

Returns

an open file-like object

fiftyone.core.storage.open_files(paths, mode='r', skip_failures=False, progress=None)

Opens the given files for reading or writing.

Parameters

• paths – a list of paths

• mode ("r") – the mode. Supported values are ("r", "rb", "w", "wb")

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a list of open file-like objects

fiftyone.core.storage.read_file(path, binary=False)

Reads the file.

Parameters

• path – the filepath

• binary (False) – whether to read the file in binary mode

Returns

the file contents

fiftyone.core.storage.read_files(paths, binary=False, skip_failures=False, progress=None)

Reads the specified files into memory.

Parameters

• paths – a list of filepaths

• binary (False) – whether to read the files in binary mode

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

a list of file contents

fiftyone.core.storage.write_file(str_or_bytes, path)

Writes the given string/bytes to a file.

If a string is provided, it is encoded via .encode().

Parameters

• str_or_bytes – the string or bytes

• path – the filepath

fiftyone.core.storage.sep(path)

Returns the path separator for the given path.

Parameters

path – the filepath

Returns

the path separator

fiftyone.core.storage.join(a, *p)

Joins the given path components into a single path.

Parameters

• a – the root

• *p – additional path components

Returns

the joined path

fiftyone.core.storage.realpath(path)

Converts the given path to absolute, resolving symlinks and relative path indicators such as . and ...

Parameters

path – the filepath

Returns

the resolved path

fiftyone.core.storage.isabs(path)

Determines whether the given path is absolute.

Parameters

path – the filepath

Returns

True/False

fiftyone.core.storage.abspath(path)

Converts the given path to an absolute path, resolving relative path indicators such as . and ...

Parameters

path – the filepath

Returns

the absolute path

fiftyone.core.storage.normpath(path)

Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.

Use this function when you need a version of os.path.normpath that converts \ to / on Unix.

Parameters

path – a path

Returns

the normalized path

fiftyone.core.storage.exists(path)

Determines whether the given file or directory exists.

Parameters

path – the file or directory path

Returns

True/False

fiftyone.core.storage.isfile(path)

Determines whether the given file exists.

Parameters

path – the filepath

Returns

True/False

fiftyone.core.storage.isdir(dirpath)

Determines whether the given directory exists.

Cloud “folders” are deemed to exist only if they are non-empty.

Parameters

dirpath – the directory path

Returns

True/False

fiftyone.core.storage.make_archive(dirpath, archive_path, cleanup=False)

Makes an archive containing the given directory.

Supported formats include .zip, .tar, .tar.gz, .tgz, .tar.bz and .tbz.

Parameters

• dirpath – the directory to archive

• archive_path – the archive path to write

• cleanup (False) – whether to delete the directory after archiving it

fiftyone.core.storage.extract_archive(archive_path, outdir=None, cleanup=False)

Extracts the contents of an archive.

The following formats are guaranteed to work: .zip, .tar, .tar.gz, .tgz, .tar.bz, .tbz.

If an archive not in the above list is found, extraction will be attempted via the patool package, which supports many formats but may require that additional system packages be installed.

Parameters

• archive_path – the archive path

• outdir (None) – the directory into which to extract the archive. By default, the directory containing the archive is used

• cleanup (False) – whether to delete the archive after extraction

fiftyone.core.storage.ensure_empty_dir(dirpath, cleanup=False)

Ensures that the given directory exists and is empty.

Parameters

• dirpath – the directory path

• cleanup (False) – whether to delete any existing directory contents

Raises

ValueError – if the directory is not empty and cleanup is False

fiftyone.core.storage.ensure_basedir(path)

Makes the base directory of the given path, if necessary.

Parameters

path – the filepath

fiftyone.core.storage.ensure_dir(dirpath)

Makes the given directory, if necessary.

Parameters

dirpath – the directory path

fiftyone.core.storage.load_json(path_or_str)

Loads JSON from the input argument.

Parameters

path_or_str – the filepath or JSON string

Returns

the loaded JSON

fiftyone.core.storage.read_json(path)

Reads a JSON file.

Parameters

path – the filepath

Returns

the JSON data

fiftyone.core.storage.write_json(d, path, pretty_print=False)

Writes JSON object to file.

Parameters

• d – JSON data

• path – the filepath

• pretty_print (False) – whether to render the JSON in human readable format with newlines and indentations

fiftyone.core.storage.load_ndjson(path_or_str)

Loads NDJSON from the input argument.

Parameters

path_or_str – the filepath or NDJSON string

Returns

a list of JSON dicts

fiftyone.core.storage.read_ndjson(path)

Reads an NDJSON file.

Parameters

path – the filepath

Returns

a list of JSON dicts

fiftyone.core.storage.write_ndjson(obj, path)

Writes the list of JSON dicts in NDJSON format.

Parameters

• obj – a list of JSON dicts

• path – the filepath

fiftyone.core.storage.read_yaml(path)

Reads a YAML file.

Parameters

path – the filepath

Returns

a list of JSON dicts

fiftyone.core.storage.write_yaml(obj, path, **kwargs)

Writes the object to a YAML file.

Parameters

• obj – a Python object

• path – the filepath

• **kwargs – optional arguments for yaml.dump(..., **kwargs)

fiftyone.core.storage.list_files(dirpath, abs_paths=False, recursive=False, include_hidden_files=False, return_metadata=False, sort=True)

Lists the files in the given directory.

If the directory does not exist, an empty list is returned.

Parameters

• dirpath – the path to the directory to list

• abs_paths (False) – whether to return the absolute paths to the files

• recursive (False) – whether to recursively traverse subdirectories

• include_hidden_files (False) – whether to include dot files

• return_metadata (False) – whether to return metadata dicts for each file instead of filepaths

• sort (True) – whether to sort the list of files

Returns

a list of filepaths or metadata dicts

fiftyone.core.storage.list_subdirs(dirpath, abs_paths=False, recursive=False)

Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.

Parameters

• dirpath – the path to the directory to list

• abs_paths (False) – whether to return absolute paths

• recursive (False) – whether to recursively traverse subdirectories

Returns

a list of subdirectories

fiftyone.core.storage.list_buckets(fs, abs_paths=False)

Lists the available buckets in the given file system.

This method returns subdirectories of / (or the current drive on Windows).

Parameters

• fs – a FileSystem value

• abs_paths (False) – whether to return absolute paths

Returns

a list of buckets

fiftyone.core.storage.list_available_file_systems()

Lists the file systems that are currently available for use with methods like list_files() and list_buckets().

Returns

a list of FileSystem values

fiftyone.core.storage.get_glob_matches(glob_patt)

Returns a list of file paths matching the given glob pattern.

The matches are returned in sorted order.

Parameters

glob_patt – a glob pattern like /path/to/files-*.jpg

Returns

a list of file paths

fiftyone.core.storage.get_glob_root(glob_patt)

Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.

Parameters

glob_patt – a glob pattern like /path/to/files-*.jpg

Returns

• the root

• True/False whether the pattern contains any special characters

Return type

a tuple of

fiftyone.core.storage.copy_file(inpath, outpath)

Copies the input file to the output location.

Parameters

• inpath – the input path

• outpath – the output path

fiftyone.core.storage.copy_files(inpaths, outpaths, skip_failures=False, progress=None)

Copies the files to the given locations.

Parameters

• inpaths – a list of input paths

• outpaths – a list of output paths

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.storage.copy_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None)

Copies the input directory to the output directory.

Parameters

• indir – the input directory

• outdir – the output directory

• overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.storage.move_file(inpath, outpath)

Moves the given file to a new location.

Parameters

• inpath – the input path

• outpath – the output path

fiftyone.core.storage.move_files(inpaths, outpaths, skip_failures=False, progress=None)

Moves the files to the given locations.

Parameters

• inpaths – a list of input paths

• outpaths – a list of output paths

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.storage.move_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None)

Moves the contents of the given directory into the given output directory.

Parameters

• indir – the input directory

• outdir – the output directory

• overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.storage.delete_file(path)

Deletes the file at the given path.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters

path – the filepath

fiftyone.core.storage.delete_files(paths, skip_failures=False, progress=None)

Deletes the files from the given locations.

Any empty directories are also recursively deleted from the resulting directory tree.

Parameters

• paths – a list of paths

• skip_failures (False) – whether to gracefully continue without raising an error if an operation fails

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

fiftyone.core.storage.delete_dir(dirpath)

Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.

Parameters

dirpath – the directory path

fiftyone.core.storage.run(fcn, tasks, return_results=True, num_workers=None, progress=None)

Applies the given function to each element of the given tasks.

Parameters

• fcn – a function that accepts a single argument

• tasks – an iterable of function arguments

• return_results (True) – whether to return the function results

• num_workers (None) – a suggested number of threads to use

• progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns

the list of function outputs, or None if return_results == False