fiftyone.core.storage#
File storage utilities.
Classes:
|
Enumeration of the available file systems. |
|
Context manager that creates and destroys a temporary directory. |
Functions:
|
Returns the file system enum for the given path. |
|
Splits the file system prefix from the given path. |
|
Gets the bucket name from the given path. |
|
Determines whether the given path is local. |
|
Ensures that the given path is local. |
|
Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary. |
|
Makes a temporary directory. |
|
Opens the given file for reading or writing. |
|
Opens the given files for reading or writing. |
|
Reads the file. |
|
Reads the specified files into memory. |
|
Writes the given string/bytes to a file. |
|
Returns the path separator for the given path. |
|
Joins the given path components into a single path. |
|
Converts the given path to absolute, resolving symlinks and relative path indicators such as |
|
Determines whether the given path is absolute. |
|
Converts the given path to an absolute path, resolving relative path indicators such as |
|
Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes. |
|
Determines whether the given file or directory exists. |
|
Determines whether the given file exists. |
|
Determines whether the given directory exists. |
|
Makes an archive containing the given directory. |
|
Extracts the contents of an archive. |
|
Ensures that the given directory exists and is empty. |
|
Makes the base directory of the given path, if necessary. |
|
Makes the given directory, if necessary. |
|
Loads JSON from the input argument. |
|
Reads a JSON file. |
|
Writes JSON object to file. |
|
Loads NDJSON from the input argument. |
|
Reads an NDJSON file. |
|
Writes the list of JSON dicts in NDJSON format. |
|
Reads a YAML file. |
|
Writes the object to a YAML file. |
|
Lists the files in the given directory. |
|
Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories. |
|
Lists the available buckets in the given file system. |
Lists the file systems that are currently available for use with methods like |
|
|
Returns a list of file paths matching the given glob pattern. |
|
Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters. |
|
Copies the input file to the output location. |
|
Copies the files to the given locations. |
|
Copies the input directory to the output directory. |
|
Moves the given file to a new location. |
|
Moves the files to the given locations. |
|
Moves the contents of the given directory into the given output directory. |
|
Deletes the file at the given path. |
|
Deletes the files from the given locations. |
|
Deletes the given directory and recursively deletes any empty directories from the resulting directory tree. |
|
Applies the given function to each element of the given tasks. |
- class fiftyone.core.storage.FileSystem(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)#
Bases:
Enum
Enumeration of the available file systems.
Attributes:
- LOCAL = 'local'#
- fiftyone.core.storage.get_file_system(path)#
Returns the file system enum for the given path.
- Parameters:
path β a path
- Returns:
a
FileSystem
value
- fiftyone.core.storage.split_prefix(path)#
Splits the file system prefix from the given path.
The prefix for local paths is
""
.Example usages:
import fiftyone.core.storage as fos fos.split_prefix("/path/to/file") # ('', '/path/to/file') fos.split_prefix("a/file") # ('', 'a/file')
- Parameters:
path β a path
- Returns:
a
(prefix, path)
tuple
- fiftyone.core.storage.get_bucket_name(path)#
Gets the bucket name from the given path.
The bucket name for local paths is
""
.Example usages:
import fiftyone.core.storage as fos fos.get_bucket_name("/path/to/file") # '' fos.get_bucket_name("a/file") # ''
- Parameters:
path β a path
- Returns:
the bucket name string
- fiftyone.core.storage.is_local(path)#
Determines whether the given path is local.
- Parameters:
path β a path
- Returns:
True/False
- fiftyone.core.storage.ensure_local(path)#
Ensures that the given path is local.
- Parameters:
path β a path
- fiftyone.core.storage.normalize_path(path)#
Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.
- Parameters:
path β a path
- Returns:
the normalized path
- fiftyone.core.storage.make_temp_dir(basedir=None)#
Makes a temporary directory.
- Parameters:
basedir (None) β an optional directory in which to create the new directory. The default is
fiftyone.config.default_dataset_dir
- Returns:
the temporary directory path
- class fiftyone.core.storage.TempDir(basedir=None)#
Bases:
object
Context manager that creates and destroys a temporary directory.
- Parameters:
basedir (None) β an optional directory in which to create the new directory. The default is
fiftyone.config.default_dataset_dir
- fiftyone.core.storage.open_file(path, mode='r')#
Opens the given file for reading or writing.
Example usage:
import fiftyone.core.storage as fos with fos.open_file("/tmp/file.txt", "w") as f: f.write("Hello, world!") with fos.open_file("/tmp/file.txt", "r") as f: print(f.read())
- Parameters:
path β the path
mode ("r") β the mode. Supported values are
("r", "rb", "w", "wb")
- Returns:
an open file-like object
- fiftyone.core.storage.open_files(paths, mode='r', skip_failures=False, progress=None)#
Opens the given files for reading or writing.
- Parameters:
paths β a list of paths
mode ("r") β the mode. Supported values are
("r", "rb", "w", "wb")
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- Returns:
a list of open file-like objects
- fiftyone.core.storage.read_file(path, binary=False)#
Reads the file.
- Parameters:
path β the filepath
binary (False) β whether to read the file in binary mode
- Returns:
the file contents
- fiftyone.core.storage.read_files(paths, binary=False, skip_failures=False, progress=None)#
Reads the specified files into memory.
- Parameters:
paths β a list of filepaths
binary (False) β whether to read the files in binary mode
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- Returns:
a list of file contents
- fiftyone.core.storage.write_file(str_or_bytes, path)#
Writes the given string/bytes to a file.
If a string is provided, it is encoded via
.encode()
.- Parameters:
str_or_bytes β the string or bytes
path β the filepath
- fiftyone.core.storage.sep(path)#
Returns the path separator for the given path.
- Parameters:
path β the filepath
- Returns:
the path separator
- fiftyone.core.storage.join(a, *p)#
Joins the given path components into a single path.
- Parameters:
a β the root
*p β additional path components
- Returns:
the joined path
- fiftyone.core.storage.realpath(path)#
Converts the given path to absolute, resolving symlinks and relative path indicators such as
.
and..
.- Parameters:
path β the filepath
- Returns:
the resolved path
- fiftyone.core.storage.isabs(path)#
Determines whether the given path is absolute.
- Parameters:
path β the filepath
- Returns:
True/False
- fiftyone.core.storage.abspath(path)#
Converts the given path to an absolute path, resolving relative path indicators such as
.
and..
.- Parameters:
path β the filepath
- Returns:
the absolute path
- fiftyone.core.storage.normpath(path)#
Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.
Use this function when you need a version of
os.path.normpath
that converts\
to/
on Unix.- Parameters:
path β a path
- Returns:
the normalized path
- fiftyone.core.storage.exists(path)#
Determines whether the given file or directory exists.
- Parameters:
path β the file or directory path
- Returns:
True/False
- fiftyone.core.storage.isfile(path)#
Determines whether the given file exists.
- Parameters:
path β the filepath
- Returns:
True/False
- fiftyone.core.storage.isdir(dirpath)#
Determines whether the given directory exists.
Cloud βfoldersβ are deemed to exist only if they are non-empty.
- Parameters:
dirpath β the directory path
- Returns:
True/False
- fiftyone.core.storage.make_archive(dirpath, archive_path, cleanup=False)#
Makes an archive containing the given directory.
Supported formats include
.zip
,.tar
,.tar.gz
,.tgz
,.tar.bz
and.tbz
.- Parameters:
dirpath β the directory to archive
archive_path β the archive path to write
cleanup (False) β whether to delete the directory after archiving it
- fiftyone.core.storage.extract_archive(archive_path, outdir=None, cleanup=False)#
Extracts the contents of an archive.
The following formats are guaranteed to work:
.zip
,.tar
,.tar.gz
,.tgz
,.tar.bz
,.tbz
.If an archive not in the above list is found, extraction will be attempted via the
patool
package, which supports many formats but may require that additional system packages be installed.- Parameters:
archive_path β the archive path
outdir (None) β the directory into which to extract the archive. By default, the directory containing the archive is used
cleanup (False) β whether to delete the archive after extraction
- fiftyone.core.storage.ensure_empty_dir(dirpath, cleanup=False)#
Ensures that the given directory exists and is empty.
- Parameters:
dirpath β the directory path
cleanup (False) β whether to delete any existing directory contents
- Raises:
ValueError β if the directory is not empty and
cleanup
is False
- fiftyone.core.storage.ensure_basedir(path)#
Makes the base directory of the given path, if necessary.
- Parameters:
path β the filepath
- fiftyone.core.storage.ensure_dir(dirpath)#
Makes the given directory, if necessary.
- Parameters:
dirpath β the directory path
- fiftyone.core.storage.load_json(path_or_str)#
Loads JSON from the input argument.
- Parameters:
path_or_str β the filepath or JSON string
- Returns:
the loaded JSON
- fiftyone.core.storage.read_json(path)#
Reads a JSON file.
- Parameters:
path β the filepath
- Returns:
the JSON data
- fiftyone.core.storage.write_json(d, path, pretty_print=False)#
Writes JSON object to file.
- Parameters:
d β JSON data
path β the filepath
pretty_print (False) β whether to render the JSON in human readable format with newlines and indentations
- fiftyone.core.storage.load_ndjson(path_or_str)#
Loads NDJSON from the input argument.
- Parameters:
path_or_str β the filepath or NDJSON string
- Returns:
a list of JSON dicts
- fiftyone.core.storage.read_ndjson(path)#
Reads an NDJSON file.
- Parameters:
path β the filepath
- Returns:
a list of JSON dicts
- fiftyone.core.storage.write_ndjson(obj, path)#
Writes the list of JSON dicts in NDJSON format.
- Parameters:
obj β a list of JSON dicts
path β the filepath
- fiftyone.core.storage.read_yaml(path)#
Reads a YAML file.
- Parameters:
path β the filepath
- Returns:
a list of JSON dicts
- fiftyone.core.storage.write_yaml(obj, path, **kwargs)#
Writes the object to a YAML file.
- Parameters:
obj β a Python object
path β the filepath
**kwargs β optional arguments for
yaml.dump(..., **kwargs)
- fiftyone.core.storage.list_files(dirpath, abs_paths=False, recursive=False, include_hidden_files=False, return_metadata=False, sort=True)#
Lists the files in the given directory.
If the directory does not exist, an empty list is returned.
- Parameters:
dirpath β the path to the directory to list
abs_paths (False) β whether to return the absolute paths to the files
recursive (False) β whether to recursively traverse subdirectories
include_hidden_files (False) β whether to include dot files
return_metadata (False) β whether to return metadata dicts for each file instead of filepaths
sort (True) β whether to sort the list of files
- Returns:
a list of filepaths or metadata dicts
- fiftyone.core.storage.list_subdirs(dirpath, abs_paths=False, recursive=False)#
Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.
- Parameters:
dirpath β the path to the directory to list
abs_paths (False) β whether to return absolute paths
recursive (False) β whether to recursively traverse subdirectories
- Returns:
a list of subdirectories
- fiftyone.core.storage.list_buckets(fs, abs_paths=False)#
Lists the available buckets in the given file system.
This method returns subdirectories of
/
(or the current drive on Windows).- Parameters:
fs β a
FileSystem
valueabs_paths (False) β whether to return absolute paths
- Returns:
a list of buckets
- fiftyone.core.storage.list_available_file_systems()#
Lists the file systems that are currently available for use with methods like
list_files()
andlist_buckets()
.- Returns:
a list of
FileSystem
values
- fiftyone.core.storage.get_glob_matches(glob_patt)#
Returns a list of file paths matching the given glob pattern.
The matches are returned in sorted order.
- Parameters:
glob_patt β a glob pattern like
/path/to/files-*.jpg
- Returns:
a list of file paths
- fiftyone.core.storage.get_glob_root(glob_patt)#
Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.
- Parameters:
glob_patt β a glob pattern like
/path/to/files-*.jpg
- Returns:
the root
True/False whether the pattern contains any special characters
- Return type:
a tuple of
- fiftyone.core.storage.copy_file(inpath, outpath)#
Copies the input file to the output location.
- Parameters:
inpath β the input path
outpath β the output path
- fiftyone.core.storage.copy_files(inpaths, outpaths, skip_failures=False, progress=None)#
Copies the files to the given locations.
- Parameters:
inpaths β a list of input paths
outpaths β a list of output paths
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- fiftyone.core.storage.copy_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None)#
Copies the input directory to the output directory.
- Parameters:
indir β the input directory
outdir β the output directory
overwrite (True) β whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- fiftyone.core.storage.move_file(inpath, outpath)#
Moves the given file to a new location.
- Parameters:
inpath β the input path
outpath β the output path
- fiftyone.core.storage.move_files(inpaths, outpaths, skip_failures=False, progress=None)#
Moves the files to the given locations.
- Parameters:
inpaths β a list of input paths
outpaths β a list of output paths
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- fiftyone.core.storage.move_dir(indir, outdir, overwrite=True, skip_failures=False, progress=None)#
Moves the contents of the given directory into the given output directory.
- Parameters:
indir β the input directory
outdir β the output directory
overwrite (True) β whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- fiftyone.core.storage.delete_file(path)#
Deletes the file at the given path.
Any empty directories are also recursively deleted from the resulting directory tree.
- Parameters:
path β the filepath
- fiftyone.core.storage.delete_files(paths, skip_failures=False, progress=None)#
Deletes the files from the given locations.
Any empty directories are also recursively deleted from the resulting directory tree.
- Parameters:
paths β a list of paths
skip_failures (False) β whether to gracefully continue without raising an error if an operation fails
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- fiftyone.core.storage.delete_dir(dirpath)#
Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.
- Parameters:
dirpath β the directory path
- fiftyone.core.storage.run(fcn, tasks, return_results=True, num_workers=None, progress=None)#
Applies the given function to each element of the given tasks.
- Parameters:
fcn β a function that accepts a single argument
tasks β an iterable of function arguments
return_results (True) β whether to return the function results
num_workers (None) β a suggested number of threads to use
progress (None) β whether to render a progress bar (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke instead
- Returns:
the list of function outputs, or None if
return_results == False