fiftyone.core.storage¶
File storage utilities.
Classes:
Enumeration of the available file systems. |
|
|
Context manager that creates and destroys a temporary directory. |
Functions:
|
Converts the given path to an absolute path. |
|
Copies the input directory to the output directory. |
|
Copies the input file to the output location. |
|
Copies the files to the given locations. |
|
Deletes the given directory and recursively deletes any empty directories from the resulting directory tree. |
|
Deletes the file at the given path. |
|
Deletes the files from the given locations. |
|
Makes the base directory of the given path, if necessary. |
|
Makes the given directory, if necessary. |
|
Ensures that the given directory exists and is empty. |
|
Ensures that the given path is local. |
|
Determines whether the given file or directory exists. |
|
Extracts the contents of an archive. |
|
Gets the bucket name from the given path. |
|
Returns the file system enum for the given path. |
|
Returns a list of file paths matching the given glob pattern. |
|
Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters. |
|
Determines whether the given path is local. |
|
Determines whether the given path is absolute. |
|
Determines whether the given directory exists. |
|
Determines whether the given file exists. |
|
Joins the given path components into a single path. |
Lists the file systems that are currently available for use with methods like |
|
|
Lists the available buckets in the given file system. |
|
Lists the files in the given directory. |
|
Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories. |
|
Loads JSON from the input argument. |
|
Loads NDJSON from the input argument. |
|
Makes an archive containing the given directory. |
|
Makes a temporary directory. |
|
Moves the contents of the given directory into the given output directory. |
|
Moves the given file to a new location. |
|
Moves the files to the given locations. |
|
Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary. |
|
Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes. |
|
Opens the given file for reading or writing. |
|
Reads the file. |
|
Reads a JSON file. |
|
Reads an NDJSON file. |
|
Reads a YAML file. |
|
Applies the given function to each element of the given tasks. |
|
Returns the path separator for the given path. |
|
Splits the file system prefix from the given path. |
|
Writes the given string/bytes to a file. |
|
Writes JSON object to file. |
|
Writes the list of JSON dicts in NDJSON format. |
|
Writes the object to a YAML file. |
-
class
fiftyone.core.storage.
FileSystem
¶ Bases:
enum.Enum
Enumeration of the available file systems.
Attributes:
-
LOCAL
= 'local'¶
-
-
fiftyone.core.storage.
get_file_system
(path)¶ Returns the file system enum for the given path.
- Parameters
path – a path
- Returns
a
FileSystem
value
-
fiftyone.core.storage.
split_prefix
(path)¶ Splits the file system prefix from the given path.
The prefix for local paths is
""
.Example usages:
import fiftyone.core.storage as fos fos.split_prefix("/path/to/file") # ('', '/path/to/file') fos.split_prefix("a/file") # ('', 'a/file')
- Parameters
path – a path
- Returns
a
(prefix, path)
tuple
-
fiftyone.core.storage.
get_bucket_name
(path)¶ Gets the bucket name from the given path.
The bucket name for local paths is
""
.Example usages:
import fiftyone.core.storage as fos fos.get_bucket_name("/path/to/file") # '' fos.get_bucket_name("a/file") # ''
- Parameters
path – a path
- Returns
the bucket name string
-
fiftyone.core.storage.
is_local
(path)¶ Determines whether the given path is local.
- Parameters
path – a path
- Returns
True/False
-
fiftyone.core.storage.
ensure_local
(path)¶ Ensures that the given path is local.
- Parameters
path – a path
-
fiftyone.core.storage.
normalize_path
(path)¶ Normalizes the given path by converting it to an absolute path and expanding the user directory, if necessary.
- Parameters
path – a path
- Returns
the normalized path
-
fiftyone.core.storage.
make_temp_dir
(basedir=None)¶ Makes a temporary directory.
- Parameters
basedir (None) – an optional directory in which to create the new directory. The default is
fiftyone.config.default_dataset_dir
- Returns
the temporary directory path
-
class
fiftyone.core.storage.
TempDir
(basedir=None)¶ Bases:
object
Context manager that creates and destroys a temporary directory.
- Parameters
basedir (None) – an optional directory in which to create the new directory. The default is
fiftyone.config.default_dataset_dir
-
fiftyone.core.storage.
open_file
(path, mode='r')¶ Opens the given file for reading or writing.
This function must be used as a context manager.
Example usage:
import fiftyone.core.storage as fos with fos.open_file("/tmp/file.txt", "w") as f: f.write("Hello, world!") with fos.open_file("/tmp/file.txt", "r") as f: print(f.read())
- Parameters
path – the path
mode ("r") – the mode. Supported values are
("r", "rb", "w", "wb")
-
fiftyone.core.storage.
read_file
(path, binary=False)¶ Reads the file.
- Parameters
path – the filepath
binary (False) – whether to read the file in binary mode
- Returns
the file contents
-
fiftyone.core.storage.
write_file
(str_or_bytes, path)¶ Writes the given string/bytes to a file.
If a string is provided, it is encoded via
.encode()
.- Parameters
str_or_bytes – the string or bytes
path – the filepath
-
fiftyone.core.storage.
sep
(path)¶ Returns the path separator for the given path.
For local paths,
os.path.sep
is returned.For remote paths,
"/"
is returned.- Parameters
path – the filepath
- Returns
the path separator
-
fiftyone.core.storage.
join
(a, *p)¶ Joins the given path components into a single path.
- Parameters
a – the root
*p – additional path components
- Returns
the joined path
-
fiftyone.core.storage.
isabs
(path)¶ Determines whether the given path is absolute.
Remote paths are always considered absolute.
- Parameters
path – the filepath
- Returns
True/False
-
fiftyone.core.storage.
abspath
(path)¶ Converts the given path to an absolute path.
Remote paths are returned unchanged.
- Parameters
path – the filepath
- Returns
the absolute path
-
fiftyone.core.storage.
normpath
(path)¶ Normalizes the given path by converting all slashes to forward slashes on Unix and backslashes on Windows and removing duplicate slashes.
Use this function when you need a version of
os.path.normpath
that converts\
to/
on Unix.- Parameters
path – a path
- Returns
the normalized path
-
fiftyone.core.storage.
exists
(path)¶ Determines whether the given file or directory exists.
- Parameters
path – the file or directory path
- Returns
True/False
-
fiftyone.core.storage.
isfile
(path)¶ Determines whether the given file exists.
- Parameters
path – the filepath
- Returns
True/False
-
fiftyone.core.storage.
isdir
(dirpath)¶ Determines whether the given directory exists.
Cloud “folders” are deemed to exist only if they are non-empty.
- Parameters
dirpath – the directory path
- Returns
True/False
-
fiftyone.core.storage.
make_archive
(dirpath, archive_path, cleanup=False)¶ Makes an archive containing the given directory.
Supported formats include
.zip
,.tar
,.tar.gz
,.tgz
,.tar.bz
and.tbz
.- Parameters
dirpath – the directory to archive
archive_path – the archive path to write
cleanup (False) – whether to delete the directory after archiving it
-
fiftyone.core.storage.
extract_archive
(archive_path, outdir=None, cleanup=False)¶ Extracts the contents of an archive.
The following formats are guaranteed to work:
.zip
,.tar
,.tar.gz
,.tgz
,.tar.bz
,.tbz
.If an archive not in the above list is found, extraction will be attempted via the
patool
package, which supports many formats but may require that additional system packages be installed.- Parameters
archive_path – the archive path
outdir (None) – the directory into which to extract the archive. By default, the directory containing the archive is used
cleanup (False) – whether to delete the archive after extraction
-
fiftyone.core.storage.
ensure_empty_dir
(dirpath, cleanup=False)¶ Ensures that the given directory exists and is empty.
- Parameters
dirpath – the directory path
cleanup (False) – whether to delete any existing directory contents
- Raises
ValueError – if the directory is not empty and
cleanup
is False
-
fiftyone.core.storage.
ensure_basedir
(path)¶ Makes the base directory of the given path, if necessary.
- Parameters
path – the filepath
-
fiftyone.core.storage.
ensure_dir
(dirpath)¶ Makes the given directory, if necessary.
- Parameters
dirpath – the directory path
-
fiftyone.core.storage.
load_json
(path_or_str)¶ Loads JSON from the input argument.
- Parameters
path_or_str – the filepath or JSON string
- Returns
the loaded JSON
-
fiftyone.core.storage.
read_json
(path)¶ Reads a JSON file.
- Parameters
path – the filepath
- Returns
the JSON data
-
fiftyone.core.storage.
write_json
(d, path, pretty_print=False)¶ Writes JSON object to file.
- Parameters
d – JSON data
path – the filepath
pretty_print (False) – whether to render the JSON in human readable format with newlines and indentations
-
fiftyone.core.storage.
load_ndjson
(path_or_str)¶ Loads NDJSON from the input argument.
- Parameters
path_or_str – the filepath or NDJSON string
- Returns
a list of JSON dicts
-
fiftyone.core.storage.
read_ndjson
(path)¶ Reads an NDJSON file.
- Parameters
path – the filepath
- Returns
a list of JSON dicts
-
fiftyone.core.storage.
write_ndjson
(obj, path)¶ Writes the list of JSON dicts in NDJSON format.
- Parameters
obj – a list of JSON dicts
path – the filepath
-
fiftyone.core.storage.
read_yaml
(path)¶ Reads a YAML file.
- Parameters
path – the filepath
- Returns
a list of JSON dicts
-
fiftyone.core.storage.
write_yaml
(obj, path, **kwargs)¶ Writes the object to a YAML file.
- Parameters
obj – a Python object
path – the filepath
**kwargs – optional arguments for
yaml.dump(..., **kwargs)
-
fiftyone.core.storage.
list_files
(dirpath, abs_paths=False, recursive=False, include_hidden_files=False, return_metadata=False, sort=True)¶ Lists the files in the given directory.
If the directory does not exist, an empty list is returned.
- Parameters
dirpath – the path to the directory to list
abs_paths (False) – whether to return the absolute paths to the files
recursive (False) – whether to recursively traverse subdirectories
include_hidden_files (False) – whether to include dot files
return_metadata (False) – whether to return metadata dicts for each file instead of filepaths
sort (True) – whether to sort the list of files
- Returns
a list of filepaths or metadata dicts
-
fiftyone.core.storage.
list_subdirs
(dirpath, abs_paths=False, recursive=False)¶ Lists the subdirectories in the given directory, sorted alphabetically and excluding hidden directories.
- Parameters
dirpath – the path to the directory to list
abs_paths (False) – whether to return absolute paths
recursive (False) – whether to recursively traverse subdirectories
- Returns
a list of subdirectories
-
fiftyone.core.storage.
list_buckets
(fs, abs_paths=False)¶ Lists the available buckets in the given file system.
This method returns subdirectories of
/
(or the current drive on Windows).- Parameters
fs – a
FileSystem
valueabs_paths (False) – whether to return absolute paths
- Returns
a list of buckets
-
fiftyone.core.storage.
list_available_file_systems
()¶ Lists the file systems that are currently available for use with methods like
list_files()
andlist_buckets()
.- Returns
a list of
FileSystem
values
-
fiftyone.core.storage.
get_glob_matches
(glob_patt)¶ Returns a list of file paths matching the given glob pattern.
The matches are returned in sorted order.
- Parameters
glob_patt – a glob pattern like
/path/to/files-*.jpg
- Returns
a list of file paths
-
fiftyone.core.storage.
get_glob_root
(glob_patt)¶ Finds the root directory of the given glob pattern, i.e., the deepest subdirectory that contains no glob characters.
- Parameters
glob_patt – a glob pattern like
/path/to/files-*.jpg
- Returns
the root
True/False whether the pattern contains any special characters
- Return type
a tuple of
-
fiftyone.core.storage.
copy_file
(inpath, outpath)¶ Copies the input file to the output location.
- Parameters
inpath – the input path
outpath – the output path
-
fiftyone.core.storage.
copy_files
(inpaths, outpaths, skip_failures=False, progress=False)¶ Copies the files to the given locations.
- Parameters
inpaths – a list of input paths
outpaths – a list of output paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation
-
fiftyone.core.storage.
copy_dir
(indir, outdir, overwrite=True, skip_failures=False, progress=False)¶ Copies the input directory to the output directory.
- Parameters
indir – the input directory
outdir – the output directory
overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation
-
fiftyone.core.storage.
move_file
(inpath, outpath)¶ Moves the given file to a new location.
- Parameters
inpath – the input path
outpath – the output path
-
fiftyone.core.storage.
move_files
(inpaths, outpaths, skip_failures=False, progress=False)¶ Moves the files to the given locations.
- Parameters
inpaths – a list of input paths
outpaths – a list of output paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation
-
fiftyone.core.storage.
move_dir
(indir, outdir, overwrite=True, skip_failures=False, progress=False)¶ Moves the contents of the given directory into the given output directory.
- Parameters
indir – the input directory
outdir – the output directory
overwrite (True) – whether to delete an existing output directory (True) or merge its contents (False)
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation
-
fiftyone.core.storage.
delete_file
(path)¶ Deletes the file at the given path.
Any empty directories are also recursively deleted from the resulting directory tree.
- Parameters
path – the filepath
-
fiftyone.core.storage.
delete_files
(paths, skip_failures=False, progress=False)¶ Deletes the files from the given locations.
Any empty directories are also recursively deleted from the resulting directory tree.
- Parameters
paths – a list of paths
skip_failures (False) – whether to gracefully continue without raising an error if an operation fails
progress (False) – whether to render a progress bar tracking the status of the operation
-
fiftyone.core.storage.
delete_dir
(dirpath)¶ Deletes the given directory and recursively deletes any empty directories from the resulting directory tree.
- Parameters
dirpath – the directory path
-
fiftyone.core.storage.
run
(fcn, tasks, num_workers=None, progress=False)¶ Applies the given function to each element of the given tasks.
- Parameters
fcn – a function that accepts a single argument
tasks – an iterable of function aguments
num_workers (None) – a suggested number of threads to use
progress (False) – whether to render a progress bar tracking the status of the operation
- Returns
the list of function outputs