fiftyone.core.utils¶
Core utilities.
Functions:
|
Extracts keyword arguments for the given class’s constructor from the given kwargs. |
|
Extracts keyword arguments for the given function from the given kwargs. |
|
Pretty-prints the Python object. |
|
Returns a pretty string representation of the Python object. |
|
Splits the given fields into sample and frame fields. |
|
Streams the iterable of objects to stdout via |
|
Indents the lines in the given string. |
|
Justifies the headings in a list of |
Returns the available patterns that can be used by |
|
|
Fills the patterns in in the given string. |
|
Finds all files in the given root directory whose filename matches the given glob pattern(s). |
|
Installs the given package via |
|
Verifies that the given package is installed. |
|
Loads the package requirements from a |
|
Installs the package requirements from a |
|
Verifies that the package requirements from a |
|
Verifies that the given requirement is installed and importable. |
|
Verifies that |
|
Verifies that |
|
Verifies that |
|
Handles the error at the specified error level. |
|
Returns a proxy module object that will lazily import the given module the first time it is used. |
|
Loads the XML file as a JSON dictionary. |
|
Parses the given object as an instance of the given |
|
Uses the |
|
Wraps the provided progress function such that it will only be called at the specified increments or time intervals. |
|
Returns a |
|
Parses the given batching strategy configuration, applying any default config settings as necessary. |
|
Computes a recommended batch size for the given value type such that a request involving a list of values of this size will be less than |
Context manager that temporarily disables all progress bars. |
|
|
A safe version of |
|
Computes the hash of the given file. |
|
Serializes a numpy array. |
|
Loads a serialized numpy array generated by |
|
Iterates over the given iterable in batches. |
|
Iterates over batches of the given object via slicing. |
|
Registers the given callback function so that it will be called when the process exits for (almost) any reason |
Determines whether the system is an ARM-based Mac (Apple Silicon). |
|
Determines whether the system is 32-bit. |
|
Determines if we’re currently running as a container. |
|
Returns the preferred |
|
|
Recommends a number of workers for a thread pool. |
|
Recommends a number of workers for a process pool. |
|
Run a synchronous function as an async background task. |
Converts a datetime.date or datetime.datetime to milliseconds since epoch. |
|
Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime. |
|
|
Converts a datetime.timedelta to milliseconds. |
|
Returns the URL-friendly slug for the given string. |
|
Validates that the given value is a valid css color name. |
|
Validates that the given value is a hex color string or css name. |
Classes:
|
Context manager that allows for a temporary change to the level of a |
||||||||||||||||||||||||||||||||||||||||
|
Proxy module that lazily imports the underlying module the first time it is actually used. |
||||||||||||||||||||||||||||||||||||||||
|
Context manager that allows for a temporary change to a resource limit exposed by the |
||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||
|
Base class for iterating over the elements of an iterable in batches. |
||||||||||||||||||||||||||||||||||||||||
|
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired target measurement. |
||||||||||||||||||||||||||||||||||||||||
|
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency. |
||||||||||||||||||||||||||||||||||||||||
|
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size. |
||||||||||||||||||||||||||||||||||||||||
|
Class for iterating over the elements of an iterable with a static batch size. |
||||||||||||||||||||||||||||||||||||||||
|
A class that generates unique output paths in a directory. |
||||||||||||||||||||||||||||||||||||||||
|
Context manager that temporarily monkey patches the given function. |
||||||||||||||||||||||||||||||||||||||||
|
Context manager that temporarily sets the attributes of a class to new values. |
||||||||||||||||||||||||||||||||||||||||
|
Context manager that temporarily disables system-wide logging. |
||||||||||||||||||||||||||||||||||||||||
|
Context manager that temporarily inserts a path to |
||||||||||||||||||||||||||||||||||||||||
|
Wrapper around a |
-
fiftyone.core.utils.
extract_kwargs_for_class
(cls, kwargs)¶ Extracts keyword arguments for the given class’s constructor from the given kwargs.
- Parameters
cls – a class
kwargs – a dictionary of keyword arguments
- Returns
a tuple of
class_kwargs: a dictionary of keyword arguments for
cls
other_kwargs: a dictionary containing the remaining
kwargs
-
fiftyone.core.utils.
extract_kwargs_for_function
(fcn, kwargs)¶ Extracts keyword arguments for the given function from the given kwargs.
- Parameters
fcn – a function
kwargs – a dictionary of keyword arguments
- Returns
a tuple of
fcn_kwargs: a dictionary of keyword arguments for
fcn
other_kwargs: a dictionary containing the remaining
kwargs
-
fiftyone.core.utils.
pprint
(obj, stream=None, indent=4, width=80, depth=None)¶ Pretty-prints the Python object.
- Parameters
obj – the Python object
stream (None) – the stream to write to. The default is
sys.stdout
indent (4) – the number of spaces to use when indenting
width (80) – the max width of each line in the pretty representation
depth (None) – the maximum depth at which to pretty render nested dicts
-
fiftyone.core.utils.
pformat
(obj, indent=4, width=80, depth=None)¶ Returns a pretty string representation of the Python object.
- Parameters
obj – the Python object
indent (4) – the number of spaces to use when indenting
width (80) – the max width of each line in the pretty representation
depth (None) – the maximum depth at which to pretty render nested dicts
- Returns
the pretty-formatted string
-
fiftyone.core.utils.
split_frame_fields
(fields)¶ Splits the given fields into sample and frame fields.
Frame fields are those prefixed by
"frames."
, and this prefix is removed from the returned frame fields.- Parameters
fields – a field, iterable of fields, or dict mapping field names to new field names
- Returns
a tuple of
sample_fields: a list or dict of sample fields
frame_fields: a list or dict of frame fields
-
fiftyone.core.utils.
stream_objects
(objects)¶ Streams the iterable of objects to stdout via
less
.The output can be interactively traversed via scrolling and can be terminated via keyboard interrupt.
- Parameters
objects – an iterable of objects that can be printed via
str(obj)
-
fiftyone.core.utils.
indent_lines
(s, indent=4, skip=0)¶ Indents the lines in the given string.
- Parameters
s – the string
indent (4) – the number of spaces to indent
skip (0) – the number of lines to skip before indenting
- Returns
the indented string
-
fiftyone.core.utils.
justify_headings
(elements, width=None)¶ Justifies the headings in a list of
(heading, content)
string tuples by appending whitespace as necessary to eachheading
.- Parameters
elements – a list of
(heading, content)
tupleswidth (None) – an optional justification width. By default, the maximum heading length is used
- Returns
a list of justified
(heading, content)
tuples
-
fiftyone.core.utils.
available_patterns
()¶ Returns the available patterns that can be used by
fill_patterns()
.- Returns
a dict mapping patterns to their replacements
-
fiftyone.core.utils.
fill_patterns
(string)¶ Fills the patterns in in the given string.
Use
available_patterns()
to see the available patterns that can be used.- Parameters
string – a string
- Returns
a copy of string with any patterns replaced
-
fiftyone.core.utils.
find_files
(root_dir, patt, max_depth=1)¶ Finds all files in the given root directory whose filename matches the given glob pattern(s).
Both
root_dir
andpatt
may contain glob patterns.Exammples:
import fiftyone.core.utils as fou # Find .txt files in `/tmp` fou.find_files("/tmp", "*.txt") # Find .txt files in subdirectories of `/tmp` that begin with `foo-` fou.find_files("/tmp/foo-*", "*.txt") # Find .txt files in `/tmp` or its subdirectories fou.find_files("/tmp", "*.txt", max_depth=2)
- Parameters
root_dir – the root directory
patt – a glob pattern or list of patterns
max_depth (1) – a maximum depth to search. 1 means
root_dir
only, 2 meansroot_dir
and its immediate subdirectories, etc
- Returns
a list of matching paths
-
fiftyone.core.utils.
install_package
(requirement_str, error_level=None, error_msg=None)¶ Installs the given package via
pip
.Installation is performed via:
python -m pip install <requirement_str>
- Parameters
requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”
error_level (None) –
the error level to use, defined as:
0: raise error if the install fails
1: log warning if the install fails
2: ignore install fails
error_msg (None) – an optional custom error message to use
-
fiftyone.core.utils.
ensure_package
(requirement_str, error_level=None, error_msg=None, log_success=False)¶ Verifies that the given package is installed.
This function uses
importlib.metadata
to locate the package by its pip name and does not actually import the module.Therefore, unlike
ensure_import()
,requirement_str
should refer to the package name (e.g., “tensorflow-gpu”), not the module name (e.g., “tensorflow”).- Parameters
requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usederror_msg (None) – an optional custom error message to use
log_success (False) – whether to generate a log message if the requirement is satisfied
- Returns
True/False whether the requirement is satisfied
-
fiftyone.core.utils.
load_requirements
(requirements_path)¶ Loads the package requirements from a
requirements.txt
file on disk.Comments and extra whitespace are automatically stripped.
- Parameters
requirements_path – the path to a requirements file
- Returns
a list of requirement strings
-
fiftyone.core.utils.
install_requirements
(requirements_path, error_level=None)¶ Installs the package requirements from a
requirements.txt
file on disk.- Parameters
requirements_path – the path to a requirements file
error_level (None) –
the error level to use, defined as:
0: raise error if the install fails
1: log warning if the install fails
2: ignore install fails
By default,
fiftyone.config.requirement_error_level
is used
-
fiftyone.core.utils.
ensure_requirements
(requirements_path, error_level=None, log_success=False)¶ Verifies that the package requirements from a
requirements.txt
file on disk are installed.- Parameters
requirements_path – the path to a requirements file
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usedlog_success (False) – whether to generate a log message if a requirement is satisfied
-
fiftyone.core.utils.
ensure_import
(requirement_str, error_level=None, error_msg=None, log_success=False)¶ Verifies that the given requirement is installed and importable.
This function imports the specified module and optionally enforces any version requirements included in
requirement_str
.Therefore, unlike
ensure_package()
,requirement_str
should refer to the module name (e.g., “tensorflow”), not the package name (e.g., “tensorflow-gpu”).- Parameters
requirement_str – a PEP 440-like module requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usederror_msg (None) – an optional custom error message to use
log_success (False) – whether to generate a log message if the requirement is satisfied
- Returns
True/False whether the requirement is satisfied
-
fiftyone.core.utils.
ensure_tf
(eager=False, error_level=None, error_msg=None)¶ Verifies that
tensorflow
is installed and importable.- Parameters
eager (False) – whether to require that TF is executing eagerly. If True and TF is not currently executing eagerly, this method will attempt to enable it
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usederror_msg (None) – an optional custom error message to print
- Returns
True/False whether the requirement is satisfied
-
fiftyone.core.utils.
ensure_tfds
(error_level=None, error_msg=None)¶ Verifies that
tensorflow_datasets
is installed and importable.- Parameters
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usederror_msg (None) – an optional custom error message to print
- Returns
True/False whether the requirement is satisfied
-
fiftyone.core.utils.
ensure_torch
(error_level=None, error_msg=None)¶ Verifies that
torch
andtorchvision
are installed and importable.- Parameters
error_level (None) –
the error level to use, defined as:
0: raise error if requirement is not satisfied
1: log warning if requirement is not satisfied
2: ignore unsatisifed requirements
By default,
fiftyone.config.requirement_error_level
is usederror_msg (None) – an optional custom error message to print
- Returns
True/False whether the requirement is satisfied
-
fiftyone.core.utils.
handle_error
(error, error_level, base_error=None)¶ Handles the error at the specified error level.
- Parameters
error – an Exception instance
error_level – the error level to use, defined as:
0 (-) – raise the error
1 (-) – log the error as a warning
2 (-) – ignore the error
base_error – (optional) a base Exception from which to raise
error
-
class
fiftyone.core.utils.
LoggingLevel
(level, logger=None)¶ Bases:
object
Context manager that allows for a temporary change to the level of a
logging.Logger
.Example:
import logging import fiftyone.core.utils as fou with fou.LoggingLevel(logging.CRITICAL): # do things with all logging at CRITICAL with fou.LoggingLevel(logging.ERROR, logger="fiftyone"): # do things with FiftyOne logging at ERROR Args: level: the logging level to use, e.g., ``logging.ERROR`` logger (None): a ``logging.Logger`` or the name of a logger. By default, the root logger is used
-
fiftyone.core.utils.
lazy_import
(module_name, callback=None)¶ Returns a proxy module object that will lazily import the given module the first time it is used.
Example usage:
# Lazy version of `import tensorflow as tf` tf = lazy_import("tensorflow") # Other commands # Now the module is loaded tf.__version__
- Parameters
module_name – the fully-qualified module name to import
callback (None) – a callback function to call before importing the module
- Returns
a proxy module object that will be lazily imported when first used
-
class
fiftyone.core.utils.
LazyModule
(module_name, callback=None)¶ Bases:
module
Proxy module that lazily imports the underlying module the first time it is actually used.
- Parameters
module_name – the fully-qualified module name to import
callback (None) – a callback function to call before importing the module
-
fiftyone.core.utils.
load_xml_as_json_dict
(xml_path)¶ Loads the XML file as a JSON dictionary.
- Parameters
xml_path – the path to the XML file
- Returns
a JSON dict
-
fiftyone.core.utils.
parse_serializable
(obj, cls)¶ Parses the given object as an instance of the given
eta.core.serial.Serializable
class.- Parameters
obj – an instance of
cls
, or a serialized string or dictionary representation of onecls – a
eta.core.serial.Serializable
class
- Returns
an instance of
cls
-
fiftyone.core.utils.
set_resource_limit
(limit, soft=None, hard=None, warn_on_failure=False)¶ Uses the
resource
package to change a resource limit for the current process.If the
resource
package cannot be imported, this command does nothing.- Parameters
limit – the name of the resource to limit. Must be the name of a constant in the
resource
module starting withRLIMIT
. See the documentation of theresource
module for supported valuessoft (None) – a new soft limit to apply, which cannot exceed the hard limit. If omitted, the current soft limit is maintained
hard (None) – a new hard limit to apply. If omitted, the current hard limit is maintained
warn_on_failure (False) – whether to issue a warning rather than an error if the resource limit change is not successful
-
class
fiftyone.core.utils.
ResourceLimit
(limit, soft=None, hard=None, warn_on_failure=False)¶ Bases:
object
Context manager that allows for a temporary change to a resource limit exposed by the
resource
package.Example:
import resource import fiftyone.core.utils as fou with fou.ResourceLimit(resource.RLIMIT_NOFILE, soft=4096): # temporarily do things with up to 4096 open files Args: limit: the name of the resource to limit. Must be the name of a constant in the ``resource`` module starting with ``RLIMIT``. See the documentation of the ``resource`` module for supported values soft (None): a new soft limit to apply, which cannot exceed the hard limit. If omitted, the current soft limit is maintained hard (None): a new hard limit to apply. If omitted, the current hard limit is maintained warn_on_failure (False): whether to issue a warning rather than an error if the resource limit change is not successful
-
class
fiftyone.core.utils.
ProgressBar
(total=None, progress=None, quiet=None, **kwargs)¶ Bases:
eta.core.utils.ProgressBar
-
class
ProgressBar
(total=None, show_percentage=True, show_iteration=True, show_elapsed_time=True, show_remaining_time=True, show_iter_rate=True, iters_str='it', start_msg=None, complete_msg=None, use_bits=False, use_bytes=False, max_width=None, num_decimals=1, max_fps=10, quiet=None)¶ Class for printing a self-updating progress bar to stdout that tracks the progress of an iterative process towards completion.
The simplest use of ProgressBar is to create an instance and then call it (__call__) with an iterable argument, which will pass through elements of the iterable when iterating over it, updating the progress bar each time an element is emitted.
Alternatively, ProgressBar`s can track the progress of a task towards a total iteration count, where the current iteration is updated manually by calling either `update(), which will increment the iteration count by 1, or via set_iteration(), which lets you manually specify the new iteration status. By default, both methods will automatically redraw the bar. If manual control over drawing is required, you can pass draw=False to either method and then manually call draw() as desired.
It is highly recommended that you always invoke ProgressBar`s using their context manager interface via the `with keyword, which will automatically handle calling start() and close() to appropriately initialize and finalize the task. Among other things, this ensures that stdout redirection is appropriately ended when an exception is encountered.
When start() is called on the a ProgressBar, an internal timer is started to track the elapsed time of the task. In addition, stdout is automatically cached between calls to draw() and flushed each time draw() is called, which means that you can freely mix print statements into your task without interfering with the progress bar. When you are done tracking the task, call the close() method to finalize the progress bar. Both of these actions are automatically taken when the bar’s context manager interface is invoked or when it is called with an iterable.
If you want want to full manual control over the progress bar, call start() to start the task, call pause() before any print statements, and call close() when the task is finalized.
ProgressBar can optionally be configured to print any of the following statistics about the task:
the elapsed time since the task was started
the estimated time remaining until the task completes
the current iteration rate of the task, in iterations per second
customized status messages passed via the suffix argument
Example (wrapping an iterator):
import time import eta.core.utils as etau with etau.ProgressBar() as pb: for _ in pb(range(100)): time.sleep(0.05)
Example (with print statements interleaved):
import time import eta.core.utils as etau with etau.ProgressBar(100) as pb: while not pb.complete: if pb.iteration in {25, 50, 75}: print("Progress = %.2f" % pb.progress) time.sleep(0.05) pb.update()
Example (tracking a bit total):
import random import time import eta.core.utils as etau with etau.ProgressBar(1024 ** 2, use_bits=True) as pb: while not pb.complete: pb.update(random.randint(1, 10 * 1024)) time.sleep(0.05)
Attributes:
is_iterable
Whether the progress bar is tracking an iterable.
is_running
Whether the progress bar is running, i.e., it is between start() and close() calls.
is_finalized
Whether the progress bar is finalized, i.e., close() has been called.
is_capturing_stdout
Whether stdout is being captured between calls to draw().
has_dynamic_width
Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
has_total
Whether the progress bar has a total iteration count.
total
The total iterations, or None if not available.
iteration
The current iteration.
progress
The current progress, in [0, 1], or None if the progress bar has no total.
complete
Whether the task is 100%% complete, or None if the progress bar has no total.
elapsed_time
The elapsed time since the task was started, or None if this progress bar is not timing.
time_remaining
The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
iter_rate
The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
quiet
Whether the progress bar is in quiet mode (no printing to stdout).
Methods:
start
()Starts the progress bar.
close
(*args)Closes the progress bar.
update
([count, suffix, draw])Increments the current iteration count by the given value (default = 1).
set_iteration
(iteration[, suffix, draw])Sets the current iteration.
pause
()Pauses the progress bar so that other information can be printed.
draw
([force])Draws the progress bar at its current progress.
-
property
ProgressBar.
is_iterable
¶ Whether the progress bar is tracking an iterable.
-
property
ProgressBar.
is_running
¶ Whether the progress bar is running, i.e., it is between start() and close() calls.
-
property
ProgressBar.
is_finalized
¶ Whether the progress bar is finalized, i.e., close() has been called.
-
property
ProgressBar.
is_capturing_stdout
¶ Whether stdout is being captured between calls to draw().
-
property
ProgressBar.
has_dynamic_width
¶ Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
-
property
ProgressBar.
has_total
¶ Whether the progress bar has a total iteration count.
-
property
ProgressBar.
total
¶ The total iterations, or None if not available.
-
property
ProgressBar.
iteration
¶ The current iteration.
-
property
ProgressBar.
progress
¶ The current progress, in [0, 1], or None if the progress bar has no total.
-
property
ProgressBar.
complete
¶ Whether the task is 100%% complete, or None if the progress bar has no total.
-
property
ProgressBar.
elapsed_time
¶ The elapsed time since the task was started, or None if this progress bar is not timing.
-
property
ProgressBar.
time_remaining
¶ The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
-
property
ProgressBar.
iter_rate
¶ The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
-
property
ProgressBar.
quiet
¶ Whether the progress bar is in quiet mode (no printing to stdout).
-
ProgressBar.
start
()¶ Starts the progress bar.
-
ProgressBar.
close
(*args)¶ Closes the progress bar.
-
ProgressBar.
update
(count=1, suffix=None, draw=True)¶ Increments the current iteration count by the given value (default = 1).
This method is shorthand for self.set_iteration(self.iteration + count).
- Parameters
count – the iteration increment. The default is 1
suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged
draw – whether to call draw() at the end of this method. By default, this is True
-
ProgressBar.
set_iteration
(iteration, suffix=None, draw=True)¶ Sets the current iteration.
- Parameters
iteration – the new iteration
suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”
draw – whether to call draw() at the end of this method. By default, this is True
-
ProgressBar.
pause
()¶ Pauses the progress bar so that other information can be printed.
This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.
-
ProgressBar.
draw
(force=False)¶ Draws the progress bar at its current progress.
- Parameters
force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second
Methods:
set_iteration
(*args, **kwargs)Sets the current iteration.
close
(*args)Closes the progress bar.
draw
([force])Draws the progress bar at its current progress.
pause
()Pauses the progress bar so that other information can be printed.
start
()Starts the progress bar.
update
([count, suffix, draw])Increments the current iteration count by the given value (default = 1).
Attributes:
Whether the task is 100%% complete, or None if the progress bar has no total.
The elapsed time since the task was started, or None if this progress bar is not timing.
Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
Whether the progress bar has a total iteration count.
Whether stdout is being captured between calls to draw().
Whether the progress bar is finalized, i.e., close() has been called.
Whether the progress bar is tracking an iterable.
Whether the progress bar is running, i.e., it is between start() and close() calls.
The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
The current iteration.
The current progress, in [0, 1], or None if the progress bar has no total.
Whether the progress bar is in quiet mode (no printing to stdout).
The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
The total iterations, or None if not available.
-
set_iteration
(*args, **kwargs)¶ Sets the current iteration.
- Parameters
iteration – the new iteration
suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”
draw – whether to call draw() at the end of this method. By default, this is True
-
close
(*args)¶ Closes the progress bar.
-
property
complete
¶ Whether the task is 100%% complete, or None if the progress bar has no total.
-
draw
(force=False)¶ Draws the progress bar at its current progress.
- Parameters
force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second
-
property
elapsed_time
¶ The elapsed time since the task was started, or None if this progress bar is not timing.
-
property
has_dynamic_width
¶ Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
-
property
has_total
¶ Whether the progress bar has a total iteration count.
-
property
is_capturing_stdout
¶ Whether stdout is being captured between calls to draw().
-
property
is_finalized
¶ Whether the progress bar is finalized, i.e., close() has been called.
-
property
is_iterable
¶ Whether the progress bar is tracking an iterable.
-
property
is_running
¶ Whether the progress bar is running, i.e., it is between start() and close() calls.
-
property
iter_rate
¶ The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
-
property
iteration
¶ The current iteration.
-
pause
()¶ Pauses the progress bar so that other information can be printed.
This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.
-
property
progress
¶ The current progress, in [0, 1], or None if the progress bar has no total.
-
property
quiet
¶ Whether the progress bar is in quiet mode (no printing to stdout).
-
start
()¶ Starts the progress bar.
-
property
time_remaining
¶ The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
-
property
total
¶ The total iterations, or None if not available.
-
update
(count=1, suffix=None, draw=True)¶ Increments the current iteration count by the given value (default = 1).
This method is shorthand for self.set_iteration(self.iteration + count).
- Parameters
count – the iteration increment. The default is 1
suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged
draw – whether to call draw() at the end of this method. By default, this is True
-
class
-
fiftyone.core.utils.
report_progress
(progress, n=None, dt=None)¶ Wraps the provided progress function such that it will only be called at the specified increments or time intervals.
Example usage:
import fiftyone as fo import fiftyone.zoo as foz def print_progress(pb): if pb.complete: print("COMPLETE") else: print("PROGRESS: %0.3f" % pb.progress) dataset = foz.load_zoo_dataset("cifar10", split="test") # Print progress at 10 equally-spaced increments progress = fo.report_progress(print_progress, n=10) dataset.compute_metadata(progress=progress) # Print progress every 0.5 seconds progress = fo.report_progress(print_progress, dt=0.5) dataset.compute_metadata(progress=progress, overwrite=True)
- Parameters
progress – a function that accepts a
ProgressBar
as inputn (None) – a number of equally-spaced increments to invoke
progress
dt (None) – a number of seconds between
progress
calls
- Returns
a function that accepts a
ProgressBar
as input
-
class
fiftyone.core.utils.
Batcher
(iterable, return_views=False, progress=False, total=None)¶ Bases:
abc.ABC
Base class for iterating over the elements of an iterable in batches.
Attributes:
Methods:
apply_backpressure
(*args, **kwargs)Apply backpressure needed to rightsize the next batch.
-
manual_backpressure
= False¶
-
apply_backpressure
(*args, **kwargs)¶ Apply backpressure needed to rightsize the next batch.
Required to be implemented and called every iteration, if
self.manual_backpressure == True
.Subclass defines arguments and behavior of this method.
-
-
class
fiftyone.core.utils.
BaseDynamicBatcher
(iterable, target_measurement, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, return_views=False, progress=False, total=None)¶ Bases:
fiftyone.core.utils.Batcher
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired target measurement.
The batch sizes emitted when iterating over this object are dynamically scaled such that the measurement between
next()
calls is as close as possible to a specified target.Concrete base classes define the target measurement and method of calculation.
Methods:
apply_backpressure
(*args, **kwargs)Apply backpressure needed to rightsize the next batch.
Attributes:
-
apply_backpressure
(*args, **kwargs)¶ Apply backpressure needed to rightsize the next batch.
Required to be implemented and called every iteration, if
self.manual_backpressure == True
.Subclass defines arguments and behavior of this method.
-
manual_backpressure
= False¶
-
-
class
fiftyone.core.utils.
LatencyDynamicBatcher
(iterable, target_latency=0.2, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, return_views=False, progress=False, total=None)¶ Bases:
fiftyone.core.utils.BaseDynamicBatcher
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.
The batch sizes emitted when iterating over this object are dynamically scaled such that the latency between
next()
calls is as close as possible to a specified target latency.This class is often used in conjunction with a
ProgressBar
to keep the user appraised on the status of a long-running task.Example usage:
import fiftyone.core.utils as fou elements = range(int(1e7)) batcher = fou.LatencyDynamicBatcher( elements, target_latency=0.1, max_batch_beta=2.0 ) for batch in batcher: print("batch size: %d" % len(batch)) batcher = fou.LatencyDynamicBatcher( elements, target_latency=0.1, max_batch_beta=2.0, progress=True, ) with batcher: for batch in batcher: print("batch size: %d" % len(batch))
- Parameters
iterable – an iterable to batch over. If
None
, the result ofnext()
will be a batch size instead of a batch, and is an infinite iterator.target_latency (0.2) – the target latency between
next()
calls, in secondsinit_batch_size (1) – the initial batch size to use
min_batch_size (1) – the minimum allowed batch size
max_batch_size (None) – an optional maximum allowed batch size
max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes
return_views (False) – whether to return each batch as a
fiftyone.core.view.DatasetView
. Only applicable when the iterable is afiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke insteadtotal (None) – the length of
iterable
. Only applicable whenprogress=True
. If not provided, it is computed vialen(iterable)
, if possible
Methods:
apply_backpressure
(*args, **kwargs)Apply backpressure needed to rightsize the next batch.
Attributes:
-
apply_backpressure
(*args, **kwargs)¶ Apply backpressure needed to rightsize the next batch.
Required to be implemented and called every iteration, if
self.manual_backpressure == True
.Subclass defines arguments and behavior of this method.
-
manual_backpressure
= False¶
-
fiftyone.core.utils.
DynamicBatcher
¶ alias of
fiftyone.core.utils.LatencyDynamicBatcher
Methods:apply_backpressure
(*args, **kwargs)Apply backpressure needed to rightsize the next batch.
Attributes:
manual_backpressure
-
class
fiftyone.core.utils.
ContentSizeDynamicBatcher
(iterable, target_size=1048576, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, return_views=False, progress=False, total=None)¶ Bases:
fiftyone.core.utils.BaseDynamicBatcher
Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.
The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.
This batcher requires that backpressure feedback be provided, either by providing a BSON-able batch from which the content size can be computed, or by manually providing the content size.
This class is often used in conjunction with a
ProgressBar
to keep the user appraised on the status of a long-running task.Example usage:
import fiftyone.core.utils as fou elements = range(int(1e7)) batcher = fou.ContentSizeDynamicBatcher( elements, target_size=2**20, max_batch_beta=2.0 ) # Raises ValueError after first batch, we forgot to apply backpressure for batch in batcher: print("batch size: %d" % len(batch)) # Now it works for batch in batcher: print("batch size: %d" % len(batch)) batcher.apply_backpressure(batch) batcher = fou.ContentSizeDynamicBatcher( elements, target_size=2**20, max_batch_beta=2.0, progress=True ) with batcher: for batch in batcher: print("batch size: %d" % len(batch)) batcher.apply_backpressure(batch)
- Parameters
iterable – an iterable to batch over. If
None
, the result ofnext()
will be a batch size instead of a batch, and is an infinite iterator.target_size (1048576) – the target batch bson content size, in bytes
init_batch_size (1) – the initial batch size to use
min_batch_size (1) – the minimum allowed batch size
max_batch_size (None) – an optional maximum allowed batch size
max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes
return_views (False) – whether to return each batch as a
fiftyone.core.view.DatasetView
. Only applicable when the iterable is afiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke insteadtotal (None) – the length of
iterable
. Only applicable whenprogress=True
. If not provided, it is computed vialen(iterable)
, if possible
Attributes:
Methods:
apply_backpressure
(batch_or_size)Apply backpressure needed to rightsize the next batch.
-
manual_backpressure
= True¶
-
apply_backpressure
(batch_or_size)¶ Apply backpressure needed to rightsize the next batch.
Required to be implemented and called every iteration, if
self.manual_backpressure == True
.Subclass defines arguments and behavior of this method.
-
class
fiftyone.core.utils.
StaticBatcher
(iterable, batch_size, return_views=False, progress=False, total=None)¶ Bases:
fiftyone.core.utils.Batcher
Class for iterating over the elements of an iterable with a static batch size.
This class is often used in conjunction with a
ProgressBar
to keep the user appraised on the status of a long-running task.Example usage:
import fiftyone.core.utils as fou elements = range(int(1e7)) batcher = fou.StaticBatcher(elements, batch_size=10000) for batch in batcher: print("batch size: %d" % len(batch)) batcher = fou.StaticBatcher(elements, batch_size=10000, progress=True) with batcher: for batch in batcher: print("batch size: %d" % len(batch))
- Parameters
iterable – an iterable to batch over. If
None
, the result ofnext()
will be a batch size instead of a batch, and is an infinite iterator.batch_size – size of batches to generate
return_views (False) – whether to return each batch as a
fiftyone.core.view.DatasetView
. Only applicable when the iterable is afiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke insteadtotal (None) – the length of
iterable
. Only applicable whenprogress=True
. If not provided, it is computed vialen(iterable)
, if possible
Methods:
apply_backpressure
(*args, **kwargs)Apply backpressure needed to rightsize the next batch.
Attributes:
-
apply_backpressure
(*args, **kwargs)¶ Apply backpressure needed to rightsize the next batch.
Required to be implemented and called every iteration, if
self.manual_backpressure == True
.Subclass defines arguments and behavior of this method.
-
manual_backpressure
= False¶
-
fiftyone.core.utils.
get_default_batcher
(iterable, progress=False, total=None)¶ Returns a
Batcher
overiterable
using defaults from your FiftyOne config.Uses
fiftyone.config.default_batcher
to determine the implementation to use, and related configuration values as needed for each.- Parameters
iterable – an iterable to batch over. If
None
, the result ofnext()
will be a batch size instead of a batch, and is an infinite iterator.progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value
fiftyone.config.show_progress_bars
(None), or a progress callback function to invoke insteadtotal (None) – the length of
iterable
. Only applicable whenprogress=True
. If not provided, it is computed vialen(iterable)
, if possible
- Returns
a
Batcher
-
fiftyone.core.utils.
parse_batching_strategy
(batch_size=None, batching_strategy=None)¶ Parses the given batching strategy configuration, applying any default config settings as necessary.
- Parameters
batch_size (None) – the batch size to use. If a
batching_strategy
is provided, this parameter configures that strategy as described below. If nobatching_strategy
is provided, this can either be an integer specifying the number of samples to save in a batch (in which casebatching_strategy
is implicitly set to"static"
) or a float number of seconds between batched saves (in which casebatching_strategy
is implicitly set to"latency"
)batching_strategy (None) –
the batching strategy to use for each save operation. Supported values are:
"static"
: a fixed sample batch size for each save"size"
: a target batch size, in bytes, for each save"latency"
: a target latency, in seconds, between saves
By default,
fo.config.default_batcher
is used
- Returns
a tuple of
(batch_size, batching_strategy)
-
fiftyone.core.utils.
recommend_batch_size_for_value
(value, alpha=0.9, max_size=None)¶ Computes a recommended batch size for the given value type such that a request involving a list of values of this size will be less than
alpha * fo.config.batcher_target_size_bytes
bytes.- Parameters
value – a value
alpha (0.9) – a safety factor
max_size (None) – an optional max batch size
- Returns
a recommended batch size
-
fiftyone.core.utils.
disable_progress_bars
()¶ Context manager that temporarily disables all progress bars.
Example usage:
import fiftyone as fo import fiftyone.zoo as foz with fo.disable_progress_bars(): dataset = foz.load_zoo_dataset("quickstart")
-
class
fiftyone.core.utils.
UniqueFilenameMaker
(output_dir=None, rel_dir=None, alt_dir=None, chunk_size=None, default_ext=None, ignore_exts=False, ignore_existing=False, idempotent=True)¶ Bases:
object
A class that generates unique output paths in a directory.
This class provides a
get_output_path()
method that generates unique filenames in the specified output directory.If an input path is provided, its filename is maintained, unless a name conflict in
output_dir
would occur, in which case an index of the form"-%d" % count
is appended to the filename.If no input filename is provided, an output filename of the form
<output_dir>/<count><default_ext>
is generated, wherecount
is the number of files inoutput_dir
.If no
output_dir
is provided, then unique filenames with no base directory are generated.If a
rel_dir
is provided, then this path will be stripped from each input path to generate the identifier of each file (rather than just its basename). This argument allows for populating nested subdirectories inoutput_dir
that match the shape of the input paths.If
alt_dir
is provided, you can useget_alt_path()
to retrieve the equivalent path rooted in this directory rather thanoutput_dir
.- Parameters
output_dir (None) – a directory in which to generate output paths
rel_dir (None) – an optional relative directory to strip from each path. The path is converted to an absolute path (if necessary) via
fiftyone.core.storage.normalize_path()
alt_dir (None) – an optional alternate directory in which to generate paths when
get_alt_path()
is calledchunk_size (None) – if provided, output paths will be nested in subdirectories of
output_dir
with at most this many files per subdirectory. Has no effect if arel_dir
is provideddefault_ext (None) – the file extension to use when generating default output paths
ignore_exts (False) – whether to omit file extensions when checking for duplicate filenames
ignore_existing (False) – whether to ignore existing files in
output_dir
for output filename generation purposesidempotent (True) – whether to return the same output path when the same input path is provided multiple times (True) or to generate new output paths (False)
Methods:
seen_input_path
(input_path)Checks whether we’ve already seen the given input path.
get_output_path
([input_path, output_ext])Returns a unique output path.
get_alt_path
(output_path[, alt_dir])Returns the alternate path for the given output path generated by
get_output_path()
.-
seen_input_path
(input_path)¶ Checks whether we’ve already seen the given input path.
- Parameters
input_path – an input path
- Returns
True/False
-
get_output_path
(input_path=None, output_ext=None)¶ Returns a unique output path.
- Parameters
input_path (None) – an input path
output_ext (None) – an optional output extension to use
- Returns
the output path
-
get_alt_path
(output_path, alt_dir=None)¶ Returns the alternate path for the given output path generated by
get_output_path()
.- Parameters
output_path – an output path
alt_dir (None) – a directory in which to return the alternate path. If not provided,
alt_dir
is used
- Returns
the corresponding alternate path
-
fiftyone.core.utils.
safe_relpath
(path, start=None, default=None)¶ A safe version of
os.path.relpath
that returns a configurable default value if the given path if it does not lie within the given relative start.- Parameters
path – a path
start (None) – the relative prefix to strip from
path
default (None) – a default value to return if
path
does not lie withinstart
. By default, the basename of the path is returned
- Returns
the relative path
-
fiftyone.core.utils.
compute_filehash
(filepath, method=None, chunk_size=None)¶ Computes the hash of the given file.
- Parameters
filepath – the path to the file
method (None) – an optional
hashlib
method to use. If not specified, the builtinstr.__hash__
will be usedchunk_size (None) – an optional chunk size to use to read the file, in bytes. Only applicable when a
method
is provided. The default is 64kB. If negative, the entire file is read at once
- Returns
the hash
-
fiftyone.core.utils.
serialize_numpy_array
(array, ascii=False)¶ Serializes a numpy array.
- Parameters
array – a numpy array-like
ascii (False) – whether to return a base64-encoded ASCII string instead of raw bytes
- Returns
the serialized bytes
-
fiftyone.core.utils.
deserialize_numpy_array
(numpy_bytes, ascii=False)¶ Loads a serialized numpy array generated by
serialize_numpy_array()
.- Parameters
numpy_bytes – the serialized numpy array bytes
ascii (False) – whether the bytes were generated with the
ascii == True
parameter ofserialize_numpy_array()
- Returns
the numpy array
-
fiftyone.core.utils.
iter_batches
(iterable, batch_size)¶ Iterates over the given iterable in batches.
- Parameters
iterable – an iterable
batch_size – the desired batch size, or None to return the contents in a single batch
- Returns
a generator that emits tuples of elements of the requested batch size from the input
-
fiftyone.core.utils.
iter_slices
(sliceable, batch_size)¶ Iterates over batches of the given object via slicing.
- Parameters
sliceable – an object that supports slicing
batch_size – the desired batch size, or None to return the contents in a single batch
- Returns
a generator that emits batches of elements of the requested batch size from the input
-
fiftyone.core.utils.
call_on_exit
(callback)¶ Registers the given callback function so that it will be called when the process exits for (almost) any reason
Note that this should only be used from non-interactive scripts because it intercepts ctrl + c.
Covers the following cases: - normal program termination - a Python exception is raised - a SIGTERM signal is received
- Parameters
callback – the function to execute upon termination
-
class
fiftyone.core.utils.
MonkeyPatchFunction
(module_or_fcn, monkey_fcn, fcn_name=None, namespace=None)¶ Bases:
object
Context manager that temporarily monkey patches the given function.
If a
namespace
is provided, all functions with same name as the function you are monkey patching that are imported (recursively) by themodule_or_fcn
module are also monkey patched.- Parameters
module_or_fcn – a module or function
monkey_fcn – the function to monkey patch in
fcn_name (None) – the name of the function to monkey patch. Required iff
module_or_fcn
is a modulenamespace (None) – an optional package namespace
-
class
fiftyone.core.utils.
SetAttributes
(obj, **kwargs)¶ Bases:
object
Context manager that temporarily sets the attributes of a class to new values.
- Parameters
obj – the object
**kwargs – the attribute key-values to set while the context is active
-
class
fiftyone.core.utils.
SuppressLogging
(level=50)¶ Bases:
object
Context manager that temporarily disables system-wide logging.
- Parameters
level (logging.CRITICAL) – the
logging
level at or below which to suppress all messages
-
class
fiftyone.core.utils.
add_sys_path
(path, index=0)¶ Bases:
object
Context manager that temporarily inserts a path to
sys.path
.
-
fiftyone.core.utils.
is_arm_mac
()¶ Determines whether the system is an ARM-based Mac (Apple Silicon).
- Returns
True/False
-
fiftyone.core.utils.
is_32_bit
()¶ Determines whether the system is 32-bit.
- Returns
True/False
-
fiftyone.core.utils.
is_container
()¶ Determines if we’re currently running as a container.
- Returns
True/False
-
fiftyone.core.utils.
get_multiprocessing_context
()¶ Returns the preferred
multiprocessing
context for the current OS.- Returns
a
multiprocessing
context
-
fiftyone.core.utils.
recommend_thread_pool_workers
(num_workers=None)¶ Recommends a number of workers for a thread pool.
If a
fo.config.max_thread_pool_workers
is set, this limit is applied.- Parameters
num_workers (None) – a suggested number of workers
- Returns
a number of workers
-
fiftyone.core.utils.
recommend_process_pool_workers
(num_workers=None)¶ Recommends a number of workers for a process pool.
If a
fo.config.max_process_pool_workers
is set, this limit is applied.- Parameters
num_workers (None) – a suggested number of workers
- Returns
a number of workers
-
async
fiftyone.core.utils.
run_sync_task
(func, *args)¶ Run a synchronous function as an async background task.
- Parameters
func – a synchronous callable
*args – function arguments
- Returns
the function’s return value(s)
-
fiftyone.core.utils.
datetime_to_timestamp
(dt)¶ Converts a datetime.date or datetime.datetime to milliseconds since epoch.
- Parameters
dt – a datetime.date or datetime.datetime
- Returns
the float number of milliseconds since epoch
-
fiftyone.core.utils.
timestamp_to_datetime
(ts)¶ Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime.
- Parameters
ts – a number of milliseconds since epoch
- Returns
a datetime.datetime
-
fiftyone.core.utils.
timedelta_to_ms
(td)¶ Converts a datetime.timedelta to milliseconds.
- Parameters
td – a datetime.timedelta
- Returns
the float number of milliseconds
-
class
fiftyone.core.utils.
ResponseStream
(response, chunk_size=64)¶ Bases:
object
Wrapper around a
requests.Response
that provides a file-like object interface withread()
,seek()
, andtell()
methods.- Parameters
response – a
requests.Response
chunk_size (64) – the chunk size to use to read the response’s content
Methods:
read
([size])seek
(position[, whence])tell
()-
read
(size=None)¶
-
seek
(position, whence=0)¶
-
tell
()¶
-
fiftyone.core.utils.
to_slug
(name)¶ Returns the URL-friendly slug for the given string.
The following strategy is used to generate slugs:
The characters
A-Za-z0-9
are converted to lowercaseWhitespace and
+_.-
are converted to-
All other characters are omitted
All consecutive
-
characters are reduced to a single-
All leading and trailing
-
are strippedBoth the input name and the resulting string must be
[1, 100]
characters in length
Examples:
name | slug ---------------------------------+----------------------- coco_2017 | coco-2017 c+o+c+o 2-0-1-7 | c-o-c-o-2-0-1-7 cat.DOG | cat-dog ---name---- | name Brian's #$&@ (Awesome?) Dataset! | brians-awesome-dataset sPaM aNd EgGs | spam-and-eggs
- Parameters
name – a string
- Returns
the slug string
- Raises
ValueError – if the name is invalid
-
fiftyone.core.utils.
validate_color
(value)¶ Validates that the given value is a valid css color name.
- Parameters
value – a value
- Raises
ValueError – if
value
is not a valid css color name.
-
fiftyone.core.utils.
validate_hex_color
(value)¶ Validates that the given value is a hex color string or css name.
- Parameters
value – a value
- Raises
ValueError – if
value
is not a hex color string