fiftyone.core.utils¶

Core utilities.

Copyright 2017-2025, Voxel51, Inc.
voxel51.com

Functions:

`extract_kwargs_for_class`(cls, kwargs)	Extracts keyword arguments for the given class’s constructor from the given kwargs.
`extract_kwargs_for_function`(fcn, kwargs)	Extracts keyword arguments for the given function from the given kwargs.
`pprint`(obj[, stream, indent, width, depth])	Pretty-prints the Python object.
`pformat`(obj[, indent, width, depth])	Returns a pretty string representation of the Python object.
`split_frame_fields`(fields)	Splits the given fields into sample and frame fields.
`stream_objects`(objects)	Streams the iterable of objects to stdout via `less`.
`indent_lines`(s[, indent, skip])	Indents the lines in the given string.
`justify_headings`(elements[, width])	Justifies the headings in a list of `(heading, content)` string tuples by appending whitespace as necessary to each `heading`.
`available_patterns`()	Returns the available patterns that can be used by `fill_patterns()`.
`fill_patterns`(string)	Fills the patterns in in the given string.
`find_files`(root_dir, patt[, max_depth])	Finds all files in the given root directory whose filename matches the given glob pattern(s).
`install_package`(requirement_str[, …])	Installs the given package via `pip`.
`ensure_package`(requirement_str[, …])	Verifies that the given package is installed.
`load_requirements`(requirements_path)	Loads the package requirements from a `requirements.txt` file on disk.
`install_requirements`(requirements_path[, …])	Installs the package requirements from a `requirements.txt` file on disk.
`ensure_requirements`(requirements_path[, …])	Verifies that the package requirements from a `requirements.txt` file on disk are installed.
`ensure_import`(requirement_str[, …])	Verifies that the given requirement is installed and importable.
`ensure_tf`([eager, error_level, error_msg])	Verifies that `tensorflow` is installed and importable.
`ensure_tfds`([error_level, error_msg])	Verifies that `tensorflow_datasets` is installed and importable.
`ensure_torch`([error_level, error_msg])	Verifies that `torch` and `torchvision` are installed and importable.
`handle_error`(error, error_level[, base_error])	Handles the error at the specified error level.
`lazy_import`(module_name[, callback])	Returns a proxy module object that will lazily import the given module the first time it is used.
`load_xml_as_json_dict`(xml_path)	Loads the XML file as a JSON dictionary.
`parse_serializable`(obj, cls)	Parses the given object as an instance of the given `eta.core.serial.Serializable` class.
`set_resource_limit`(limit[, soft, hard, …])	Uses the `resource` package to change a resource limit for the current process.
`report_progress`(progress[, n, dt])	Wraps the provided progress function such that it will only be called at the specified increments or time intervals.
`default_calc_size`(obj)
`get_default_batcher`(iterable[, batcher, …])	Returns a `Batcher` over `iterable` using defaults from your FiftyOne config.
`parse_batching_strategy`([batch_size, …])	Parses the given batching strategy configuration, applying any default config settings as necessary.
`recommend_batch_size_for_value`(value[, …])	Computes a recommended batch size for the given value type such that a request involving a list of values of this size will be less than `alpha * fo.config.batcher_target_size_bytes` bytes.
`disable_progress_bars`()	Context manager that temporarily disables all progress bars.
`safe_relpath`(path[, start, default])	A safe version of `os.path.relpath` that returns a configurable default value if the given path if it does not lie within the given relative start.
`get_module_name`(path[, start])	Gets the Python module name for the given file or directory path.
`compute_filehash`(filepath[, method, chunk_size])	Computes the hash of the given file.
`serialize_numpy_array`(array[, ascii])	Serializes a numpy array.
`deserialize_numpy_array`(numpy_bytes[, ascii])	Loads a serialized numpy array generated by `serialize_numpy_array()`.
`iter_batches`(iterable, batch_size)	Iterates over the given iterable in batches.
`iter_slices`(sliceable, batch_size)	Iterates over batches of the given object via slicing.
`call_on_exit`(callback)	Registers the given callback function so that it will be called when the process exits for (almost) any reason
`is_arm_mac`()	Determines whether the system is an ARM-based Mac (Apple Silicon).
`is_32_bit`()	Determines whether the system is 32-bit.
`is_container`()	Determines if we’re currently running as a container.
`get_multiprocessing_context`()	Returns the preferred `multiprocessing` context for the current OS.
`recommend_thread_pool_workers`([num_workers])	Recommends a number of workers for a thread pool.
`recommend_process_pool_workers`([num_workers])	Recommends a number of workers for a process pool.
`run_sync_task`(func, *args)	Run a synchronous function as an async background task.
`datetime_to_timestamp`(dt)	Converts a datetime.date or datetime.datetime to milliseconds since epoch.
`timestamp_to_datetime`(ts)	Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime.
`timedelta_to_ms`(td)	Converts a datetime.timedelta to milliseconds.
`to_slug`(name)	Returns the URL-friendly slug for the given string.
`validate_color`(value)	Validates that the given value is a valid css color name.
`validate_hex_color`(value)	Validates that the given value is a hex color string or css name.

Classes:

LoggingLevel(level[, logger])

Context manager that allows for a temporary change to the level of a logging.Logger.

LazyModule(module_name[, callback])

Proxy module that lazily imports the underlying module the first time it is actually used.

ResourceLimit(limit[, soft, hard, …])

Context manager that allows for a temporary change to a resource limit exposed by the resource package.

ProgressBar([total, progress, quiet])

class eta.core.utils.ProgressBar(total=None, show_percentage=True, show_iteration=True, show_elapsed_time=True, show_remaining_time=True, show_iter_rate=True, iters_str='it', start_msg=None, complete_msg=None, use_bits=False, use_bytes=False, max_width=None, num_decimals=1, max_fps=10, quiet=None)¶

Class for printing a self-updating progress bar to stdout that tracks the progress of an iterative process towards completion.

The simplest use of ProgressBar is to create an instance and then call it (__call__) with an iterable argument, which will pass through elements of the iterable when iterating over it, updating the progress bar each time an element is emitted.

Alternatively, ProgressBar`s can track the progress of a task towards a total iteration count, where the current iteration is updated manually by calling either `update(), which will increment the iteration count by 1, or via set_iteration(), which lets you manually specify the new iteration status. By default, both methods will automatically redraw the bar. If manual control over drawing is required, you can pass draw=False to either method and then manually call draw() as desired.

It is highly recommended that you always invoke ProgressBar`s using their context manager interface via the `with keyword, which will automatically handle calling start() and close() to appropriately initialize and finalize the task. Among other things, this ensures that stdout redirection is appropriately ended when an exception is encountered.

When start() is called on the a ProgressBar, an internal timer is started to track the elapsed time of the task. In addition, stdout is automatically cached between calls to draw() and flushed each time draw() is called, which means that you can freely mix print statements into your task without interfering with the progress bar. When you are done tracking the task, call the close() method to finalize the progress bar. Both of these actions are automatically taken when the bar’s context manager interface is invoked or when it is called with an iterable.

If you want want to full manual control over the progress bar, call start() to start the task, call pause() before any print statements, and call close() when the task is finalized.

ProgressBar can optionally be configured to print any of the following statistics about the task:

the elapsed time since the task was started
the estimated time remaining until the task completes
the current iteration rate of the task, in iterations per second
customized status messages passed via the suffix argument

Example (wrapping an iterator):

import time
import eta.core.utils as etau

with etau.ProgressBar() as pb:
    for _ in pb(range(100)):
        time.sleep(0.05)

Example (with print statements interleaved):

import time
import eta.core.utils as etau

with etau.ProgressBar(100) as pb:
    while not pb.complete:
        if pb.iteration in {25, 50, 75}:
            print("Progress = %.2f" % pb.progress)

        time.sleep(0.05)
        pb.update()

Example (tracking a bit total):

import random
import time
import eta.core.utils as etau

with etau.ProgressBar(1024 ** 2, use_bits=True) as pb:
    while not pb.complete:
        pb.update(random.randint(1, 10 * 1024))
        time.sleep(0.05)

Attributes:

`is_iterable`	Whether the progress bar is tracking an iterable.
`is_running`	Whether the progress bar is running, i.e., it is between start() and close() calls.
`is_finalized`	Whether the progress bar is finalized, i.e., close() has been called.
`is_capturing_stdout`	Whether stdout is being captured between calls to draw().
`has_dynamic_width`	Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
`has_total`	Whether the progress bar has a total iteration count.
`total`	The total iterations, or None if not available.
`iteration`	The current iteration.
`progress`	The current progress, in [0, 1], or None if the progress bar has no total.
`complete`	Whether the task is 100%% complete, or None if the progress bar has no total.
`elapsed_time`	The elapsed time since the task was started, or None if this progress bar is not timing.
`time_remaining`	The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
`iter_rate`	The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
`quiet`	Whether the progress bar is in quiet mode (no printing to stdout).

Methods:

`start`()	Starts the progress bar.
`close`(*args)	Closes the progress bar.
`update`([count, suffix, draw])	Increments the current iteration count by the given value (default = 1).
`set_iteration`(iteration[, suffix, draw])	Sets the current iteration.
`pause`()	Pauses the progress bar so that other information can be printed.
`draw`([force])	Draws the progress bar at its current progress.

property is_iterable¶: Whether the progress bar is tracking an iterable.

property is_running¶: Whether the progress bar is running, i.e., it is between start() and close() calls.

property is_finalized¶: Whether the progress bar is finalized, i.e., close() has been called.

property is_capturing_stdout¶: Whether stdout is being captured between calls to draw().

property has_dynamic_width¶: Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property has_total¶: Whether the progress bar has a total iteration count.

property total¶: The total iterations, or None if not available.

property iteration¶: The current iteration.

property progress¶: The current progress, in [0, 1], or None if the progress bar has no total.

property complete¶: Whether the task is 100%% complete, or None if the progress bar has no total.

property elapsed_time¶: The elapsed time since the task was started, or None if this progress bar is not timing.

property time_remaining¶: The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property iter_rate¶: The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property quiet¶: Whether the progress bar is in quiet mode (no printing to stdout).

start()¶: Starts the progress bar.

close(*args)¶: Closes the progress bar.

update(count=1, suffix=None, draw=True)¶

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters

count – the iteration increment. The default is 1
suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged
draw – whether to call draw() at the end of this method. By default, this is True

set_iteration(iteration, suffix=None, draw=True)¶

Sets the current iteration.

Parameters

iteration – the new iteration
suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”
draw – whether to call draw() at the end of this method. By default, this is True

pause()¶

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

draw(force=False)¶

Draws the progress bar at its current progress.

Parameters: force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

Batcher(iterable[, transform_fn, …])

Base class for iterating over the elements of an iterable in batches.

BaseChunkyBatcher(iterable[, transform_fn, …])

Base class for iterating over the elements of an iterable in chunks.

BaseDynamicBatcher(iterable, target_measurement)

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired target measurement.

LatencyDynamicBatcher(iterable[, …])

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.

DynamicBatcher

alias of fiftyone.core.utils.LatencyDynamicBatcher

ContentSizeDynamicBatcher(iterable[, …])

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

StaticBatcher(iterable, batch_size[, …])

Class for iterating over the elements of an iterable with a static batch size.

ContentSizeBatcher(iterable[, target_size, …])

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

UniqueFilenameMaker([output_dir, rel_dir, …])

A class that generates unique output paths in a directory.

MultiProcessUniqueFilenameMaker(ppid[, …])

A class that generates unique output paths in a directory.

MonkeyPatchFunction(module_or_fcn, monkey_fcn)

Context manager that temporarily monkey patches the given function.

SetAttributes(obj, **kwargs)

Context manager that temporarily sets the attributes of a class to new values.

SuppressLogging([level])

Context manager that temporarily disables system-wide logging.

add_sys_path(path[, index])

Context manager that temporarily inserts a path to sys.path.

ResponseStream(response[, chunk_size])

Wrapper around a requests.Response that provides a file-like object interface with read(), seek(), and tell() methods.

fiftyone.core.utils.extract_kwargs_for_class(cls, kwargs)¶

Extracts keyword arguments for the given class’s constructor from the given kwargs.

Parameters

cls – a class
kwargs – a dictionary of keyword arguments

Returns

a tuple of

class_kwargs: a dictionary of keyword arguments for cls
other_kwargs: a dictionary containing the remaining kwargs

fiftyone.core.utils.extract_kwargs_for_function(fcn, kwargs)¶

Extracts keyword arguments for the given function from the given kwargs.

Parameters

fcn – a function
kwargs – a dictionary of keyword arguments

Returns

a tuple of

fcn_kwargs: a dictionary of keyword arguments for fcn
other_kwargs: a dictionary containing the remaining kwargs

fiftyone.core.utils.pprint(obj, stream=None, indent=4, width=80, depth=None)¶

Pretty-prints the Python object.

Parameters

obj – the Python object
stream (None) – the stream to write to. The default is sys.stdout
indent (4) – the number of spaces to use when indenting
width (80) – the max width of each line in the pretty representation
depth (None) – the maximum depth at which to pretty render nested dicts

fiftyone.core.utils.pformat(obj, indent=4, width=80, depth=None)¶

Returns a pretty string representation of the Python object.

Parameters

obj – the Python object
indent (4) – the number of spaces to use when indenting
width (80) – the max width of each line in the pretty representation
depth (None) – the maximum depth at which to pretty render nested dicts

Returns

the pretty-formatted string

fiftyone.core.utils.split_frame_fields(fields)¶

Splits the given fields into sample and frame fields.

Frame fields are those prefixed by "frames.", and this prefix is removed from the returned frame fields.

Parameters

fields – a field, iterable of fields, or dict mapping field names to new field names

Returns

a tuple of

sample_fields: a list or dict of sample fields
frame_fields: a list or dict of frame fields

fiftyone.core.utils.stream_objects(objects)¶

Streams the iterable of objects to stdout via less.

The output can be interactively traversed via scrolling and can be terminated via keyboard interrupt.

Parameters: objects – an iterable of objects that can be printed via str(obj)

fiftyone.core.utils.indent_lines(s, indent=4, skip=0)¶

Indents the lines in the given string.

Parameters

s – the string
indent (4) – the number of spaces to indent
skip (0) – the number of lines to skip before indenting

Returns

the indented string

fiftyone.core.utils.justify_headings(elements, width=None)¶

Justifies the headings in a list of (heading, content) string tuples by appending whitespace as necessary to each heading.

Parameters

elements – a list of (heading, content) tuples
width (None) – an optional justification width. By default, the maximum heading length is used

Returns

a list of justified (heading, content) tuples

fiftyone.core.utils.available_patterns()¶

Returns the available patterns that can be used by fill_patterns().

Returns: a dict mapping patterns to their replacements

fiftyone.core.utils.fill_patterns(string)¶

Fills the patterns in in the given string.

Use available_patterns() to see the available patterns that can be used.

Parameters: string – a string
Returns: a copy of string with any patterns replaced

fiftyone.core.utils.find_files(root_dir, patt, max_depth=1)¶

Finds all files in the given root directory whose filename matches the given glob pattern(s).

Both root_dir and patt may contain glob patterns.

Exammples:

import fiftyone.core.utils as fou

# Find .txt files in `/tmp`
fou.find_files("/tmp", "*.txt")

# Find .txt files in subdirectories of `/tmp` that begin with `foo-`
fou.find_files("/tmp/foo-*", "*.txt")

# Find .txt files in `/tmp` or its subdirectories
fou.find_files("/tmp", "*.txt", max_depth=2)

Parameters

root_dir – the root directory
patt – a glob pattern or list of patterns
max_depth (1) – a maximum depth to search. 1 means root_dir only, 2 means root_dir and its immediate subdirectories, etc

Returns

a list of matching paths

fiftyone.core.utils.install_package(requirement_str, error_level=None, error_msg=None)¶

Installs the given package via pip.

Installation is performed via:

python -m pip install <requirement_str>

Parameters

requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”
error_level (None) –
the error level to use, defined as:
- 0: raise error if the install fails
- 1: log warning if the install fails
- 2: ignore install fails
error_msg (None) – an optional custom error message to use

fiftyone.core.utils.ensure_package(requirement_str, error_level=None, error_msg=None, log_success=False)¶

Verifies that the given package is installed.

This function uses importlib.metadata to locate the package by its pip name and does not actually import the module.

Therefore, unlike ensure_import(), requirement_str should refer to the package name (e.g., “tensorflow-gpu”), not the module name (e.g., “tensorflow”).

Parameters

requirement_str – a PEP 440 compliant package requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed
error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
error_msg (None) – an optional custom error message to use
log_success (False) – whether to generate a log message if the requirement is satisfied

Returns

True/False whether the requirement is satisfied

fiftyone.core.utils.load_requirements(requirements_path)¶

Loads the package requirements from a requirements.txt file on disk.

Comments and extra whitespace are automatically stripped.

Parameters: requirements_path – the path to a requirements file
Returns: a list of requirement strings

fiftyone.core.utils.install_requirements(requirements_path, error_level=None)¶

Installs the package requirements from a requirements.txt file on disk.

Parameters

requirements_path – the path to a requirements file
error_level (None) –
the error level to use, defined as:
- 0: raise error if the install fails
- 1: log warning if the install fails
- 2: ignore install fails
By default, fiftyone.config.requirement_error_level is used

fiftyone.core.utils.ensure_requirements(requirements_path, error_level=None, log_success=False)¶

Verifies that the package requirements from a requirements.txt file on disk are installed.

Parameters

requirements_path – the path to a requirements file
error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
log_success (False) – whether to generate a log message if a requirement is satisfied

fiftyone.core.utils.ensure_import(requirement_str, error_level=None, error_msg=None, log_success=False)¶

Verifies that the given requirement is installed and importable.

This function imports the specified module and optionally enforces any version requirements included in requirement_str.

Therefore, unlike ensure_package(), requirement_str should refer to the module name (e.g., “tensorflow”), not the package name (e.g., “tensorflow-gpu”).

Parameters

requirement_str – a PEP 440-like module requirement, like “tensorflow”, “tensorflow<2”, “tensorflow==2.3.0”, or “tensorflow>=1.13,<1.15”. This can also be an iterable of multiple requirements, all of which must be installed, or this can be a single “|”-delimited string specifying multiple requirements, at least one of which must be installed
error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
error_msg (None) – an optional custom error message to use
log_success (False) – whether to generate a log message if the requirement is satisfied

Returns

True/False whether the requirement is satisfied

fiftyone.core.utils.ensure_tf(eager=False, error_level=None, error_msg=None)¶

Verifies that tensorflow is installed and importable.

Parameters

eager (False) – whether to require that TF is executing eagerly. If True and TF is not currently executing eagerly, this method will attempt to enable it
error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisfied

fiftyone.core.utils.ensure_tfds(error_level=None, error_msg=None)¶

Verifies that tensorflow_datasets is installed and importable.

Parameters

error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisfied

fiftyone.core.utils.ensure_torch(error_level=None, error_msg=None)¶

Verifies that torch and torchvision are installed and importable.

Parameters

error_level (None) –
the error level to use, defined as:
- 0: raise error if requirement is not satisfied
- 1: log warning if requirement is not satisfied
- 2: ignore unsatisifed requirements
By default, fiftyone.config.requirement_error_level is used
error_msg (None) – an optional custom error message to print

Returns

True/False whether the requirement is satisfied

fiftyone.core.utils.handle_error(error, error_level, base_error=None)¶

Handles the error at the specified error level.

Parameters

error – an Exception instance
error_level – the error level to use, defined as:
0 (-) – raise the error
1 (-) – log the error as a warning
2 (-) – ignore the error
base_error – (optional) a base Exception from which to raise error

class fiftyone.core.utils.LoggingLevel(level, logger=None)¶

Bases: object

Context manager that allows for a temporary change to the level of a logging.Logger.

Example:

   import logging
   import fiftyone.core.utils as fou

   with fou.LoggingLevel(logging.CRITICAL):
       # do things with all logging at CRITICAL

   with fou.LoggingLevel(logging.ERROR, logger="fiftyone"):
       # do things with FiftyOne logging at ERROR

Args:
   level: the logging level to use, e.g., ``logging.ERROR``
   logger (None): a ``logging.Logger`` or the name of a logger. By
       default, the root logger is used

fiftyone.core.utils.lazy_import(module_name, callback=None)¶

Returns a proxy module object that will lazily import the given module the first time it is used.

Example usage:

# Lazy version of `import tensorflow as tf`
tf = lazy_import("tensorflow")

# Other commands

# Now the module is loaded
tf.__version__

Parameters

module_name – the fully-qualified module name to import
callback (None) – a callback function to call before importing the module

Returns

a proxy module object that will be lazily imported when first used

class fiftyone.core.utils.LazyModule(module_name, callback=None)¶

Bases: module

Proxy module that lazily imports the underlying module the first time it is actually used.

Parameters

module_name – the fully-qualified module name to import
callback (None) – a callback function to call before importing the module

fiftyone.core.utils.load_xml_as_json_dict(xml_path)¶

Loads the XML file as a JSON dictionary.

Parameters: xml_path – the path to the XML file
Returns: a JSON dict

fiftyone.core.utils.parse_serializable(obj, cls)¶

Parses the given object as an instance of the given eta.core.serial.Serializable class.

Parameters

obj – an instance of cls, or a serialized string or dictionary representation of one
cls – a eta.core.serial.Serializable class

Returns

an instance of cls

fiftyone.core.utils.set_resource_limit(limit, soft=None, hard=None, warn_on_failure=False)¶

Uses the resource package to change a resource limit for the current process.

If the resource package cannot be imported, this command does nothing.

Parameters

limit – the name of the resource to limit. Must be the name of a constant in the resource module starting with RLIMIT. See the documentation of the resource module for supported values
soft (None) – a new soft limit to apply, which cannot exceed the hard limit. If omitted, the current soft limit is maintained
hard (None) – a new hard limit to apply. If omitted, the current hard limit is maintained
warn_on_failure (False) – whether to issue a warning rather than an error if the resource limit change is not successful

class fiftyone.core.utils.ResourceLimit(limit, soft=None, hard=None, warn_on_failure=False)¶

Bases: object

Context manager that allows for a temporary change to a resource limit exposed by the resource package.

Example:

   import resource
   import fiftyone.core.utils as fou

   with fou.ResourceLimit(resource.RLIMIT_NOFILE, soft=4096):
       # temporarily do things with up to 4096 open files

Args:
   limit: the name of the resource to limit. Must be the name of a
       constant in the ``resource`` module starting with ``RLIMIT``. See
       the documentation of the ``resource`` module for supported values
   soft (None): a new soft limit to apply, which cannot exceed the hard
       limit. If omitted, the current soft limit is maintained
   hard (None): a new hard limit to apply. If omitted, the current hard
       limit is maintained
   warn_on_failure (False): whether to issue a warning rather than an
       error if the resource limit change is not successful

class fiftyone.core.utils.ProgressBar(total=None, progress=None, quiet=None, **kwargs)¶

Bases: eta.core.utils.ProgressBar

class ProgressBar(total=None, show_percentage=True, show_iteration=True, show_elapsed_time=True, show_remaining_time=True, show_iter_rate=True, iters_str='it', start_msg=None, complete_msg=None, use_bits=False, use_bytes=False, max_width=None, num_decimals=1, max_fps=10, quiet=None)¶

Class for printing a self-updating progress bar to stdout that tracks the progress of an iterative process towards completion.

The simplest use of ProgressBar is to create an instance and then call it (__call__) with an iterable argument, which will pass through elements of the iterable when iterating over it, updating the progress bar each time an element is emitted.

Alternatively, ProgressBar`s can track the progress of a task towards a total iteration count, where the current iteration is updated manually by calling either `update(), which will increment the iteration count by 1, or via set_iteration(), which lets you manually specify the new iteration status. By default, both methods will automatically redraw the bar. If manual control over drawing is required, you can pass draw=False to either method and then manually call draw() as desired.

It is highly recommended that you always invoke ProgressBar`s using their context manager interface via the `with keyword, which will automatically handle calling start() and close() to appropriately initialize and finalize the task. Among other things, this ensures that stdout redirection is appropriately ended when an exception is encountered.

When start() is called on the a ProgressBar, an internal timer is started to track the elapsed time of the task. In addition, stdout is automatically cached between calls to draw() and flushed each time draw() is called, which means that you can freely mix print statements into your task without interfering with the progress bar. When you are done tracking the task, call the close() method to finalize the progress bar. Both of these actions are automatically taken when the bar’s context manager interface is invoked or when it is called with an iterable.

If you want want to full manual control over the progress bar, call start() to start the task, call pause() before any print statements, and call close() when the task is finalized.

ProgressBar can optionally be configured to print any of the following statistics about the task:

the elapsed time since the task was started
the estimated time remaining until the task completes
the current iteration rate of the task, in iterations per second
customized status messages passed via the suffix argument

Example (wrapping an iterator):

import time
import eta.core.utils as etau

with etau.ProgressBar() as pb:
    for _ in pb(range(100)):
        time.sleep(0.05)

Example (with print statements interleaved):

import time
import eta.core.utils as etau

with etau.ProgressBar(100) as pb:
    while not pb.complete:
        if pb.iteration in {25, 50, 75}:
            print("Progress = %.2f" % pb.progress)

        time.sleep(0.05)
        pb.update()

Example (tracking a bit total):

import random
import time
import eta.core.utils as etau

with etau.ProgressBar(1024 ** 2, use_bits=True) as pb:
    while not pb.complete:
        pb.update(random.randint(1, 10 * 1024))
        time.sleep(0.05)

Attributes:

`is_iterable`	Whether the progress bar is tracking an iterable.
`is_running`	Whether the progress bar is running, i.e., it is between start() and close() calls.
`is_finalized`	Whether the progress bar is finalized, i.e., close() has been called.
`is_capturing_stdout`	Whether stdout is being captured between calls to draw().
`has_dynamic_width`	Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
`has_total`	Whether the progress bar has a total iteration count.
`total`	The total iterations, or None if not available.
`iteration`	The current iteration.
`progress`	The current progress, in [0, 1], or None if the progress bar has no total.
`complete`	Whether the task is 100%% complete, or None if the progress bar has no total.
`elapsed_time`	The elapsed time since the task was started, or None if this progress bar is not timing.
`time_remaining`	The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
`iter_rate`	The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
`quiet`	Whether the progress bar is in quiet mode (no printing to stdout).

Methods:

`start`()	Starts the progress bar.
`close`(*args)	Closes the progress bar.
`update`([count, suffix, draw])	Increments the current iteration count by the given value (default = 1).
`set_iteration`(iteration[, suffix, draw])	Sets the current iteration.
`pause`()	Pauses the progress bar so that other information can be printed.
`draw`([force])	Draws the progress bar at its current progress.

property ProgressBar.is_iterable¶: Whether the progress bar is tracking an iterable.

property ProgressBar.is_running¶: Whether the progress bar is running, i.e., it is between start() and close() calls.

property ProgressBar.is_finalized¶: Whether the progress bar is finalized, i.e., close() has been called.

property ProgressBar.is_capturing_stdout¶: Whether stdout is being captured between calls to draw().

property ProgressBar.has_dynamic_width¶: Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property ProgressBar.has_total¶: Whether the progress bar has a total iteration count.

property ProgressBar.total¶: The total iterations, or None if not available.

property ProgressBar.iteration¶: The current iteration.

property ProgressBar.progress¶: The current progress, in [0, 1], or None if the progress bar has no total.

property ProgressBar.complete¶: Whether the task is 100%% complete, or None if the progress bar has no total.

property ProgressBar.elapsed_time¶: The elapsed time since the task was started, or None if this progress bar is not timing.

property ProgressBar.time_remaining¶: The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property ProgressBar.iter_rate¶: The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property ProgressBar.quiet¶: Whether the progress bar is in quiet mode (no printing to stdout).

ProgressBar.start()¶: Starts the progress bar.

ProgressBar.close(*args)¶: Closes the progress bar.

ProgressBar.update(count=1, suffix=None, draw=True)¶

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters

count – the iteration increment. The default is 1
suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged
draw – whether to call draw() at the end of this method. By default, this is True

ProgressBar.set_iteration(iteration, suffix=None, draw=True)¶

Sets the current iteration.

Parameters

iteration – the new iteration
suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”
draw – whether to call draw() at the end of this method. By default, this is True

ProgressBar.pause()¶

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

ProgressBar.draw(force=False)¶

Draws the progress bar at its current progress.

Parameters: force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

Methods:

`set_iteration`(args, *kwargs)	Sets the current iteration.
`close`(*args)	Closes the progress bar.
`draw`([force])	Draws the progress bar at its current progress.
`pause`()	Pauses the progress bar so that other information can be printed.
`start`()	Starts the progress bar.
`update`([count, suffix, draw])	Increments the current iteration count by the given value (default = 1).

Attributes:

`complete`	Whether the task is 100%% complete, or None if the progress bar has no total.
`elapsed_time`	The elapsed time since the task was started, or None if this progress bar is not timing.
`has_dynamic_width`	Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.
`has_total`	Whether the progress bar has a total iteration count.
`is_capturing_stdout`	Whether stdout is being captured between calls to draw().
`is_finalized`	Whether the progress bar is finalized, i.e., close() has been called.
`is_iterable`	Whether the progress bar is tracking an iterable.
`is_running`	Whether the progress bar is running, i.e., it is between start() and close() calls.
`iter_rate`	The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.
`iteration`	The current iteration.
`progress`	The current progress, in [0, 1], or None if the progress bar has no total.
`quiet`	Whether the progress bar is in quiet mode (no printing to stdout).
`time_remaining`	The estimated time remaining for the task, or None if the progress bar has no total or is not timing.
`total`	The total iterations, or None if not available.

set_iteration(*args, **kwargs)¶

Sets the current iteration.

Parameters

iteration – the new iteration
suffix – an optional suffix string to append to the progress bar. Once a custom suffix is provided, it will remain unchanged until you update it or remove it by passing suffix == “”
draw – whether to call draw() at the end of this method. By default, this is True

close(*args)¶: Closes the progress bar.

property complete¶: Whether the task is 100%% complete, or None if the progress bar has no total.

draw(force=False)¶

Draws the progress bar at its current progress.

Parameters: force – whether to force the progress bar to be drawn. By default, it is only redrawn a maximum of self.max_fps times per second

property elapsed_time¶: The elapsed time since the task was started, or None if this progress bar is not timing.

property has_dynamic_width¶: Whether the progress bar’s width is adjusted dynamically based on the width of the terminal window.

property has_total¶: Whether the progress bar has a total iteration count.

property is_capturing_stdout¶: Whether stdout is being captured between calls to draw().

property is_finalized¶: Whether the progress bar is finalized, i.e., close() has been called.

property is_iterable¶: Whether the progress bar is tracking an iterable.

property is_running¶: Whether the progress bar is running, i.e., it is between start() and close() calls.

property iter_rate¶: The current iteration rate, in iterations per second, of the task, or None if the progress bar is not running.

property iteration¶: The current iteration.

pause()¶

Pauses the progress bar so that other information can be printed.

This function overwrites the current progress bar with whitespace and appends a carriage return so that any other information that is printed will overwrite the current progress bar.

property progress¶: The current progress, in [0, 1], or None if the progress bar has no total.

property quiet¶: Whether the progress bar is in quiet mode (no printing to stdout).

start()¶: Starts the progress bar.

property time_remaining¶: The estimated time remaining for the task, or None if the progress bar has no total or is not timing.

property total¶: The total iterations, or None if not available.

update(count=1, suffix=None, draw=True)¶

Increments the current iteration count by the given value (default = 1).

This method is shorthand for self.set_iteration(self.iteration + count).

Parameters

count – the iteration increment. The default is 1
suffix – an optional suffix string to append to the progress bar. By default, the suffix is unchanged
draw – whether to call draw() at the end of this method. By default, this is True

fiftyone.core.utils.report_progress(progress, n=None, dt=None)¶

Wraps the provided progress function such that it will only be called at the specified increments or time intervals.

Example usage:

import fiftyone as fo
import fiftyone.zoo as foz

def print_progress(pb):
    if pb.complete:
        print("COMPLETE")
    else:
        print("PROGRESS: %0.3f" % pb.progress)

dataset = foz.load_zoo_dataset("cifar10", split="test")

# Disable builtin progress bars
fo.config.show_progress_bars = False

# Print progress at 10 equally-spaced increments
progress = fo.report_progress(print_progress, n=10)
dataset.compute_metadata(progress=progress)

# Print progress every 0.5 seconds
progress = fo.report_progress(print_progress, dt=0.5)
dataset.compute_metadata(progress=progress, overwrite=True)

Parameters

progress – a function that accepts a ProgressBar as input
n (None) – a number of equally-spaced increments to invoke progress
dt (None) – a number of seconds between progress calls

Returns

a function that accepts a ProgressBar as input

class fiftyone.core.utils.Batcher(iterable, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: abc.ABC

Base class for iterating over the elements of an iterable in batches.

Parameters

iterable – an iterable to batch over
transform_fn (None) – a transform function to apply to each item
return_views (False) – whether to return each batch as a fiftyone.core.view.DatasetView. Only applicable when the iterable is a fiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

class fiftyone.core.utils.BaseChunkyBatcher(iterable, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: fiftyone.core.utils.Batcher

Base class for iterating over the elements of an iterable in chunks.

Batch sizes are determined per chunk using _compute_batch_size.

class fiftyone.core.utils.BaseDynamicBatcher(iterable, target_measurement, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: fiftyone.core.utils.BaseChunkyBatcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired target measurement.

The batch sizes emitted when iterating over this object are dynamically scaled such that the measurement between next() calls is as close as possible to a specified target.

Concrete base classes define the target measurement and method of calculation.

class fiftyone.core.utils.LatencyDynamicBatcher(iterable, target_latency=0.2, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: fiftyone.core.utils.BaseDynamicBatcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired latency.

The batch sizes emitted when iterating over this object are dynamically scaled such that the latency between next() calls is as close as possible to a specified target latency.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.LatencyDynamicBatcher(
    elements, target_latency=0.1, max_batch_beta=2.0
)

for batch in batcher:
    print("batch size: %d" % len(batch))

batcher = fou.LatencyDynamicBatcher(
    elements,
    target_latency=0.1,
    max_batch_beta=2.0,
    progress=True,
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))

Parameters

iterable – an iterable to batch over. If None, the result of next() will be a batch size instead of a batch, and is an infinite iterator
target_latency (0.2) – the target latency between next() calls, in seconds
init_batch_size (1) – the initial batch size to use
min_batch_size (1) – the minimum allowed batch size
max_batch_size (None) – an optional maximum allowed batch size
max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes
transform_fn (None) – a transform function to apply to each item
return_views (False) – whether to return each batch as a fiftyone.core.view.DatasetView. Only applicable when the iterable is a fiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

fiftyone.core.utils.DynamicBatcher¶: alias of fiftyone.core.utils.LatencyDynamicBatcher

class fiftyone.core.utils.ContentSizeDynamicBatcher(iterable, target_size=1048576, init_batch_size=1, min_batch_size=1, max_batch_size=None, max_batch_beta=None, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: fiftyone.core.utils.BaseDynamicBatcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.

This batcher requires that backpressure feedback be provided, either by providing a BSON-able batch from which the content size can be computed, or by manually providing the content size.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.ContentSizeDynamicBatcher(
    elements, target_size=2**20, max_batch_beta=2.0
)

# Raises ValueError after first batch, we forgot to apply backpressure
for batch in batcher:
    print("batch size: %d" % len(batch))

# Now it works
for batch in batcher:
    print("batch size: %d" % len(batch))
    batcher.apply_backpressure(batch)

batcher = fou.ContentSizeDynamicBatcher(
    elements,
    target_size=2**20,
    max_batch_beta=2.0,
    progress=True
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))
        batcher.apply_backpressure(batch)

Parameters

iterable – an iterable to batch over. If None, the result of next() will be a batch size instead of a batch, and is an infinite iterator
target_size (1048576) – the target batch bson content size, in bytes
init_batch_size (1) – the initial batch size to use
min_batch_size (1) – the minimum allowed batch size
max_batch_size (None) – an optional maximum allowed batch size
max_batch_beta (None) – an optional lower/upper bound on the ratio between successive batch sizes
transform_fn (None) – a transform function to apply to each item
return_views (False) – whether to return each batch as a fiftyone.core.view.DatasetView. Only applicable when the iterable is a fiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

Methods:

apply_backpressure(batch_or_size)

apply_backpressure(batch_or_size)¶

class fiftyone.core.utils.StaticBatcher(iterable, batch_size, transform_fn=None, return_views=False, progress=False, total=None)¶

Bases: fiftyone.core.utils.BaseChunkyBatcher

Class for iterating over the elements of an iterable with a static batch size.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.StaticBatcher(elements, batch_size=10000)

for batch in batcher:
    print("batch size: %d" % len(batch))

batcher = fou.StaticBatcher(elements, batch_size=10000, progress=True)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))

Parameters

iterable – an iterable to batch over. If None, the result of next() will be a batch size instead of a batch, and is an infinite iterator
batch_size – size of batches to generate
transform_fn (None) – a transform function to apply to each item
return_views (False) – whether to return each batch as a fiftyone.core.view.DatasetView. Only applicable when the iterable is a fiftyone.core.collections.SampleCollection
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

class fiftyone.core.utils.ContentSizeBatcher(iterable, target_size=1048576, max_batch_size=None, transform_fn=None, size_calc_fn=None, progress=False, total=None)¶

Bases: fiftyone.core.utils.Batcher

Class for iterating over the elements of an iterable with a dynamic batch size to achieve a desired content size.

The batch sizes emitted when iterating over this object are dynamically scaled such that the total content size of the batch is as close as possible to a specified target size.

This batcher does not require backpressure feedback because it calculates the total size of the iterable object before batching.

This class is often used in conjunction with a ProgressBar to keep the user appraised on the status of a long-running task.

Example usage:

import fiftyone.core.utils as fou

elements = range(int(1e7))

batcher = fou.ContentSizeBatcher(
    elements,
    target_size=2**20,
    progress=True
)

with batcher:
    for batch in batcher:
        print("batch size: %d" % len(batch))

Parameters

iterable – an iterable to batch over. If None, the result of next() will be a batch size instead of a batch, and is an infinite iterator
target_size (1048576) – the target batch BSON content size, in bytes
transform_fn (None) – a transform function to apply to each item
size_calc_fn (None) – a function that calculates the size of each item. This is applied after transform_fn if both are provided
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

Methods:

set_encoding_ratio(encoded_batch_size)

set_encoding_ratio(encoded_batch_size)¶

fiftyone.core.utils.default_calc_size(obj)¶

fiftyone.core.utils.get_default_batcher(iterable, batcher=None, transform_fn=None, size_calc_fn=None, progress=False, total=None)¶

Returns a Batcher over iterable using defaults from your FiftyOne config.

If no batcher is provided, this method uses fiftyone.config.default_batcher to determine the implementation to use and related configuration values as needed for each.

Parameters

iterable – an iterable to batch over. If None, the result of next() will be a batch size instead of a batch, and is an infinite iterator
batcher (None) – a specific Batcher subclass to use, or False to disable batching
transform_fn (None) – a transform function to apply to each item
size_calc_fn (None) – a function that calculates the size of each item. This is applied after transform_fn if both are provided. Only applicable when fiftyone.config.default_batcher="size"
progress (False) – whether to render a progress bar tracking the consumption of the batches (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
total (None) – the length of iterable. Only applicable when progress=True. If not provided, it is computed via len(iterable), if possible

Returns

a Batcher instance

fiftyone.core.utils.parse_batching_strategy(batch_size=None, batching_strategy=None)¶

Parses the given batching strategy configuration, applying any default config settings as necessary.

Parameters

batch_size (None) – the batch size to use. If a batching_strategy is provided, this parameter configures that strategy as described below. If no batching_strategy is provided, this can either be an integer specifying the number of samples to save in a batch (in which case batching_strategy is implicitly set to "static") or a float number of seconds between batched saves (in which case batching_strategy is implicitly set to "latency")
batching_strategy (None) –
the batching strategy to use for each save operation. Supported values are:
- "static": a fixed sample batch size for each save
- "size": a target batch size, in bytes, for each save
- "latency": a target latency, in seconds, between saves
By default, fo.config.default_batcher is used

Returns

a tuple of (batch_size, batching_strategy)

fiftyone.core.utils.recommend_batch_size_for_value(value, alpha=0.9, max_size=None)¶

Computes a recommended batch size for the given value type such that a request involving a list of values of this size will be less than alpha * fo.config.batcher_target_size_bytes bytes.

Parameters

value – a value
alpha (0.9) – a safety factor
max_size (None) – an optional max batch size

Returns

a recommended batch size

fiftyone.core.utils.disable_progress_bars()¶

Context manager that temporarily disables all progress bars.

Example usage:

import fiftyone as fo
import fiftyone.zoo as foz

with fo.disable_progress_bars():
    dataset = foz.load_zoo_dataset("quickstart")

class fiftyone.core.utils.UniqueFilenameMaker(output_dir=None, rel_dir=None, alt_dir=None, chunk_size=None, default_ext=None, ignore_exts=False, ignore_existing=False, idempotent=True)¶

Bases: object

A class that generates unique output paths in a directory.

This class provides a get_output_path() method that generates unique filenames in the specified output directory.

If an input path is provided, its filename is maintained, unless a name conflict in output_dir would occur, in which case an index of the form "-%d" % count is appended to the filename.

If no input filename is provided, an output filename of the form <output_dir>/<count><default_ext> is generated, where count is the number of files in output_dir.

If no output_dir is provided, then unique filenames with no base directory are generated.

If a rel_dir is provided, then this path will be stripped from each input path to generate the identifier of each file (rather than just its basename). This argument allows for populating nested subdirectories in output_dir that match the shape of the input paths.

If alt_dir is provided, you can use get_alt_path() to retrieve the equivalent path rooted in this directory rather than output_dir.

Parameters

output_dir (None) – a directory in which to generate output paths
rel_dir (None) – an optional relative directory to strip from each path. The path is converted to an absolute path (if necessary) via fiftyone.core.storage.normalize_path()
alt_dir (None) – an optional alternate directory in which to generate paths when get_alt_path() is called
chunk_size (None) – if provided, output paths will be nested in subdirectories of output_dir with at most this many files per subdirectory. Has no effect if a rel_dir is provided
default_ext (None) – the file extension to use when generating default output paths
ignore_exts (False) – whether to omit file extensions when checking for duplicate filenames
ignore_existing (False) – whether to ignore existing files in output_dir for output filename generation purposes
idempotent (True) – whether to return the same output path when the same input path is provided multiple times (True) or to generate new output paths (False)

Methods:

`seen_input_path`(input_path)	Checks whether we’ve already seen the given input path.
`get_output_path`([input_path, output_ext])	Returns a unique output path.
`get_alt_path`(output_path[, alt_dir])	Returns the alternate path for the given output path generated by `get_output_path()`.

seen_input_path(input_path)¶

Checks whether we’ve already seen the given input path.

Parameters: input_path – an input path
Returns: True/False

get_output_path(input_path=None, output_ext=None)¶

Returns a unique output path.

Parameters

input_path (None) – an input path
output_ext (None) – an optional output extension to use

Returns

the output path

get_alt_path(output_path, alt_dir=None)¶

Returns the alternate path for the given output path generated by get_output_path().

Parameters

output_path – an output path
alt_dir (None) – a directory in which to return the alternate path. If not provided, alt_dir is used

Returns

the corresponding alternate path

class fiftyone.core.utils.MultiProcessUniqueFilenameMaker(ppid, output_dir=None, rel_dir=None, alt_dir=None, default_ext=None, ignore_exts=False, ignore_existing=False, idempotent=True)¶

Bases: object

A class that generates unique output paths in a directory.

This class is multiprocess safe and uses a shared temporary directory structure organized by parent process ID and configuration hash. The approach is robust and handles edge cases like idempotency and file extensions.

This class provides a get_output_path() method that generates unique filenames in the specified output directory.

If an input path is provided, its filename is maintained, unless a name conflict in output_dir would occur, in which case an index of the form "-%d" % count is appended to the filename.

If no input filename is provided, an output filename of the form <output_dir>/<count><default_ext> is generated, where count is the number of files in output_dir.

If no output_dir is provided, then unique filenames with no base directory are generated.

If a rel_dir is provided, then this path will be stripped from each input path to generate the identifier of each file (rather than just its basename). This argument allows for populating nested subdirectories in output_dir that match the shape of the input paths.

If alt_dir is provided, you can use get_alt_path() to retrieve the equivalent path rooted in this directory rather than output_dir.

Parameters

output_dir (None) – a directory in which to generate output paths
rel_dir (None) – an optional relative directory to strip from each path. The path is converted to an absolute path (if necessary) via fiftyone.core.storage.normalize_path()
alt_dir (None) – an optional alternate directory in which to generate paths when get_alt_path() is called
default_ext (None) – the file extension to use when generating default output paths
ignore_exts (False) – whether to omit file extensions when checking for duplicate filenames
ignore_existing (False) – whether to ignore existing files in output_dir for output filename generation purposes
idempotent (True) – whether to return the same output path when the same input path is provided multiple times (True) or to generate new output paths (False)

Methods:

`seen_input_path`(input_path)	Checks whether we’ve already seen the given input path.
`get_output_path`([input_path, output_ext])	Returns a unique output path.
`get_alt_path`(output_path[, alt_dir])	Returns the alternate path for the given output path generated by `get_output_path()`.
`gettempdir`(pid)

seen_input_path(input_path)¶

Checks whether we’ve already seen the given input path.

Parameters: input_path – an input path
Returns: True/False

get_output_path(input_path=None, output_ext=None)¶

Returns a unique output path.

Parameters

input_path (None) – an input path
output_ext (None) – an optional output extension to use

Returns

the output path

get_alt_path(output_path, alt_dir=None)¶

Returns the alternate path for the given output path generated by get_output_path().

Parameters

output_path – an output path
alt_dir (None) – a directory in which to return the alternate path. If not provided, alt_dir is used

Returns

the corresponding alternate path

static gettempdir(pid)¶

fiftyone.core.utils.safe_relpath(path, start=None, default=None)¶

A safe version of os.path.relpath that returns a configurable default value if the given path if it does not lie within the given relative start.

Parameters

path – a path
start (None) – the relative prefix to strip from path
default (None) – a default value to return if path does not lie within start. By default, the basename of the path is returned

Returns

the relative path

fiftyone.core.utils.get_module_name(path, start=None)¶

Gets the Python module name for the given file or directory path.

Parameters

path – a file or directory path
start (None) – the relative prefix to strip from path

Returns

a module.name

fiftyone.core.utils.compute_filehash(filepath, method=None, chunk_size=None)¶

Computes the hash of the given file.

Parameters

filepath – the path to the file
method (None) – an optional hashlib method to use. If not specified, the builtin str.__hash__ will be used
chunk_size (None) – an optional chunk size to use to read the file, in bytes. Only applicable when a method is provided. The default is 64kB. If negative, the entire file is read at once

Returns

the hash

fiftyone.core.utils.serialize_numpy_array(array, ascii=False)¶

Serializes a numpy array.

Parameters

array – a numpy array-like
ascii (False) – whether to return a base64-encoded ASCII string instead of raw bytes

Returns

the serialized bytes

fiftyone.core.utils.deserialize_numpy_array(numpy_bytes, ascii=False)¶

Loads a serialized numpy array generated by serialize_numpy_array().

Parameters

numpy_bytes – the serialized numpy array bytes
ascii (False) – whether the bytes were generated with the ascii == True parameter of serialize_numpy_array()

Returns

the numpy array

fiftyone.core.utils.iter_batches(iterable, batch_size)¶

Iterates over the given iterable in batches.

Parameters

iterable – an iterable
batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits tuples of elements of the requested batch size from the input

fiftyone.core.utils.iter_slices(sliceable, batch_size)¶

Iterates over batches of the given object via slicing.

Parameters

sliceable – an object that supports slicing
batch_size – the desired batch size, or None to return the contents in a single batch

Returns

a generator that emits batches of elements of the requested batch size from the input

fiftyone.core.utils.call_on_exit(callback)¶

Registers the given callback function so that it will be called when the process exits for (almost) any reason

Note that this should only be used from non-interactive scripts because it intercepts ctrl + c.

Covers the following cases: - normal program termination - a Python exception is raised - a SIGTERM signal is received

Parameters: callback – the function to execute upon termination

class fiftyone.core.utils.MonkeyPatchFunction(module_or_fcn, monkey_fcn, fcn_name=None, namespace=None)¶

Bases: object

Context manager that temporarily monkey patches the given function.

If a namespace is provided, all functions with same name as the function you are monkey patching that are imported (recursively) by the module_or_fcn module are also monkey patched.

Parameters

module_or_fcn – a module or function
monkey_fcn – the function to monkey patch in
fcn_name (None) – the name of the function to monkey patch. Required iff module_or_fcn is a module
namespace (None) – an optional package namespace

class fiftyone.core.utils.SetAttributes(obj, **kwargs)¶

Bases: object

Context manager that temporarily sets the attributes of a class to new values.

Parameters

obj – the object
**kwargs – the attribute key-values to set while the context is active

class fiftyone.core.utils.SuppressLogging(level=50)¶

Bases: object

Context manager that temporarily disables system-wide logging.

Parameters: level (logging.CRITICAL) – the logging level at or below which to suppress all messages

class fiftyone.core.utils.add_sys_path(path, index=0)¶

Bases: object

Context manager that temporarily inserts a path to sys.path.

fiftyone.core.utils.is_arm_mac()¶

Determines whether the system is an ARM-based Mac (Apple Silicon).

Returns: True/False

fiftyone.core.utils.is_32_bit()¶

Determines whether the system is 32-bit.

Returns: True/False

fiftyone.core.utils.is_container()¶

Determines if we’re currently running as a container.

Returns: True/False

fiftyone.core.utils.get_multiprocessing_context()¶

Returns the preferred multiprocessing context for the current OS.

When running on macOS or Linux with no start method configured, this method will set the default start method to "fork".

Returns: a multiprocessing context

fiftyone.core.utils.recommend_thread_pool_workers(num_workers=None)¶

Recommends a number of workers for a thread pool.

If a fo.config.max_thread_pool_workers is set, this limit is applied.

Parameters: num_workers (None) – a suggested number of workers
Returns: a number of workers

fiftyone.core.utils.recommend_process_pool_workers(num_workers=None)¶

Recommends a number of workers for a process pool.

If a fo.config.max_process_pool_workers is set, this limit is applied.

Parameters: num_workers (None) – a suggested number of workers
Returns: a number of workers

async fiftyone.core.utils.run_sync_task(func, *args)¶

Run a synchronous function as an async background task.

Parameters

func – a synchronous callable
*args – function arguments

Returns

the function’s return value(s)

fiftyone.core.utils.datetime_to_timestamp(dt)¶

Converts a datetime.date or datetime.datetime to milliseconds since epoch.

Parameters: dt – a datetime.date or datetime.datetime
Returns: the float number of milliseconds since epoch

fiftyone.core.utils.timestamp_to_datetime(ts)¶

Converts a timestamp (number of milliseconds since epoch) to a datetime.datetime.

Parameters: ts – a number of milliseconds since epoch
Returns: a datetime.datetime

fiftyone.core.utils.timedelta_to_ms(td)¶

Converts a datetime.timedelta to milliseconds.

Parameters: td – a datetime.timedelta
Returns: the float number of milliseconds

class fiftyone.core.utils.ResponseStream(response, chunk_size=64)¶

Bases: object

Wrapper around a requests.Response that provides a file-like object interface with read(), seek(), and tell() methods.

Source:: https://gist.github.com/obskyr/b9d4b4223e7eaf4eedcd9defabb34f13

Parameters

response – a requests.Response
chunk_size (64) – the chunk size to use to read the response’s content

Methods:

`read`([size])
`seek`(position[, whence])
`tell`()

read(size=None)¶

seek(position, whence=0)¶

tell()¶

fiftyone.core.utils.to_slug(name)¶

Returns the URL-friendly slug for the given string.

The following strategy is used to generate slugs:

The characters A-Za-z0-9 are converted to lowercase

Whitespace and +_.- are converted to -

All other characters are omitted

All consecutive - characters are reduced to a single -

All leading and trailing - are stripped

Both the input name and the resulting string must be [1, 100] characters in length

Examples:

name                             | slug
---------------------------------+-----------------------
coco_2017                        | coco-2017
c+o+c+o 2-0-1-7                  | c-o-c-o-2-0-1-7
cat.DOG                          | cat-dog
---name----                      | name
Brian's #$&@ (Awesome?) Dataset! | brians-awesome-dataset
sPaM     aNd  EgGs               | spam-and-eggs

Parameters: name – a string
Returns: the slug string
Raises: ValueError – if the name is invalid

fiftyone.core.utils.validate_color(value)¶

Validates that the given value is a valid css color name.

Parameters: value – a value
Raises: ValueError – if value is not a valid css color name.

fiftyone.core.utils.validate_hex_color(value)¶

Validates that the given value is a hex color string or css name.

Parameters: value – a value
Raises: ValueError – if value is not a hex color string