fiftyone.core.models#

FiftyOne models.

Copyright 2017-2025, Voxel51, Inc.
voxel51.com

Functions:

`apply_model`(samples, model[, label_field, ...])	Applies the model to the samples in the collection.
`compute_embeddings`(samples, model[, ...])	Computes embeddings for the samples in the collection using the given model.
`compute_patch_embeddings`(samples, model, ...)	Computes embeddings for the image patches defined by `patches_field` of the samples in the collection using the given model.
`load_model`(model_config_dict[, model_path])	Loads the model specified by the given `ModelConfig` dict.

Classes:

`ErrorHandlingCollate`(skip_failures, ...[, ...])
`ModelConfig`(d)	Base configuration class that encapsulates the name of a `Model` and an instance of its associated Config class.
`Model`()	Abstract base class for models.
`LogitsMixin`()	Mixin for `Model` classes that can generate logits for their predictions.
`EmbeddingsMixin`()	Mixin for `Model` classes that can generate embeddings for their predictions.
`PromptMixin`()	Mixin for `Model` classes that can generate prompt embeddings.
`SamplesMixin`()	Mixin for `Model` classes that need samples for prediction.
`TorchModelMixin`()	Mixin for `Model` classes that support feeding data for inference via a `torch.utils.data.DataLoader`.
`SupportsGetItem`()	Mixin for models that support inference with `fiftyone.utils.torch.FiftyOneTorchDataset`.
`ModelManagerConfig`(d)	Config settings for a `ModelManager`.
`ModelManager`(config)	Class for downloading FiftyOne models from the web.

fiftyone.core.models.apply_model(samples, model, label_field='predictions', confidence_thresh=None, store_logits=False, batch_size=None, num_workers=None, skip_failures=True, output_dir=None, rel_dir=None, progress=None, **kwargs)#

Applies the model to the samples in the collection.

This method supports all of the following cases:

Applying an image model to an image collection
Applying an image model to the frames of a video collection
Applying a video model to a video collection

Parameters:

samples – a fiftyone.core.collections.SampleCollection
model – a Model, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
label_field ("predictions") – the name of the field in which to store the model predictions. When performing inference on video frames, the “frames.” prefix is optional
confidence_thresh (None) – an optional confidence threshold to apply to any applicable labels generated by the model
store_logits (False) – whether to store logits for the model predictions. This is only supported when the provided model has logits, model.has_logits == True
batch_size (None) – an optional batch size to use, if the model supports batching
num_workers (None) – the number of workers to use when loading images. Only applicable for Torch-based models
skip_failures (True) – whether to gracefully continue without raising an error if predictions cannot be generated for a sample. Only applicable to Model instances
output_dir (None) – an optional output directory in which to write segmentation images. Only applicable if the model generates segmentations. If none is provided, the segmentations are stored in the database
rel_dir (None) – an optional relative directory to strip from each input filepath to generate a unique identifier that is joined with output_dir to generate an output path for each segmentation image. This argument allows for populating nested subdirectories in output_dir that match the shape of the input paths. The path is converted to an absolute path (if necessary) via fiftyone.core.storage.normalize_path()
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional model-specific keyword arguments passed through to the underlying inference implementation

class fiftyone.core.models.ErrorHandlingCollate(skip_failures, ragged_batches, use_numpy, user_collate_fn=None)#

Bases: object

Methods:

handle_errors(batch)

static handle_errors(batch)#

fiftyone.core.models.compute_embeddings(samples, model, embeddings_field=None, batch_size=None, num_workers=None, skip_failures=True, progress=None, **kwargs)#

Computes embeddings for the samples in the collection using the given model.

This method supports all the following cases:

Using an image model to compute embeddings for an image collection
Using an image model to compute frame embeddings for a video collection
Using a video model to compute embeddings for a video collection

The model must expose embeddings, i.e., Model.has_embeddings() must return True.

If an embeddings_field is provided, the embeddings are saved to the samples; otherwise, the embeddings are returned in-memory.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
model – a Model, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
embeddings_field (None) – the name of a field in which to store the embeddings. When computing video frame embeddings, the “frames.” prefix is optional
batch_size (None) – an optional batch size to use, if the model supports batching
num_workers (None) – the number of workers to use when loading images. Only applicable for Torch-based models
skip_failures (True) – whether to gracefully continue without raising an error if embeddings cannot be generated for a sample. Only applicable to Model instances
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead
**kwargs – optional model-specific keyword arguments passed through to the underlying inference implementation

Returns:

None, if an embeddings_field is provided
a num_samples x num_dim array of embeddings, when computing embeddings for image/video collections with image/video models, respectively, and no embeddings_field is provided. If skip_failures is True and any errors are detected, a list of length num_samples is returned instead containing all successfully computed embedding vectors along with None entries for samples for which embeddings could not be computed
a dictionary mapping sample IDs to num_frames x num_dim arrays of embeddings, when computing frame embeddings for video collections using an image model. If skip_failures is True and any errors are detected, the values of this dictionary will contain arrays of embeddings for all frames 1, 2, … until the error occurred, or None if no embeddings were computed at all

Return type:

one of the following

fiftyone.core.models.compute_patch_embeddings(samples, model, patches_field, embeddings_field=None, force_square=False, alpha=None, handle_missing='skip', batch_size=None, num_workers=None, skip_failures=True, progress=None)#

Computes embeddings for the image patches defined by patches_field of the samples in the collection using the given model.

This method supports all the following cases:

Using an image model to compute patch embeddings for an image collection
Using an image model to compute frame patch embeddings for a video collection

The model must expose embeddings, i.e., Model.has_embeddings() must return True.

If an embeddings_field is provided, the embeddings are saved to the samples; otherwise, the embeddings are returned in-memory.

Parameters:

samples – a fiftyone.core.collections.SampleCollection
model – a Model, Hugging Face Transformers model, Ultralytics model, SuperGradients model, or Lightning Flash model
patches_field – the name of the field defining the image patches in each sample to embed. Must be of type fiftyone.core.labels.Detection, fiftyone.core.labels.Detections, fiftyone.core.labels.Polyline, or fiftyone.core.labels.Polylines. When computing video frame embeddings, the “frames.” prefix is optional
embeddings_field (None) – the name of a label attribute in which to store the embeddings
force_square (False) – whether to minimally manipulate the patch bounding boxes into squares prior to extraction
alpha (None) – an optional expansion/contraction to apply to the patches before extracting them, in [-1, inf). If provided, the length and width of the box are expanded (or contracted, when alpha < 0) by (100 * alpha)%. For example, set alpha = 0.1 to expand the boxes by 10%, and set alpha = -0.1 to contract the boxes by 10%
handle_missing ("skip") –
how to handle images with no patches. Supported values are:
- ”skip”: skip the image and assign its embedding as None
- ”image”: use the whole image as a single patch
- ”error”: raise an error
batch_size (None) – an optional batch size to use, if the model supports batching
num_workers (None) – the number of workers to use when loading images. Only applicable for Torch models
skip_failures (True) – whether to gracefully continue without raising an error if embeddings cannot be generated for a sample
progress (None) – whether to render a progress bar (True/False), use the default value fiftyone.config.show_progress_bars (None), or a progress callback function to invoke instead

Returns:

None, if an embeddings_field is provided
a dict mapping sample IDs to num_patches x num_dim arrays of patch embeddings, when computing patch embeddings for image collections and no embeddings_field is provided. If skip_failures is True and any errors are detected, this dictionary will contain None values for any samples for which embeddings could not be computed
a dict of dicts mapping sample IDs to frame numbers to num_patches x num_dim arrays of patch embeddings, when computing patch embeddings for the frames of video collections and no embeddings_field is provided. If skip_failures is True and any errors are detected, this nested dict will contain missing or None values to indicate uncomputable embeddings

Return type:

one of the following

fiftyone.core.models.load_model(model_config_dict, model_path=None, **kwargs)#

Loads the model specified by the given ModelConfig dict.

Parameters:

model_config_dict – a ModelConfig dict
model_path (None) – an optional model path to inject into the model_path field of the model’s Config instance, which must implement the eta.core.learning.HasPublishedModel interface. This is useful when working with a model whose weights are stored locally and do not need to be downloaded
**kwargs – optional keyword arguments to inject into the model’s Config instance

Returns:

a Model instance

class fiftyone.core.models.ModelConfig(d)#

Bases: ModelConfig

Base configuration class that encapsulates the name of a Model and an instance of its associated Config class.

Parameters:

type – the fully-qualified class name of the Model subclass
config – an instance of the Config class associated with the model

Methods:

`attributes`()	Returns a list of class attributes to be serialized.
`build`()	Factory method that builds the Model instance from the config specified by this class.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a Config object from a JSON dictionary.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

attributes()#

Returns a list of class attributes to be serialized.

This method is called internally by serialize() to determine the class attributes to serialize.

Subclasses can override this method, but, by default, all attributes in vars(self) are returned, minus private attributes, i.e., those starting with “_”. The order of the attributes in this list is preserved when serializing objects, so a common pattern is for subclasses to override this method if they want their JSON files to be organized in a particular way.

Returns:: a list of class attributes to be serialized

build()#

Factory method that builds the Model instance from the config specified by this class.

Returns:: a Model instance

classmethod builder()#: Returns a ConfigBuilder instance for this class.

copy()#

Returns a deep copy of the object.

Returns:: a Serializable instance

custom_attributes(dynamic=False, private=False)#

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters:

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns:

a list of class attributes

classmethod default()#

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)#

Constructs a Config object from a JSON dictionary.

Config subclass constructors accept JSON dictionaries, so this method simply passes the dictionary to cls().

Parameters:: d – a dict of fields expected by cls
Returns:: an instance of cls

classmethod from_json(path, *args, **kwargs)#

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters:

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod from_kwargs(**kwargs)#

Constructs a Config object from keyword arguments.

Parameters:: **kwargs – keyword arguments that define the fields expected by cls
Returns:: an instance of cls

classmethod from_str(s, *args, **kwargs)#

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters:

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod get_class_name()#: Returns the fully-qualified class name string of this object.

classmethod load_default()#

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw array attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns:

a list of raw (untouched) values

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)#

Parses a boolean value.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns:

True/False

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)#

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters:

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field, which is equal to a value from choices

Raises:

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)#

Parses a dictionary attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns:

a dictionary

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)#

Parses an integer attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns:

an int

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)#

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Returns:: the (field, value) that was set
Raises:: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)#

Parses a number attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns:

a number (e.g. int, float)

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns:

an instance of cls

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns:

a list of cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns:

a dictionary whose values are cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)#

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a path string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw (arbitrary) JSON field.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field

Raises:

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)#

Parses a string attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

serialize(reflective=False)#

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters:: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns:: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)#

Returns a string representation of this object.

Parameters:

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns:

a string representation of the object

static validate_all_or_nothing_fields(fields)#

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Raises:: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)#

Serializes the object and writes it to disk.

Parameters:

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.core.models.Model#

Bases: Model

Abstract base class for models.

This class declares the following conventions:

Model.__init__() should take a single config argument that is an instance of <Model>Config
Models implement the context manager interface. This means that models can optionally use context to perform any necessary setup and teardown, and so any code that builds a model should use the with syntax

Attributes:

`media_type`	The media type processed by the model.
`has_logits`	Whether this instance can generate logits for its predictions.
`has_embeddings`	Whether this instance can generate embeddings.
`can_embed_prompts`	Whether this instance can generate prompt embeddings.
`ragged_batches`	True/False whether `transforms()` may return tensors of different sizes.
`transforms`	The preprocessing function that will/must be applied to each input before prediction, or `None` if no preprocessing is performed.
`preprocess`	Whether to apply `transforms()` during inference (True) or to assume that they have already been applied (False).

Methods:

`predict`(arg)	Performs prediction on the given data.
`predict_all`(args)	Performs prediction on the given iterable of data.
`from_config`(config)	Instantiates a Configurable class from a <cls>Config instance.
`from_dict`(d)	Instantiates a Configurable class from a <cls>Config dict.
`from_json`(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
`from_kwargs`(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
`parse`(class_name[, module_name])	Parses a Configurable subclass name string.
`validate`(config)	Validates that the given config is an instance of <cls>Config.

property media_type#

The media type processed by the model.

Supported values are “image” and “video”.

property has_logits#

Whether this instance can generate logits for its predictions.

This method returns False by default. Methods that can generate logits will override this via implementing the LogitsMixin interface.

property has_embeddings#

Whether this instance can generate embeddings.

This method returns False by default. Methods that can generate embeddings will override this via implementing the EmbeddingsMixin interface.

property can_embed_prompts#

Whether this instance can generate prompt embeddings.

This method returns False by default. Methods that can generate prompt embeddings will override this via implementing the PromptMixin interface.

property ragged_batches#: True/False whether transforms() may return tensors of different sizes. If True, then passing ragged lists of data to predict_all() is not allowed.

property transforms#: The preprocessing function that will/must be applied to each input before prediction, or None if no preprocessing is performed.

property preprocess#: Whether to apply transforms() during inference (True) or to assume that they have already been applied (False).

predict(arg)#

Performs prediction on the given data.

Image models should support, at minimum, processing arg values that are uint8 numpy arrays (HWC).

Video models should support, at minimum, processing arg values that are eta.core.video.VideoReader instances.

Parameters:: arg – the data
Returns:: a fiftyone.core.labels.Label instance or dict of fiftyone.core.labels.Label instances containing the predictions

predict_all(args)#

Performs prediction on the given iterable of data.

Image models should support, at minimum, processing args values that are either lists of uint8 numpy arrays (HWC) or numpy array tensors (NHWC).

Video models should support, at minimum, processing args values that are lists of eta.core.video.VideoReader instances.

Subclasses can override this method to increase efficiency, but, by default, this method simply iterates over the data and applies predict() to each.

Parameters:: args – an iterable of data
Returns:: a list of fiftyone.core.labels.Label instances or a list of dicts of fiftyone.core.labels.Label instances containing the predictions

classmethod from_config(config)#: Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)#

Instantiates a Configurable class from a <cls>Config dict.

Parameters:: d – a dict to construct a <cls>Config
Returns:: an instance of cls

classmethod from_json(json_path)#

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters:: json_path – path to a JSON file for type <cls>Config
Returns:: an instance of cls

classmethod from_kwargs(**kwargs)#

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters:: **kwargs – keyword arguments that define the fields of a <cls>Config dict
Returns:: an instance of cls

static parse(class_name, module_name=None)#

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters:

class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”
module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns:

the Configurable class config_cls: the Config class associated with cls

Return type:

cls

classmethod validate(config)#

Validates that the given config is an instance of <cls>Config.

Raises:: ConfigurableError – if config is not an instance of <cls>Config

class fiftyone.core.models.LogitsMixin#

Bases: object

Mixin for Model classes that can generate logits for their predictions.

This mixin allows for the possibility that only some instances of a class are capable of generating logits, per the value of the has_logits() property.

Attributes:

`store_logits`	Whether the model should store logits in its predictions.
`has_logits`	Whether this instance can generate logits.

property store_logits#: Whether the model should store logits in its predictions.

property has_logits#: Whether this instance can generate logits.

class fiftyone.core.models.EmbeddingsMixin#

Bases: object

Mixin for Model classes that can generate embeddings for their predictions.

This mixin allows for the possibility that only some instances of a class are capable of generating embeddings, per the value of the has_embeddings() property.

Attributes:

has_embeddings

Whether this instance has embeddings.

Methods:

`get_embeddings`()	Returns the embeddings generated by the last forward pass of the model.
`embed`(arg)	Generates an embedding for the given data.
`embed_all`(args)	Generates embeddings for the given iterable of data.

property has_embeddings#: Whether this instance has embeddings.

get_embeddings()#

Returns the embeddings generated by the last forward pass of the model.

By convention, this method should always return an array whose first axis represents batch size (which will always be 1 when predict() was last used).

Returns:: a numpy array containing the embedding(s)

embed(arg)#

Generates an embedding for the given data.

Subclasses can override this method to increase efficiency, but, by default, this method simply calls predict() and then returns get_embeddings().

Parameters:: arg – the data. See predict() for details
Returns:: a numpy array containing the embedding

embed_all(args)#

Generates embeddings for the given iterable of data.

Subclasses can override this method to increase efficiency, but, by default, this method simply iterates over the data and applies embed() to each.

Parameters:: args – an iterable of data. See predict_all() for details
Returns:: a numpy array containing the embeddings stacked along axis 0

class fiftyone.core.models.PromptMixin#

Bases: object

Mixin for Model classes that can generate prompt embeddings.

This mixin allows for the possibility that only some instances of a class are capable of generating prompt embeddings, per the value of the can_embed_prompts() property.

Attributes:

can_embed_prompts

Whether this instance can generate prompt embeddings.

Methods:

`embed_prompt`(arg)	Generates an embedding for the given prompt.
`embed_prompts`(args)	Generates embeddings for the given prompts.

property can_embed_prompts#: Whether this instance can generate prompt embeddings.

embed_prompt(arg)#

Generates an embedding for the given prompt.

Parameters:: arg – the prompt
Returns:: a numpy array containing the embedding

embed_prompts(args)#

Generates embeddings for the given prompts.

Subclasses can override this method to increase efficiency, but, by default, this method simply iterates over the data and applies embed_prompt() to each.

Parameters:: args – an iterable of prompts
Returns:: a numpy array containing the embeddings stacked along axis 0

class fiftyone.core.models.SamplesMixin#

Bases: object

Mixin for Model classes that need samples for prediction.

Models can implement this mixin to declare that they require one or more fields of the current sample when performing inference on its media.

The fields are get/set via needs_fields(), which is a dict that maps model-specific keys to sample field names:

model.needs_fields = {"key1": "field1", "key2": "field2", ...}

Attributes:

needs_fields

A dict mapping model-specific keys to sample field names.

Methods:

`predict`(arg[, sample])	Performs prediction on the given data.
`predict_all`(args[, samples])	Performs prediction on the given iterable of data.

property needs_fields#: A dict mapping model-specific keys to sample field names.

predict(arg, sample=None)#

Performs prediction on the given data.

Image models should support, at minimum, processing arg values that are uint8 numpy arrays (HWC).

Video models should support, at minimum, processing arg values that are eta.core.video.VideoReader instances.

Parameters:

arg – the data
sample (None) – the fiftyone.core.sample.Sample associated with the data

Returns:

a fiftyone.core.labels.Label instance or dict of fiftyone.core.labels.Label instances containing the predictions

predict_all(args, samples=None)#

Performs prediction on the given iterable of data.

Image models should support, at minimum, processing args values that are either lists of uint8 numpy arrays (HWC) or numpy array tensors (NHWC).

Video models should support, at minimum, processing args values that are lists of eta.core.video.VideoReader instances.

Subclasses can override this method to increase efficiency, but, by default, this method simply iterates over the data and applies predict() to each.

Parameters:

args – an iterable of data
samples (None) – an iterable of fiftyone.core.sample.Sample instances associated with the data

Returns:

a list of fiftyone.core.labels.Label instances or a list of dicts of fiftyone.core.labels.Label instances containing the predictions

class fiftyone.core.models.TorchModelMixin#

Bases: object

Mixin for Model classes that support feeding data for inference via a torch.utils.data.DataLoader.

Models implementing this mixin must expose via their Model.transforms() property the torchvision:torchvision.transforms function that will/must be applied to each input before prediction.

Attributes:

has_collate_fn

Whether this model has a custom collate function.

Methods:

collate_fn(batch)

The collate function to use when creating dataloaders for this model.

property has_collate_fn#

Whether this model has a custom collate function.

Set this to True if you want collate_fn() to be used during inference.

static collate_fn(batch)#

The collate function to use when creating dataloaders for this model.

In order to enable this functionality, the model’s has_collate_fn() property must return True.

By default, this is the identity function, but subclasses can override this method as necessary.

Note that this function must be serializable so it is compatible with multiprocessing for dataloaders.

Parameters:: batch – a list of items to collate
Returns:: the collated batch, which will be fed directly to the model

class fiftyone.core.models.SupportsGetItem#

Bases: object

Mixin for models that support inference with fiftyone.utils.torch.FiftyOneTorchDataset.

Models that implement this mixin must implement build_get_item() to build the fiftyone.utils.torch.GetItem instance that defines how their data should be loaded by data loaders.

Attributes:

required_keys

The required keys that must be provided as parameters to methods like apply_model() and compute_embeddings() at runtime.

Methods:

build_get_item([field_mapping])

Builds the fiftyone.utils.torch.GetItem instance that defines how the model's data should be loaded by data loaders.

property required_keys#: The required keys that must be provided as parameters to methods like apply_model() and compute_embeddings() at runtime.

build_get_item(field_mapping=None)#

Builds the fiftyone.utils.torch.GetItem instance that defines how the model’s data should be loaded by data loaders.

Parameters:: field_mapping (None) – a user-provided dict mapping required keys to dataset field names
Returns:: a fiftyone.utils.torch.GetItem instance

class fiftyone.core.models.ModelManagerConfig(d)#

Bases: ModelManagerConfig

Config settings for a ModelManager.

Parameters:

url (None) – the URL of the file
google_drive_id (None) – the ID of the file in Google Drive
extract_archive (None) – whether to extract the downloaded model, which is assumed to be an archive
delete_archive (None) – whether to delete the archive after extracting it, if applicable

Methods:

`attributes`()	Returns a list of attributes to be serialized.
`builder`()	Returns a ConfigBuilder instance for this class.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`default`()	Returns the default config instance.
`from_dict`(d)	Constructs a Config object from a JSON dictionary.
`from_json`(path, args, *kwargs)	Constructs a Serializable object from a JSON file.
`from_kwargs`(**kwargs)	Constructs a Config object from keyword arguments.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`load_default`()	Loads the default config instance from file.
`parse_array`(d, key[, default])	Parses a raw array attribute.
`parse_bool`(d, key[, default])	Parses a boolean value.
`parse_categorical`(d, key, choices[, default])	Parses a categorical JSON field, which must take a value from among the given choices.
`parse_dict`(d, key[, default])	Parses a dictionary attribute.
`parse_int`(d, key[, default])	Parses an integer attribute.
`parse_mutually_exclusive_fields`(fields)	Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.
`parse_number`(d, key[, default])	Parses a number attribute.
`parse_object`(d, key, cls[, default])	Parses an object attribute.
`parse_object_array`(d, key, cls[, default])	Parses an array of objects.
`parse_object_dict`(d, key, cls[, default])	Parses a dictionary whose values are objects.
`parse_path`(d, key[, default])	Parses a path attribute.
`parse_raw`(d, key[, default])	Parses a raw (arbitrary) JSON field.
`parse_string`(d, key[, default])	Parses a string attribute.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate_all_or_nothing_fields`(fields)	Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.

attributes()#

Returns a list of attributes to be serialized.

Returns:: a list of attributes

classmethod builder()#: Returns a ConfigBuilder instance for this class.

copy()#

Returns a deep copy of the object.

Returns:: a Serializable instance

custom_attributes(dynamic=False, private=False)#

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters:

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns:

a list of class attributes

classmethod default()#

Returns the default config instance.

By default, this method instantiates the class from an empty dictionary, which will only succeed if all attributes are optional. Otherwise, subclasses should override this method to provide the desired default configuration.

classmethod from_dict(d)#

Constructs a Config object from a JSON dictionary.

Config subclass constructors accept JSON dictionaries, so this method simply passes the dictionary to cls().

Parameters:: d – a dict of fields expected by cls
Returns:: an instance of cls

classmethod from_json(path, *args, **kwargs)#

Constructs a Serializable object from a JSON file.

Subclasses may override this method, but, by default, this method simply reads the JSON and calls from_dict(), which subclasses must implement.

Parameters:

path – the path to the JSON file on disk
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod from_kwargs(**kwargs)#

Constructs a Config object from keyword arguments.

Parameters:: **kwargs – keyword arguments that define the fields expected by cls
Returns:: an instance of cls

classmethod from_str(s, *args, **kwargs)#

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters:

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod get_class_name()#: Returns the fully-qualified class name string of this object.

classmethod load_default()#

Loads the default config instance from file.

Subclasses must implement this method if they intend to support default instances.

static parse_array(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw array attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default list to return if key is not present

Returns:

a list of raw (untouched) values

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_bool(d, key, default=<eta.core.config.NoDefault object>)#

Parses a boolean value.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default bool to return if key is not present

Returns:

True/False

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_categorical(d, key, choices, default=<eta.core.config.NoDefault object>)#

Parses a categorical JSON field, which must take a value from among the given choices.

Parameters:

d – a JSON dictionary
key – the key to parse
choices – either an iterable of possible values or an enum-like class whose attributes define the possible values
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field, which is equal to a value from choices

Raises:

ConfigError – if the key was present in the dictionary but its value was not an allowed choice, or if no default value was provided and the key was not found in the dictionary

static parse_dict(d, key, default=<eta.core.config.NoDefault object>)#

Parses a dictionary attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default dict to return if key is not present

Returns:

a dictionary

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_int(d, key, default=<eta.core.config.NoDefault object>)#

Parses an integer attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default integer value to return if key is not present

Returns:

an int

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_mutually_exclusive_fields(fields)#

Parses a mutually exclusive dictionary of pre-parsed fields, which must contain exactly one field with a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Returns:: the (field, value) that was set
Raises:: ConfigError – if zero or more than one truthy value was found

static parse_number(d, key, default=<eta.core.config.NoDefault object>)#

Parses a number attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default numeric value to return if key is not present

Returns:

a number (e.g. int, float)

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an object attribute.

The value of d[key] can be either an instance of cls or a serialized dict from an instance of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of d[key]
default – a default cls instance to return if key is not present

Returns:

an instance of cls

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_array(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses an array of objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the elements of list d[key]
default – the default list to return if key is not present

Returns:

a list of cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_object_dict(d, key, cls, default=<eta.core.config.NoDefault object>)#

Parses a dictionary whose values are objects.

The values in d[key] can be either instances of cls or serialized dicts from instances of cls.

Parameters:

d – a JSON dictionary
key – the key to parse
cls – the class of the values of dictionary d[key]
default – the default dict of cls instances to return if key is not present

Returns:

a dictionary whose values are cls instances

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_path(d, key, default=<eta.core.config.NoDefault object>)#

Parses a path attribute.

The path is converted to an absolute path if necessary via os.path.abspath(os.path.expanduser(value)).

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a path string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

static parse_raw(d, key, default=<eta.core.config.NoDefault object>)#

Parses a raw (arbitrary) JSON field.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default value to return if key is not present

Returns:

the raw (untouched) value of the given field

Raises:

ConfigError – if no default value was provided and the key was not found in the dictionary

static parse_string(d, key, default=<eta.core.config.NoDefault object>)#

Parses a string attribute.

Parameters:

d – a JSON dictionary
key – the key to parse
default – a default string to return if key is not present

Returns:

a string

Raises:

ConfigError – if the field value was the wrong type or no default value was provided and the key was not found in the dictionary

serialize(reflective=False)#

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters:: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns:: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)#

Returns a string representation of this object.

Parameters:

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns:

a string representation of the object

static validate_all_or_nothing_fields(fields)#

Validates a dictionary of pre-parsed fields checking that either all or none of the fields have a truthy value.

Parameters:: fields – a dictionary of pre-parsed fields
Raises:: ConfigError – if some values are truth and some are not

write_json(path, pretty_print=False, **kwargs)#

Serializes the object and writes it to disk.

Parameters:

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

class fiftyone.core.models.ModelManager(config)#

Bases: ModelManager

Class for downloading FiftyOne models from the web.

Methods:

`attributes`()	Returns a list of attributes to be serialized.
`copy`()	Returns a deep copy of the object.
`custom_attributes`([dynamic, private])	Returns a customizable list of class attributes.
`download_model`(model_path[, force])	Downloads the model to the given local path.
`flush_model`(model_path)	Flushes the copy of the model at the given local path, if necessary.
`from_config`(config)	Instantiates a Configurable class from a <cls>Config instance.
`from_dict`(d)	Builds the ModelManager subclass from a JSON dictionary.
`from_json`(json_path)	Instantiates a Configurable class from a <cls>Config JSON file.
`from_kwargs`(**kwargs)	Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.
`from_str`(s, args, *kwargs)	Constructs a Serializable object from a JSON string.
`get_class_name`()	Returns the fully-qualified class name string of this object.
`is_model_downloaded`(model_path)	Determines whether the model is downloaded to the given location.
`parse`(class_name[, module_name])	Parses a Configurable subclass name string.
`serialize`([reflective])	Serializes the object into a dictionary.
`to_str`([pretty_print])	Returns a string representation of this object.
`validate`(config)	Validates that the given config is an instance of <cls>Config.
`write_json`(path[, pretty_print])	Serializes the object and writes it to disk.
`upload_model`(model_path, args, *kwargs)
`delete_model`()	Deletes the model from remote storage.

attributes()#

Returns a list of attributes to be serialized.

Returns:: a list of attributes

copy()#

Returns a deep copy of the object.

Returns:: a Serializable instance

custom_attributes(dynamic=False, private=False)#

Returns a customizable list of class attributes.

By default, all attributes in vars(self) are returned, minus private attributes (those starting with “_”).

Parameters:

dynamic – whether to include dynamic properties, e.g., those defined by getter/setter methods or the @property decorator. By default, this is False
private – whether to include private properties, i.e., those starting with “_”. By default, this is False

Returns:

a list of class attributes

download_model(model_path, force=False)#

Downloads the model to the given local path.

If the download is forced, any existing model is overwritten. If the download is not forced, the model will only be downloaded if it does not already exist locally.

Parameters:

model_path – the path to which to download the model
force – whether to force download the model. If True, the model is always downloaded. If False, the model is only downloaded if necessary. The default is False

Raises:

ModelError – if model downloading is not currently allowed

flush_model(model_path)#

Flushes the copy of the model at the given local path, if necessary.

Parameters:: model_path – the path on disk for the model

classmethod from_config(config)#: Instantiates a Configurable class from a <cls>Config instance.

classmethod from_dict(d)#

Builds the ModelManager subclass from a JSON dictionary.

Parameters:: d – a JSON dictionary
Returns:: a ModelManager instance

classmethod from_json(json_path)#

Instantiates a Configurable class from a <cls>Config JSON file.

Parameters:: json_path – path to a JSON file for type <cls>Config
Returns:: an instance of cls

classmethod from_kwargs(**kwargs)#

Instantiates a Configurable class from keyword arguments defining the attributes of a <cls>Config.

Parameters:: **kwargs – keyword arguments that define the fields of a <cls>Config dict
Returns:: an instance of cls

classmethod from_str(s, *args, **kwargs)#

Constructs a Serializable object from a JSON string.

Subclasses may override this method, but, by default, this method simply parses the string and calls from_dict(), which subclasses must implement.

Parameters:

s – a JSON string representation of a Serializable object
*args – optional positional arguments for self.from_dict()
**kwargs – optional keyword arguments for self.from_dict()

Returns:

an instance of the Serializable class

classmethod get_class_name()#: Returns the fully-qualified class name string of this object.

is_model_downloaded(model_path)#

Determines whether the model is downloaded to the given location.

If model_path is an archive, this method will also return True if a directory with the same basename as model_path exists.

Parameters:: model_path – the path on disk for the model
Returns:: True/False

static parse(class_name, module_name=None)#

Parses a Configurable subclass name string.

Assumes both the Configurable class and the Config class are defined in the same module. The module containing the classes will be loaded if necessary.

Parameters:

class_name – a string containing the name of the Configurable class, e.g. “ClassName”, or a fully-qualified class name, e.g. “eta.core.config.ClassName”
module_name – a string containing the fully-qualified module name, e.g. “eta.core.config”, or None if class_name includes the module name. Set module_name = __name__ to load a class from the calling module

Returns:

the Configurable class config_cls: the Config class associated with cls

Return type:

cls

serialize(reflective=False)#

Serializes the object into a dictionary.

Serialization is applied recursively to all attributes in the object, including element-wise serialization of lists and dictionary values.

Parameters:: reflective – whether to include reflective attributes when serializing the object. By default, this is False
Returns:: a JSON dictionary representation of the object

to_str(pretty_print=True, **kwargs)#

Returns a string representation of this object.

Parameters:

pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is True
**kwargs – optional keyword arguments for self.serialize()

Returns:

a string representation of the object

classmethod validate(config)#

Validates that the given config is an instance of <cls>Config.

Raises:: ConfigurableError – if config is not an instance of <cls>Config

write_json(path, pretty_print=False, **kwargs)#

Serializes the object and writes it to disk.

Parameters:

path – the output path
pretty_print – whether to render the JSON in human readable format with newlines and indentations. By default, this is False
**kwargs – optional keyword arguments for self.serialize()

static upload_model(model_path, *args, **kwargs)#

delete_model()#: Deletes the model from remote storage.