fiftyone.operators.evaluation_metric

Evaluation metric operators.

Copyright 2017-2025, Voxel51, Inc.

voxel51.com

Classes:

EvaluationMetricConfig(name[, label, ...])	Configuration class for evaluation metrics.
EvaluationMetric([_builtin])	Base class for evaluation metric operators.

class fiftyone.operators.evaluation_metric.EvaluationMetricConfig(name, label=None, description=None, eval_types=None, aggregate_key=None, lower_is_better=True, **kwargs)

Bases: OperatorConfig

Configuration class for evaluation metrics.

Parameters:

• name – the name of the evaluation metric

• label (name) – a label for the evaluation metric

• description (None) – a description of the evaluation metric

• eval_types (None) – an optional list of evaluation method types that this metric supports

• aggregate_key (None) – an optional key under which to store the metric’s aggregate value. This is used, for example, by metrics(). By default, the metric’s name is used as its key

• lower_is_better (True) – whether lower values of the metric are better

• **kwargs – other kwargs for fiftyone.operators.OperatorConfig

Methods:

to_json()

to_json()

class fiftyone.operators.evaluation_metric.EvaluationMetric(_builtin=False)

Bases: Operator

Base class for evaluation metric operators.

Methods:

resolve_input(ctx)	Defines any necessary properties to collect this evaluation metric's parameters from a user during prompting.
parse_parameters(ctx, params)	Performs any necessary execution-time formatting to this evaluation metric's parameters.
compute(samples, results, **kwargs)	Computes the evaluation metric for the given collection.
get_fields(samples, config, eval_key)	Lists the fields that were populated by the evaluation metric with the given key, if any.
rename(samples, config, eval_key, new_eval_key)	Performs any necessary operations required to rename this evaluation metric's key.
cleanup(samples, config, eval_key)	Cleans up the results of the evaluation metric with the given key from the collection.
add_secrets(secrets)	Adds secrets to the operator.
execute(ctx)	Executes the operator.
method_to_uri(method_name)	Converts a method name to a URI.
resolve_delegation(ctx)	Returns the resolved forced delegation flag.
resolve_execution_options(ctx)	Returns the resolved execution options.
resolve_output(ctx)	Returns the resolved output property.
resolve_placement(ctx)	Returns the resolved placement of the operator.
resolve_run_name(ctx)	Returns the resolved run name of the operator.
resolve_type(ctx, type)	Returns the resolved input or output property.
to_json()	Returns a JSON representation of the operator.

Attributes:

builtin	Whether the operator is builtin.
config	The OperatorConfig for the operator.
name
uri	The unique identifier of the operator: plugin_name/operator_name.

resolve_input(ctx)

Defines any necessary properties to collect this evaluation metric’s parameters from a user during prompting.

Parameters:

ctx – an fiftyone.operators.ExecutionContext

Returns:

a fiftyone.operators.types.Property, or None

parse_parameters(ctx, params)

Performs any necessary execution-time formatting to this evaluation metric’s parameters.

Parameters:

• ctx – an fiftyone.operators.ExecutionContext

• params – a params dict

compute(samples, results, **kwargs)

Computes the evaluation metric for the given collection.

Parameters:

• samples – a fiftyone.core.collections.SampleCollection

• results – an fiftyone.core.evaluation.EvaluationResults

• **kwargs – arbitrary metric-specific parameters

Returns:

an optional aggregate metric value to store on the results

get_fields(samples, config, eval_key)

Lists the fields that were populated by the evaluation metric with the given key, if any.

Parameters:

• samples – a fiftyone.core.collections.SampleCollection

• config – an fiftyone.core.evaluation.EvaluationMethodConfig

• eval_key – an evaluation key

Returns:

a list of fields

rename(samples, config, eval_key, new_eval_key)

Performs any necessary operations required to rename this evaluation metric’s key.

Parameters:

• samples – a fiftyone.core.collections.SampleCollection

• config – an fiftyone.core.evaluation.EvaluationMethodConfig

• eval_key – an evaluation key

• new_eval_key – a new evaluation key

cleanup(samples, config, eval_key)

Cleans up the results of the evaluation metric with the given key from the collection.

Parameters:

• samples – a fiftyone.core.collections.SampleCollection

• config – an fiftyone.core.evaluation.EvaluationMethodConfig

• eval_key – an evaluation key

add_secrets(secrets)

Adds secrets to the operator.

Parameters:

secrets – a list of secrets

property builtin

Whether the operator is builtin.

property config

The OperatorConfig for the operator.

execute(ctx)

Executes the operator.

Subclasses must implement this method.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

JSON serializable data, or None

method_to_uri(method_name)

Converts a method name to a URI.

Parameters:

method_name – the method name

Returns:

a URI

property name

resolve_delegation(ctx)

Returns the resolved forced delegation flag.

Subclasses can implement this method to decide if delegated execution should be forced for the given operation.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

whether the operation should be delegated (True), run immediately (False), or None to defer to resolve_execution_options() to specify the available options

resolve_execution_options(ctx)

Returns the resolved execution options.

Subclasses can implement this method to define the execution options available for the operation.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

a fiftyone.operators.executor.ExecutionOptions instance

resolve_output(ctx)

Returns the resolved output property.

Subclasses can implement this method to define the outputs of the operator.

By default, this method is called once when the operator is created. If the operator is dynamic, this method is called after the operator is executed.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

a fiftyone.operators.types.Property, or None

resolve_placement(ctx)

Returns the resolved placement of the operator.

Subclasses can implement this method to define the placement of the operator.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

a fiftyone.operators.types.Placement, or None

resolve_run_name(ctx)

Returns the resolved run name of the operator.

Subclasses can implement this method to define the run name of the operator.

Parameters:

ctx – the fiftyone.operators.executor.ExecutionContext

Returns:

a string, or None

resolve_type(ctx, type)

Returns the resolved input or output property.

Parameters:

• ctx – the fiftyone.operators.executor.ExecutionContext

• type – the type of property to resolve, either "inputs" or "outputs"

Returns:

a fiftyone.operators.types.Property, or None

to_json()

Returns a JSON representation of the operator.

Returns:

a JSON dict

property uri

The unique identifier of the operator: plugin_name/operator_name.