# fiftyone.operators.evaluation_metric Evaluation metric operators. Copyright 2017-2026, Voxel51, Inc.
[voxel51.com](https://voxel51.com/)

**Classes:** | [`EvaluationMetricConfig`](#fiftyone.operators.evaluation_metric.EvaluationMetricConfig)(name[, label, ...]) | Configuration class for evaluation metrics. | |----------------------------------------------------------------------------------------------------------------|-----------------------------------------------| | [`EvaluationMetric`](#fiftyone.operators.evaluation_metric.EvaluationMetric)([_builtin]) | Base class for evaluation metric operators. | ### *class* fiftyone.operators.evaluation_metric.EvaluationMetricConfig(name, label=None, description=None, eval_types=None, aggregate_key=None, lower_is_better=True, \*\*kwargs) Bases: [`OperatorConfig`](fiftyone.operators.operator.md#fiftyone.operators.operator.OperatorConfig) Configuration class for evaluation metrics. * **Parameters:** * **name** – the name of the evaluation metric * **label** ([*name*](fiftyone.core.annotation.attributes.md#fiftyone.core.annotation.attributes.AttributeSpec.name)) – a label for the evaluation metric * **description** (*None*) – a description of the evaluation metric * **eval_types** (*None*) – an optional list of evaluation method types that this metric supports * **aggregate_key** (*None*) – an optional key under which to store the metric’s aggregate value. This is used, for example, by [`metrics()`](fiftyone.utils.eval.base.md#fiftyone.utils.eval.base.BaseEvaluationResults.metrics). By default, the metric’s `name` is used as its key * **lower_is_better** (*True*) – whether lower values of the metric are better * **\*\*kwargs** – other kwargs for [`fiftyone.operators.OperatorConfig`](fiftyone.operators.md#fiftyone.operators.OperatorConfig) **Attributes:** | [`risk_level`](#fiftyone.operators.evaluation_metric.EvaluationMetricConfig.risk_level) | The declared `RiskLevel` for this operator. | |-------------------------------------------------------------------------------------------|-----------------------------------------------| **Methods:** | [`to_json`](#fiftyone.operators.evaluation_metric.EvaluationMetricConfig.to_json)() | | |---------------------------------------------------------------------------------------|----| #### *property* risk_level The declared `RiskLevel` for this operator. #### to_json() ### *class* fiftyone.operators.evaluation_metric.EvaluationMetric(\_builtin=False) Bases: [`Operator`](fiftyone.operators.operator.md#fiftyone.operators.operator.Operator) Base class for evaluation metric operators. **Methods:** | [`resolve_input`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_input)(ctx) | Defines any necessary properties to collect this evaluation metric's parameters from a user during prompting. | |----------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| | [`parse_parameters`](#fiftyone.operators.evaluation_metric.EvaluationMetric.parse_parameters)(ctx, params) | Performs any necessary execution-time formatting to this evaluation metric's parameters. | | [`compute`](#fiftyone.operators.evaluation_metric.EvaluationMetric.compute)(samples, results, \*\*kwargs) | Computes the evaluation metric for the given collection. | | [`get_fields`](#fiftyone.operators.evaluation_metric.EvaluationMetric.get_fields)(samples, config, eval_key) | Lists the fields that were populated by the evaluation metric with the given key, if any. | | [`rename`](#fiftyone.operators.evaluation_metric.EvaluationMetric.rename)(samples, config, eval_key, new_eval_key) | Performs any necessary operations required to rename this evaluation metric's key. | | [`cleanup`](#fiftyone.operators.evaluation_metric.EvaluationMetric.cleanup)(samples, config, eval_key) | Cleans up the results of the evaluation metric with the given key from the collection. | | [`add_secrets`](#fiftyone.operators.evaluation_metric.EvaluationMetric.add_secrets)(secrets) | Adds secrets to the operator. | | [`execute`](#fiftyone.operators.evaluation_metric.EvaluationMetric.execute)(ctx) | Executes the operator. | | [`method_to_uri`](#fiftyone.operators.evaluation_metric.EvaluationMetric.method_to_uri)(method_name) | Converts a method name to a URI. | | [`resolve_delegation`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_delegation)(ctx) | Returns the resolved *forced* delegation flag. | | [`resolve_execution_options`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_execution_options)(ctx) | Returns the resolved execution options. | | [`resolve_output`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_output)(ctx) | Returns the resolved output property. | | [`resolve_placement`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_placement)(ctx) | Returns the resolved placement of the operator. | | [`resolve_run_name`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_run_name)(ctx) | Returns the resolved run name of the operator. | | [`resolve_type`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_type)(ctx, type) | Returns the resolved input or output property. | | [`to_json`](#fiftyone.operators.evaluation_metric.EvaluationMetric.to_json)() | Returns a JSON representation of the operator. | **Attributes:** | [`builtin`](#fiftyone.operators.evaluation_metric.EvaluationMetric.builtin) | Whether the operator is builtin. | |-----------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | [`config`](#fiftyone.operators.evaluation_metric.EvaluationMetric.config) | The `OperatorConfig` for the operator. | | [`name`](#fiftyone.operators.evaluation_metric.EvaluationMetric.name) | | | [`risk_level`](#fiftyone.operators.evaluation_metric.EvaluationMetric.risk_level) | The effective risk level of the operator, which is used by guardrail systems of an agent to classify tool calls. | | [`uri`](#fiftyone.operators.evaluation_metric.EvaluationMetric.uri) | The unique identifier of the operator: `plugin_name/operator_name`. | #### resolve_input(ctx) Defines any necessary properties to collect this evaluation metric’s parameters from a user during prompting. * **Parameters:** **ctx** – an [`fiftyone.operators.ExecutionContext`](fiftyone.operators.md#fiftyone.operators.ExecutionContext) * **Returns:** a [`fiftyone.operators.types.Property`](fiftyone.operators.types.md#fiftyone.operators.types.Property), or None #### parse_parameters(ctx, params) Performs any necessary execution-time formatting to this evaluation metric’s parameters. * **Parameters:** * **ctx** – an [`fiftyone.operators.ExecutionContext`](fiftyone.operators.md#fiftyone.operators.ExecutionContext) * **params** – a params dict #### compute(samples, results, \*\*kwargs) Computes the evaluation metric for the given collection. * **Parameters:** * **samples** – a [`fiftyone.core.collections.SampleCollection`](fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection) * **results** – an [`fiftyone.core.evaluation.EvaluationResults`](fiftyone.core.evaluation.md#fiftyone.core.evaluation.EvaluationResults) * **\*\*kwargs** – arbitrary metric-specific parameters * **Returns:** an optional aggregate metric value to store on the results #### get_fields(samples, config, eval_key) Lists the fields that were populated by the evaluation metric with the given key, if any. * **Parameters:** * **samples** – a [`fiftyone.core.collections.SampleCollection`](fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection) * **config** – an [`fiftyone.core.evaluation.EvaluationMethodConfig`](fiftyone.core.evaluation.md#fiftyone.core.evaluation.EvaluationMethodConfig) * **eval_key** – an evaluation key * **Returns:** a list of fields #### rename(samples, config, eval_key, new_eval_key) Performs any necessary operations required to rename this evaluation metric’s key. * **Parameters:** * **samples** – a [`fiftyone.core.collections.SampleCollection`](fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection) * **config** – an [`fiftyone.core.evaluation.EvaluationMethodConfig`](fiftyone.core.evaluation.md#fiftyone.core.evaluation.EvaluationMethodConfig) * **eval_key** – an evaluation key * **new_eval_key** – a new evaluation key #### cleanup(samples, config, eval_key) Cleans up the results of the evaluation metric with the given key from the collection. * **Parameters:** * **samples** – a [`fiftyone.core.collections.SampleCollection`](fiftyone.core.collections.md#fiftyone.core.collections.SampleCollection) * **config** – an [`fiftyone.core.evaluation.EvaluationMethodConfig`](fiftyone.core.evaluation.md#fiftyone.core.evaluation.EvaluationMethodConfig) * **eval_key** – an evaluation key #### add_secrets(secrets) Adds secrets to the operator. * **Parameters:** **secrets** – a list of secrets #### *property* builtin Whether the operator is builtin. #### *property* config The `OperatorConfig` for the operator. #### execute(ctx) Executes the operator. Subclasses must implement this method. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** JSON serializable data, or None #### method_to_uri(method_name) Converts a method name to a URI. * **Parameters:** **method_name** – the method name * **Returns:** a URI #### *property* name #### resolve_delegation(ctx) → [bool](fiftyone.core.stages.md#fiftyone.core.stages.Exists.bool) | None Returns the resolved *forced* delegation flag. Subclasses can implement this method to decide if delegated execution should be *forced* for the given operation. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** whether the operation should be delegated (True), run immediately (False), or None to defer to [`resolve_execution_options()`](#fiftyone.operators.evaluation_metric.EvaluationMetric.resolve_execution_options) to specify the available options #### resolve_execution_options(ctx) Returns the resolved execution options. Subclasses can implement this method to define the execution options available for the operation. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** a [`fiftyone.operators.executor.ExecutionOptions`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionOptions) instance #### resolve_output(ctx) Returns the resolved output property. Subclasses can implement this method to define the outputs of the operator. By default, this method is called once when the operator is created. If the operator is dynamic, this method is called after the operator is executed. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** a [`fiftyone.operators.types.Property`](fiftyone.operators.types.md#fiftyone.operators.types.Property), or None #### resolve_placement(ctx) Returns the resolved placement of the operator. Subclasses can implement this method to define the placement of the operator. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** a [`fiftyone.operators.types.Placement`](fiftyone.operators.types.md#fiftyone.operators.types.Placement), or None #### resolve_run_name(ctx) Returns the resolved run name of the operator. Subclasses can implement this method to define the run name of the operator. * **Parameters:** **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **Returns:** a string, or None #### resolve_type(ctx, type) Returns the resolved input or output property. * **Parameters:** * **ctx** – the [`fiftyone.operators.executor.ExecutionContext`](fiftyone.operators.executor.md#fiftyone.operators.executor.ExecutionContext) * **type** – the type of property to resolve, either `"inputs"` or `"outputs"` * **Returns:** a [`fiftyone.operators.types.Property`](fiftyone.operators.types.md#fiftyone.operators.types.Property), or None #### *property* risk_level The effective risk level of the operator, which is used by guardrail systems of an agent to classify tool calls. #### to_json() Returns a JSON representation of the operator. * **Returns:** a JSON dict #### *property* uri The unique identifier of the operator: `plugin_name/operator_name`.