plugins.panels.model_evaluation#
Module contents#
Model evaluation panel.
Classes:
|
- class plugins.panels.model_evaluation.EvaluationPanel(_builtin=False)#
Bases:
PanelAttributes:
The
OperatorConfigfor the operator.Whether the operator is builtin.
The unique identifier of the operator:
plugin_name/operator_name.Methods:
get_evaluation_id(dataset, eval_key)get_evaluation_type(config)get_permissions(ctx)can_evaluate(ctx)can_edit_note(ctx)can_edit_status(ctx)can_rename(ctx)can_delete_scenario(ctx)can_edit_scenario(ctx)on_load(ctx)get_confidences(per_class_metrics)get_avg_confidence(confidences)get_confidence_distribution(confidences)get_avg_iou(per_class_metrics)get_tp_fp_fn(info, results)get_map(results)get_mar(results)get_custom_metrics(results)set_status(ctx)set_note(ctx)rename_evaluation(ctx)delete_evaluation(ctx)load_evaluation_view(ctx)compute_avg_confs(results)compute_avg_ious(results)get_per_class_metrics(info, results)get_confusion_matrix_colorscale(matrix[, ...])get_confusion_matrices(results[, ...])get_correct_incorrect(results)get_scenario(ctx, scenario_id)load_scenarios(ctx)get_evaluation_data(ctx)load_evaluation(ctx)on_change_view(ctx)has_evaluation_results(dataset, eval_key)load_pending_evaluations(ctx[, skip_update])on_evaluate_model(ctx)load_view(ctx)get_subset_def_data(info, results, ...)get_scenario_data(ctx, scenario)get_subset_def_data_for_eval_key(ctx, scenario)Builds and returns an execution cache key for each type of scenario.
get_scenario_data_cacheable(ctx, scenario)validate_scenario_subsets(ctx, scenario)Validates the subsets in a scenario and compiles basic changes. 1. For view scenarios, check if the views exist in the dataset. 2. For sample_field and label_attribute scenarios, check if the field exists in the dataset. - If the field is tags, check if the tags exist.
load_scenario(ctx)Tries to load the scenario given its id and evaluation key If it fails, it shows an error page with options to edit/delete.
delete_scenario(ctx)render(ctx)Defines the panel's layout and events.
add_secrets(secrets)Adds secrets to the operator.
execute(ctx)Executes the operator.
method_to_uri(method_name)Converts a method name to a URI.
on_startup(ctx)resolve_delegation(ctx)Returns the resolved forced delegation flag.
Returns the resolved execution options.
resolve_input(ctx)Returns the resolved input property.
resolve_output(ctx)Returns the resolved output property.
resolve_placement(ctx)Returns the resolved placement of the operator.
resolve_run_name(ctx)Returns the resolved run name of the operator.
resolve_type(ctx, type)Returns the resolved input or output property.
to_json()Returns a JSON representation of the operator.
- property config#
The
OperatorConfigfor the operator.
- get_evaluation_id(dataset, eval_key)#
- get_evaluation_type(config)#
- get_permissions(ctx)#
- can_evaluate(ctx)#
- can_edit_note(ctx)#
- can_edit_status(ctx)#
- can_delete_evaluation(ctx)#
- can_rename(ctx)#
- can_delete_scenario(ctx)#
- can_edit_scenario(ctx)#
- on_load(ctx)#
- get_confidences(per_class_metrics)#
- get_avg_confidence(confidences)#
- get_confidence_distribution(confidences)#
- get_avg_iou(per_class_metrics)#
- get_tp_fp_fn(info, results)#
- get_map(results)#
- get_mar(results)#
- get_custom_metrics(results)#
- set_status(ctx)#
- set_note(ctx)#
- rename_evaluation(ctx)#
- delete_evaluation(ctx)#
- load_evaluation_view(ctx)#
- compute_avg_confs(results)#
- compute_avg_ious(results)#
- get_per_class_metrics(info, results)#
- get_confusion_matrix_colorscale(matrix, colorscale_name=None)#
- get_confusion_matrices(results, colorscale_name=None)#
- get_correct_incorrect(results)#
- get_scenario(ctx, scenario_id)#
- load_scenarios(ctx)#
- get_evaluation_data(ctx)#
- get_evaluation_data_cache_key_fn(ctx)#
- get_evaluation_data_cacheable(ctx)#
- load_evaluation(ctx)#
- on_change_view(ctx)#
- has_evaluation_results(dataset, eval_key)#
- load_pending_evaluations(ctx, skip_update=False)#
- on_evaluate_model_success(ctx)#
- on_evaluate_model(ctx)#
- load_view(ctx)#
- load_compare_evaluation_results(ctx)#
- get_subset_def_data(info, results, subset_def, is_compare)#
- get_scenario_data(ctx, scenario)#
- get_subset_def_data_for_eval_key(ctx, scenario)#
Builds and returns an execution cache key for each type of scenario.
- get_scenario_data_cacheable(ctx, scenario)#
- validate_scenario_subsets(ctx, scenario)#
Validates the subsets in a scenario and compiles basic changes. 1. For view scenarios, check if the views exist in the dataset. 2. For sample_field and label_attribute scenarios, check if the field exists in the dataset.
If the field is tags, check if the tags exist.
returns the validated scenario and a list of changes. If the scenario is not valid, it will return an empty list of subsets. If the scenario is valid, it will return the original scenario with the validated subsets.
- load_scenario(ctx)#
Tries to load the scenario given its id and evaluation key If it fails, it shows an error page with options to edit/delete.
- delete_scenario(ctx)#
- render(ctx)#
Defines the panel’s layout and events.
This method is called after every panel event is called (on load, button callback, context change event, etc).
- Parameters:
- Returns:
- add_secrets(secrets)#
Adds secrets to the operator.
- Parameters:
secrets – a list of secrets
- property builtin#
Whether the operator is builtin.
- execute(ctx)#
Executes the operator.
Subclasses must implement this method.
- Parameters:
- Returns:
JSON serializable data, or None
- method_to_uri(method_name)#
Converts a method name to a URI.
- Parameters:
method_name – the method name
- Returns:
a URI
- property name#
- on_startup(ctx)#
- resolve_delegation(ctx)#
Returns the resolved forced delegation flag.
Subclasses can implement this method to decide if delegated execution should be forced for the given operation.
- Parameters:
- Returns:
whether the operation should be delegated (True), run immediately (False), or None to defer to
resolve_execution_options()to specify the available options
- resolve_execution_options(ctx)#
Returns the resolved execution options.
Subclasses can implement this method to define the execution options available for the operation.
- Parameters:
- Returns:
a
fiftyone.operators.executor.ExecutionOptionsinstance
- resolve_input(ctx)#
Returns the resolved input property.
Subclasses can implement this method to define the inputs to the operator. This method should never be called directly. Instead use
resolve_type().By default, this method is called once when the operator is created. If the operator is dynamic, this method is called each time the input changes.
- Parameters:
- Returns:
a
fiftyone.operators.types.Property, or None
- resolve_output(ctx)#
Returns the resolved output property.
Subclasses can implement this method to define the outputs of the operator.
By default, this method is called once when the operator is created. If the operator is dynamic, this method is called after the operator is executed.
- Parameters:
- Returns:
a
fiftyone.operators.types.Property, or None
- resolve_placement(ctx)#
Returns the resolved placement of the operator.
Subclasses can implement this method to define the placement of the operator.
- Parameters:
- Returns:
a
fiftyone.operators.types.Placement, or None
- resolve_run_name(ctx)#
Returns the resolved run name of the operator.
Subclasses can implement this method to define the run name of the operator.
- Parameters:
- Returns:
a string, or None
- resolve_type(ctx, type)#
Returns the resolved input or output property.
- Parameters:
type – the type of property to resolve, either
"inputs"or"outputs"
- Returns:
a
fiftyone.operators.types.Property, or None
- to_json()#
Returns a JSON representation of the operator.
- Returns:
a JSON dict
- property uri#
The unique identifier of the operator:
plugin_name/operator_name.