# Image Deduplication with FiftyOne This recipe demonstrates a simple use case of using FiftyOne to detect and remove duplicate images from your dataset. ## Setup If you haven’t already, install FiftyOne: This notebook also requires the `tensorflow` package: ## Download the data First we download the dataset to disk. The dataset is a 1000 sample subset of CIFAR-100, a dataset of 32x32 pixel images with one of 100 different classification labels such as `apple`, `bicycle`, `porcupine`, etc. You can use this [helper script](https://raw.githubusercontent.com/voxel51/fiftyone/main/docs/source/recipes/image_deduplication_helpers.py). The above script uses `tensorflow.keras.datasets` to download the dataset, so you must have [TensorFlow installed](https://www.tensorflow.org/install). The dataset is organized on disk as follows: ```none /tmp/fiftyone/ └── cifar100_with_duplicates/ ├── / │ ├── .jpg │ ├── .jpg │ └── ... ├── / │ ├── .jpg │ ├── .jpg │ └── ... └── ... ``` As we will soon come to discover, some of these samples are duplicates and we have no clue which they are! ## Create a dataset Let’s start by importing the FiftyOne library: Let’s use a utililty method provided by FiftyOne to load the image classification dataset from disk: ## Explore the dataset We can poke around in the dataset: Create a view that contains only samples whose ground truth label is `mountain`: Create a view with samples sorted by their ground truth labels in reverse alphabetical order: ## Visualize the dataset Start browsing the dataset: Narrow your scope to 10 random samples: Click on some some samples in the App to select them and access their IDs from code! Create a view that contains your currently selected samples: Update the App to only show your selected samples: ## Compute file hashes Iterate over the samples and compute their file hashes: We have two ways to visualize this new information. First, you can view the sample from your Terminal: Or you can refresh the App and toggle on the new `file_hash` field: ## Check for duplicates Now let’s use a simple Python statement to locate the duplicate files in the dataset, i.e., those with the same file hashses: Now let’s create a view that contains only the samples with these duplicate file hashes: Of course, we can always use the App to visualize our work! ## Delete duplicates Now let’s delete the duplicate samples from the dataset using our `dup_view` to restrict our attention to known duplicates: ## Export the deduplicated dataset Finally, let’s export a fresh copy of our now-duplicate-free dataset: Check out the contents of `/tmp/fiftyone/image-deduplication` on disk to see how the data is organized. You can load the deduplicated dataset that you exported back into FiftyOne at any time as follows: ## Cleanup You can cleanup the files generated by this recipe by running: