fiftyone.utils.aws#
Utilities for working with Amazon Web Services <https://aws.amazon.com>.
Functions:
|
Download files from a public AWS S3 bucket using unsigned URLs. |
- fiftyone.utils.aws.download_public_s3_files(urls, download_dir=None, num_workers=None, overwrite=True)#
Download files from a public AWS S3 bucket using unsigned URLs.
The url argument either accepts:
A list of paths to objects in the s3 bucket:
urls = ["s3://bucket_name/dir1/file1.ext", ...]
When urls is a list, then the download_dir argument is required and all objects will be downloaded into that directory
A dictionary mapping the paths of objects to files on disk to store each object:
urls = { "s3://bucket_name/dir1/file1.ext": "/path/to/local/file1.ext", ... }
- Parameters:
urls – either a list of URLs to objects in an s3 bucket, or a dict mapping these URLs to locations on disk. If urls is a list, then the download_dir argument is required
download_dir (None) – the directory to store all downloaded objects. This is only used if urls is a list
num_workers (None) – a suggested number of threads to use when downloading files
overwrite (True) – whether to overwrite existing files