image-asset-migration
=====================

A command line script to copy image objects from S3 bucket to S3 bucket. This was written to support the overall asset ingestion process project. The [tech design](https://docs.google.com/a/theorchard.com/document/d/1QrGunqWX6_WknUYWbjqpXHo_eVTAvsLHjVqubPQ0eaw/edit?usp=drive_web) explains how this bit fits into the project at large.

As a part of this migration, image filenames are renamed from a UPC based filename to a `product_id` based filename. Product ID's are found using an endpoint from `ows-product`.


## Getting Started

### Installation

The installation is very straightforward, but make sure you are using at least
python 3.4. If you use a Mac, you can install it using `brew`. If you use
Windows, just ask for a remote VM that has it installed for you.

```bash
$ cd image-asset-migration
$ pyvenv env
(env) $ . env/bin/activate
(env) $ pip install -r requirements.txt
(env) $ pip install -r requirements-dev.txt
```

Copy .env.shadow to .env and add the appropriate environement variables for your
environment.

```bash
$ cp .env.shadow .env
```

Several of the .env variables control which buckets and folders this script interacts with:

`SOURCE_BUCKET_NAME` = the name of the bucket to copy objects from

`SOURCE_SUB_FOLDER` = the folder within the source bucket to iterate through

`DESTINATION_BUCKET_NAME` = the name of the bucket to copy objects to

`DESTINATION_SUB_FOLDER` = the folder within the destination bucket


The AWS related variables can be obtained from the systems team.


### Running

When all dependencies have been installed, you can run this script on your local machine.
```bash
(env) $ python image_migration_script.py
```

For development purposes you can optionally set a max count of objects to copy on the command line.
```bash
(env) $ python image_migration_script.py 10
```

In development it is often necessary to clear out the destination bucket where you have been copying objects. There is a script for this purpose.
```bash
(env) $ python delete_destination_objects_script.py
```

### Logging

Not all images found in the source buckets will return a `product_id` from ows-product. When this occurrs the script continues on to the next item and logs the 404 response along with the UPC in 'ows-product-calls.log'.

### Linting

Please continue to lint any modifications using flake8
```bash
(env) $ flake8 image_migration_script.py, delete_destination_objects_script.py
```