# Migration tool

It contains a scripts for migration historical data

## Make commands
- `make venv` - create local virtual env(base.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make clean` - clean unnecessary files
- `make docker/login` - aws configuration need to be set first
- `make docker/build` - build Docker images from a Dockerfile
- `make docker/push` - push an image to a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Usage(run script)

    migration-tool --help
    usage: migration-tool [-h] migrator
    

## Implemented migrators

1. Migration of historical Spotify data from provided s3 bucket. Example:
    python -m migration_tool spotify_streams \
        --from-s3-bucket provider-api-export-sony \
        --from-s3-region us-east-1 \
        --from-s3-key-id KEY-ID \
        --from-s3-access-key SECRET-KEY \
        --to-s3 dev-sme-data-archive \
        --pool 1 \
        --limit 1

2. Migration of historical Youtube data. Example:
   ```
    python -m migration_tool youtube  \
        --from-date 2019-09-12 \
        --to-date 2019-09-12 \
        --pool 1 \
        --report-type daily_activity_summary \
        --report-version v1 \
        --report-licensor sme \
        --to-s3-bucket dev-sme-data-archive \
        --from-s3-bucket sme-ca-prod-partners \
        --from-s3-region us-east-1 \
        --from-s3-key-id KEY-ID \
        --from-s3-access-key SECRET-KEY
   ```

3. Migration of historical Amazon data. Example:
   ```
    python -m migration_tool amazon  \
        --licensor sme \  # or theorchard
        --from-file ./amazon-keys.txt \
        --to-s3-bucket dev-sme-data-archive \
        --from-s3-bucket sme-ca-prod-partners \
        --from-s3-region us-east-1 \
        --from-s3-key-id KEY-id \
        --from-s3-access-key SECRET-KEY \
        --limit 1
   ```

4. Amazon. Script to fix keys in s3, it lowercases report type. Example:
   ```
    python -m migration_tool amazon_rename  \
        --s3-bucket dev-sme-data-archive \
        --limit 1
   ```

5. Amazon (DLP-782). Script for removing incorrect migrated data (~20 Gb). Files are in bucket with incorrect keys like: sme-data-archive/amazonmusicunlimited/<b>20171206</b>/v1/report_date=2017-12-06/report_licensor=sme.
   Example:
   ```
    python -m migration_tool amazon_remove  \
            --s3-bucket dev-sme-data-archive \
            --limit 1
   ```

# Helpers

1. `rm` command to remove keys from s3 bucket.
   Example of command:
   ```
   python -m migration_tool rm \
            --path-to-keys keys.txt \  # or s3 path
            --s3-bucket dev-sme-data-archive \
            --limit 1
   ```
   Example of file with keys:
   ```
   youtubemusic/incorrect-report-1.txt
   youtubemusic/incorrect-report-2.txt
   ```