# Bulk copier

Tool for bulk copying from one s3 bucket to another

## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Usage(run script)

    usage: bulk-copy [-h] --bucket-source BUCKET_SOURCE --s3-keys S3_KEYS
                 --bucket-destination BUCKET_DESTINATION
                 [--copy-concurrency COPY_CONCURRENCY]
                 [--results-await-timeout RESULTS_AWAIT_TIMEOUT] [--debug]
                 [--humanize]

    
      -h, --help            show this help message and exit
      --bucket-source BUCKET_SOURCE
                            S3 bucket for source files
      --s3-keys S3_KEYS     Comma-separated paths to files in source bucket
      --bucket-destination BUCKET_DESTINATION
                            Destination S3 bucket for copied files
    optional arguments:
      --copy-concurrency COPY_CONCURRENCY
                            max number of worker for processing
      --results-await-timeout RESULTS_AWAIT_TIMEOUT
                            results_await_timeout for concurrent processing
      --debug               additional log level
      --humanize            human readable output (not in JSON)


## Examples

    bulk-copy --bucket-source="stage-sme-data-archive" \
    --s3-keys="amazonprime/activity/v1/report_date=2020-01-13/report_licensor=theorchard/ORCA_JP_20200113_Daily_Activity_Report.txt.zip" \
    --bucket-destination="dev-delphi-slz-download-jobs" \
    --debug

## run with docker
    make docker/image/build
    docker run -it -v ~/.aws/:/app/.aws/ bulk_copy_task:develop --bucket-source="stage-sme-data-archive" --s3-keys="amazonprime/activity/v1/report_date=2020-01-13/report_licensor=theorchard/ORCA_JP_20200113_Daily_Activity_Report.txt.zip" --bucket-destination="dev-delphi-slz-download-jobs" --debug