# S3 parquet transformer

A Job for copying parquet reports from source S3 bucket to destination with path transformation.

## Make commands

- `make venv` - create local virtual env(base.pip dependencies)
- `make testenv` - create local virtual env(test.pip dependencies)
- `make devenv` - create local virtual env(dev.pip dependencies)
- `make test` - run tests
- `make lint` - run pylint check
- `make build` - build package
- `make upload` - upload package on PyPI
- `make clean` - delete unnecessary files

- `make docker/login`  - login to AWS docker, aws configuration need to be set first
- `make docker/test/image` - pull image used for tests execution
- `make docker/test` - run tests in docker
- `make docker/lint` - run pylint check in docker
- `make docker/build` - build package in docker
- `make docker/upload` - upload package on PyPI from docker

- `make docker/image/build` - build Docker image
- `make docker/image/push` - push an image to a registry
- `make docker/image/pull` - pull an image from a registry

(when run `make test`, `make lint`  command, all dependencies from requirements/test will be installed.
When run `make venv` base.pip dependencies will be installed)

### configure AWS:
- to check if aws configuration was already set: `aws configure list`
- to configure `https://docs.aws.amazon.com/en_us/cli/latest/userguide/cli-chap-configure.html`

## Params:

	Required:
    --dsp        dsp             Source DSP
    --date       2019-04-01      Report Date
    --type       type            Report Type
    --env        dev             Environment


## Examples

    s3-parquet-transformer --env=dev --dsp=apple --date=2019-12-01 --type=content

    s3-parquet-transformer --env=dev --dsp=spotify --date=2019-12-01 --type=tracks

## Run in batch:

    python3 s3_parquet_transformer/run_batch.py

Or with params:

    python3 s3_parquet_transformer/run_batch.py --env=dev --dsp=spotify --type=tracks --start_date=2020-01-01 --end_date=2020-05-31