# ows-dataexport

Large Data Export [TECHWEEK]

This exciting project aims to provide the capability for large dataset exports.

Tech schema: https://whimsical.com/large-query-file-generation-ibolshakov-TEVnJ7nraR8zM9mcagS6yJ

Kafka: It aims to leverage our existing Kafka infrastruture to emit completion messages, after writing the file to S3, thus overcoming the need for a proxy/gateway in the first approach

Use cases that will benefit include, but are not limited to

1. Full Insights My Catalog export with configurable number of metrics. The user can decide to export 5 or 30 columns

2. Hassle free report generation for collaborators

3. Hassle free report generation for royalties

## Requirements

* Python 3.8

## Installation

Copy (and modify) the enviroment file:

```shell
cp .env.shadow .env
```

Run the `venv` make target to setup the python virtual enviromnent and install depdendencies:

```shell
make venv
```

Alternatively you can run all the steps manually:

```shell
python -m venv venv
source venv/bin/activate
(venv) $ pip install --upgrade pip
(venv) $ pip install -r requirements.txt
(venv) $ pip install -r requirements-dev.txt
```

Another alternative is to use `pipenv` to install all dependencies and create the environment at the same time.

```shell
pipenv sync
```

Finally, setup the pre-commit hooks that will lint and format your code before you commit a change:

```shell
pre-commit install
```

## Running

First of all you need to export environment variables by running:
```shell
source .env
```

if you are using `pipenv` the `.env` file will be exported automatically.

When all dependencies have been installed, you can run the flask application on your local instance by running:

```shell
make dev
```

or

```shell
(venv) $ python dev.py
```

All make commands will automatically install all dependencies in an virtual environment folder called `venv`. You can activate a virtual environment by running `. venv/bin/activate` or `source venv/bin/activate`.

By using the development server, you will have access to specific features that are not necessarily available in production, such as the exception tracer.

## Testing

The `lint` make target is used for linting the code, `type_check` to verify type hints, and `test` for running the unit tests:

```shell
make lint

make type_check

make test
```

To run the commands without Make you can use the following commands to lint, check types, and run the tests respectively:

```shell
(env) $ flake8 dataexport/ tests/ dev.py application.py

(env) $ mypy --strict dataexport

(env) $ py.test tests/unit/
```

## Notes

Windows users will not be able to use the make commands as Make is a Unix util. Windows users can attempt to install GNUWin to get this functionality.

### Building Docker Image
You may want to debug or verify the docker image build process is working, here's how.

:warning: Don't use this process for feature development.

1. Make sure you've got [awscli v2](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed
```
$ aws --version
```

2. Download the [aws mfa enabled credentials generator script](https://github.com/theorchard/collab/blob/master/jcarrion/aws-creds-generator/generate.sh), and then navigate to directory containing script

3. Run the credentials generated script, selecting **dev** and inputting your mfa code
```bash
$ ./generate.sh
```

4. Set shell to use newly generated credentials
```bash
$ AWS_PROFILE=default
```

5. Set **dev** aws ecr credentials for your local docker install
```bash
$ aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 103233932089.dkr.ecr.us-east-1.amazonaws.com
```

6. Build image
```bash
$ docker build --build-arg AWS_ACCOUNT=103233932089 -t ows-data-export:dev .
```

7. Run most recently built image, mapping 8080 in container to 5000 on host
```bash
$ docker run -p 5000:8080 -e PORT=8080 ows-data-export:dev
```

8. Test application
```bash
$ curl --request GET --url http://localhost:5000/hello/
```