# Data Processing Tasks

This is a service for running data processing tasks.

### Adding new tasks
1. Add new modules to `service/tasks` (see examples in `service/tasks/data_model.py`)
2. Add their endpoints and behavior to `service/tasks_controller.py`
3. Rebuild (see Deployment section of this README) and profit!

### Running tests
1. From parent dir, execute `bash data_processor/run_tests.sh`

Commands in this bash file are being also executed in CircleCI and described in `.circleci/config.yml`

To increase test coverage, add new tests in `data_processor/tests`

## Setting up locally

### Dependency management using [Poetry](https://python-poetry.org/docs/)

osx / linux / bashonwindows install instructions:

```shell
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -
```

Install dependencies:

```shell
poetry install
```

PS! Read Poetry documentation here: https://python-poetry.org/docs/

### Using docker

Use Makefile commands to start docker environment locally:

Start docker:

```shell
make up
```

See running processes:

```shell
make ps
```

Check docker logs:

```shell
make logs
```

Stop docker:

```shell
make down
```

## Setting up in production

We host container image in AWS Elastic Container Registry), and spin up the container service in
ECS (Elastic Container Service). To push images to ECR (and set up the service), you need to configure
your AWS access on your local machine. Either go edit `~/.aws/config` and `~/.aws/credentials` or
run `aws configure` if setting up AWS credentials the first time. You can retrieve AWS key/secret from
AWS console, under IAM (Identity & Access Management).

### Create and tag image
1. While in data_processor dir, build container with `docker-compose build`

2. Find the image name with `docker images` and tag the image (it should be `data_processor_data-processor`)
        ```
        docker tag data_processor_data-processor 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/data_processors
        ```
#### Push image to ECR (Elastic Container Registry)
1. Authenticate with ECR (make sure you have latest version of aws cli). It should say "Login succeeded"

    ```
    aws ecr get-login-password | docker login --username AWS --password-stdin 776891437216.dkr.ecr.eu-west-1.amazonaws.com
    ```

    If this worked for you, skip to step 2.

    If you have multiple AWS profiles configured locally, and have created on for fansifter, execute this instead:

    ```
    aws ecr get-login-password --profile fansifter | \
    sudo docker login --username AWS --password-stdin 776891437216.dkr.ecr.eu-west-1.amazonaws.com
    ```

    (notice that you can avoid using sudo with docker if create docker group and add your user there:

    ```
    sudo groupadd docker
    sudo gpasswd -a $USER docker
    ```

2. Push the container to ECR (`fansifter/data_processors` is the name of the pre-created container repository)
        ```
        docker push 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/data_processors
        ```

[AWS ECR CLI Official Docs](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html)

3. Build a new service, defining 3 environment variables for the container.
    TODO! this should be using CodePipeline to automatically deploy service when merged

## Database Migrations
Please check [Guide to FanSifter Database Migrations](migrations/README.md) for DB migration instructions.

## Code standards

This project uses:
* [isort](https://pycqa.github.io/isort/index.html) - to sort imports
* [black](https://black.readthedocs.io/en/stable/) - The uncompromising code formatter
* [flake8](https://flake8.pycqa.org/en/latest/) - Style Guide Enforcement
* [mypy](https://mypy.readthedocs.io/en/latest/index.html) - Static Type checker

Install Python Static Analysis Tools and Code formatters:

```shell
make install
```

Reformat and rearrange code:

```shell
make fmt
```

Run linters:

```shell
make lint
```

# Notes

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa_bitbucket



1.create venv, inherit global site packages; in Pycharm; install requirements;

2. Edit configucartions to create new launch config
 - c