# Kafka DB Deploy: database migrations with Kafka

This tool is used to run migration from Snowflake and execute statements in Neo4j

Usages:

1. Set Snowflake + Topic (will put data in given pre-terraformed topic)
2. Set Snowflake + Cypher (will put given SF data into Neo4J running given cypher)
3. Set Cypher (will kick off a Neo4J sink connector running given cypher - good for things that don’t need Snowflake input or long running)
4. Set Snowflake + RDS SQL (will put given SF data into RDS, only supports upserts ) **
5. Set Topic + Cypher (will put given topic data into Neo4J running given cypher - must give topic cluster) **
6. Set Topic + RDS SQL (will put given topic data into RDS, only supports upserts - must give topic cluster) **

** To be done


## Requirements

* Python 3.10

## Installation

Copy (and modify) the enviroment file:

```shell
cp .env.shadow .env
```

Run the `venv` make target to setup the python virtual enviromnent and install depdendencies:

```shell
make venv
```

OR

```shell
make venv_dev
```


## Running

Run directly using python by executing:

```shell
pipenv run python application.py < [input/file/path.xml]

$ awsume dev
$ source .env
$ python application.py < neo4j.xml
```

OR

Using Docker:
Change the input filename and FLOW_NAME to match the flow that you want to run.

```shell
awsume dev
docker build -t neo4j-kafka-db-deploy:dev .

docker run --rm -i \
    -e "Environment=${Environment}" \
    -e "FLOW_NAME=kafka" \
    -e "AWS_REGION=${AWS_REGION}" \
    -e "AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}" \
    -e "AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}" \
    -e "AWS_SECURITY_TOKEN=${AWS_SESSION_TOKEN}" neo4j-kafka-db-deploy:dev < kafka.xml
```


## Testing and Linting


```shell
make lint

make type_check

make test
```

OR

```shell
docker build -t kafka-db-deploy-test -f Dockerfile.test .

docker run --rm -i kafka-db-deploy-test python -m pytest tests/unit/ -o cache_dir=/tmp/pytest_cache

docker run --rm -i kafka-db-deploy-test python -m flake8 . --exclude ./venv

docker run --rm -i kafka-db-deploy-test python -m mypy --strict dbdeploy application.py
```

## Deploying
To deploy, after having new changes merged into the master branch, run the following Jenkins job that builds the image and pushes it to ECR:
[kafka-db-deploy](https://pipeline.theorchard.io/job/theorchard/job/kafka-db-deploy/job/master)