# StatementDB to Snowflake PoC

This is a proof-of-concept showing how we can ingest sales data from StatementDB to Snowflake using an Airflow DAG to orchestrate the workflow and an ECS task to connect to the StatementDB ECS host via SSH and extract the sales data to S3.

The PoC is spinning up the AWS infrastructure locally using Localstack.

## Requirements

- Python 3.12
- Docker
- A Localstack Pro license 
- SSH access to the StatementDB EC2 instance
- MySQL access to StatementDB

## Setup

- Copy the `.env.shadow` file to a `.env` file in the root folder and fill in the values
- Do the same for the `.env.shadow` file at in the `ecs/tasks/extract-sales/` folder
- Run `make env` to create a virtual env and install the requirements

## Running

- Run `awsume <prod_profile>` to authenticate with the Orchard's AWS Prod account
- Run `make docker_login` to login to Docker
- Run `make setup_localstack` to create the infrastructure
- The Airflow UI will be available at: http://localhost.localstack.cloud:4510/ with "localstack" as both login and password