# DNA ETL Data Ingest Scheduled Service

This project provides a scheduled service for ingesting artist and track data from Snowflake DB, transforming it, and storing the results in Postgres DB and ElasticSearch. Intermediate results are stored in an S3 bucket.

## External requirements
 - Requires data in DELPHI_EXPLORATION.CHARTMETRIC Snowflake DB
 - Requires data in WHITELIST_REPLICA.MAIN Snowflake DB
 - Depends on `dna-cms` it should be up and running beforehand.
 - Requires Postgres DB to store search artifacts.
 - Requires ElasticSearch to store search index.
 - Requires S3 bucket to store intermediate results.

## Installation

The following things should be installed and configured:

- Install Python 3.9 (or create a virtual environment)
- Install dependencies by running `pip install -r requirements/dev.pip`
- AWS CLI (with proper profiles) [AWS user guide](https://data-analytics.atlassian.net/wiki/spaces/IN/pages/218398747/AWS+user+guide)

NOTE: Project uses extra pip index defined in `pip.conf`

## Environment Variables

This project uses environment variables to store sensitive configuration information. These variables are stored in a `.env` file. A `.env.sample` file is provided to show the expected format of this file.

## Running the Project

To run the project locally, you will need to assume the specified AWS role and provide the MFA key. To do this, run the following command:
```bash
./assumed_role_as_default.sh <aws-role-name> <mfa-key>
```

Once you have assumed the AWS role, you can run the project by running the following command:
```bash
python src/app.py
```

## Contributing

Before committing changes, please ensure that you run the pre-commit hooks by running the following command:
```bash
make pre-commit
```
This will ensure that your changes meet the project's style and formatting guidelines.