# DNA-Airflow-ETL

An attempt to move dna-etl-data-ingest-scheduled-service to Airflow. It's mostly done but never been properly tested and released.

## Installation

The following things should be installed and configured:
 - Docker
 - AWS CLI (with proper profiles) [AWS user guide](https://data-analytics.atlassian.net/wiki/spaces/IN/pages/218398747/AWS+user+guide)

## Local installation for development

The following steps should be done for development:
- Install Python 3.10 (or create a virtual environment)
- Install dependencies by running `pip install -r requirements.txt`

## Environment Variables

The following environment variables are required to run the project:

### Aggregates Database Connection

- `POSTGRES_USER`: The username for the PostgreSQL database connection.
- `POSTGRES_PASSWORD`: The password for the PostgreSQL database connection.
- `POSTGRES_HOST`: The host for the PostgreSQL database connection.
- `POSTGRES_PORT`: The port for the PostgreSQL database connection.
- `POSTGRES_DB`: The name of the PostgreSQL database.
- `POSTGRES_SCHEMA`: The name of the PostgreSQL schema.

### Snowflake Connection

- `SNOWFLAKE_USER`: The username for the Snowflake connection.
- `SNOWFLAKE_PASSWORD`: The password for the Snowflake connection.
- `SNOWFLAKE_ACCOUNT`: The Snowflake account name.
- `SNOWFLAKE_WAREHOUSE`: The Snowflake warehouse name.
- `SNOWFLAKE_DATABASE`: The name of the Snowflake database.
- `SNOWFLAKE_SCHEMA`: The name of the Snowflake schema.
- `SNOWFLAKE_HOST`: The Snowflake hostname.
- `SNOWFLAKE_ENV`: The Snowflake environment (e.g. DEV, PROD).

### ElasticSearch Connection

- `ELASTIC_SEARCH_URL`: The URL for the ElasticSearch connection.
- `ELASTIC_SEARCH_ENV`: The ElasticSearch environment (e.g. DEV, PROD).

### AWS S3

- `S3_ARTIFACTS_BUCKET`: The name of the S3 bucket to store artifacts.

### Atlas

- `ATALS_URL`: Atlas connection url
- `ATLAS_PAYLOAD`: Atlas json payload with `client_id`, `client_secret`, `grant_type` and `audience`

### CMS

- `CMS_URL`: CMS connection url

### SLACK

- `SLACK_TOKEN`: slack access token

Copy the `.env.sample` file and rename it to `.env`. Set the values for the environment variables in the `.env` file.

## Running the Project Locally

To run the project locally, run the following command:

```bash
./run_local.sh
```

This will run docker containers with the server on http://localhost:8080.
To access the server, use the following login credentials:

- Login: `airflow`
- Password: `airflow`

__NOTE: Usually it takes 1-2 minutes for scheduler to parse a new version of DAG after code change.__

## Local development

To run local server in debug mode use `./run_local.sh -m`

**cAdvisor** http://localhost:9091. It shows last 1.5 minutes metrics for all containers.

**Prometheus** http://localhost:9090. It records real-time metrics for all containers and airflow itself.

**Flower** http://localhost:5555. It shows airflow tasks and workers.

**Grafana** http://localhost:3000. It shows all metrics in pretty graphs.

## Cleanup local DB

Run `docker-compose down --volumes --remove-orphans`

## Deploying to DEV Environment

To run the project in the DEV environment, you need to set up AWS MWAA with Airflow version 2.4.3 and two small tier workers.

The easiest way to deploy your changes to DEV env is to run `./deploy2dev.sh`. This script will package your code and dependencies, and deploy it to the DEV environment.