# airflow-feed-ingestion

## Local run

Local run of the dag is based on https://github.com/aws/aws-mwaa-local-runner

This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.

*Please note: MWAA/AWS/DAG/Plugin issues should be raised through AWS Support or the Airflow Slack #airflow-aws channel.  Issues here should be focused on this local-runner repository.*


### Prerequisites

- **macOS**: [Install Docker Desktop](https://docs.docker.com/desktop/).
- **Linux/Ubuntu**: [Install Docker Compose](https://docs.docker.com/compose/install/) and [Install Docker Engine](https://docs.docker.com/engine/install/).
- **Windows**: Windows Subsystem for Linux (WSL) to run the bash based command `mwaa-local-env`. Please follow [Windows Subsystem for Linux Installation (WSL)](https://docs.docker.com/docker-for-windows/wsl/) and [Using Docker in WSL 2](https://code.visualstudio.com/blogs/2020/03/02/docker-in-wsl2), to get started.

### Install

```bash
make install
```

### First Run

```bash
make start
```
* wait for ~1 minute
* Go to http://localhost:8080/
* Login: admin/test
* Head to Donnections http://localhost:8080/connection/list/
  * create Snowflake connection 
    * id: "snowflake_default"
    * connection type: "Snowflake"
    * Schema: <your dev schema>
    * login: <your login>
    * password: <your password>
    * Extra: <you may put your private key here>
    * Account: "orchard"
    * Database: "dev_engineering"
  * update "aws_default": 
    * AWS Access Key ID: your dev key 
    * AWS Secret Access Key: your dev secret
    * Extra: `{"region_name": "us-east-1"}` 
  * optionally create "aws_prod" if you plan to use it

#### Add requirements

1. Add Python dependencies to `requirements/requirements.txt`.  
2. To test a requirements.txt without running Apache Airflow, use the following script:

```bash
./mwaa-local-env test-requirements
```