# TODO
- [ ] DAG name is taken from config.feed_name
- [ ] add config.PROVIDER and config.PROVIDER_URL
- [ ] save state to dynamodb
- [ ] trigger DBT cloud https://airflow.apache.org/docs/apache-airflow-providers-dbt-cloud/stable/index.html

# Minors
- [ ] rename BackfillDetectorBranchOperator to ThresholdBranchOperator
- [ ] BUG: aws_default doesn't work on MWAA


# General todo:
- [ ] implemnet MontyStateSensor (and use it for triggering DBT)
- [ ] unit test for DAG https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#unit-tests
- [ ] backfill

# Sometime in future (maybe)
- [ ] give a try for @task decorator
- [ ] var2

# Conventions
- in the DAG "tag" is 'flow_name' and 'owner' is PROVIDER for the data

# Features
- Unit tests for flow and code
- Uber Fast deployment
- Normal working QA environment
- Infrastructure saving
- Jinja SQL Tempalates (but Backward compatible with existing templates)
- Can utili|ze Pool for limiting of API limits (aka youttube API limit)
- Easier analysys and make re-ingestion from UI directly
- read from PROD write to dev
- chain geterogen serrvices in the commin pipeline

# Ideas
- mimc DBT dag and build models incremntally one-by-one
- dbt rollup only per month, not the whole period
- upgrade notification channel from SNS to Slack