# Dynamo Sync ETL

The main goal of the ETL is synchronization of data to DynamoDB.
Right now the only data source is Snowflake, but potentially it can be extended by other databases and sources.

## Flow description

The ETL accepts `scope` param. Scope is a group of entities which will be synced during an execution.
For example `mobile` scope synchronizes results of these quires:

- [analytics_metadata](https://github.com/theorchard/dim-refresh-etl/blob/master/dim_refresh_etl/flows/dynamo_sync/queries/analytics_metadata.sql)

Technically the ETL consists of the following steps:

- bootstrap task
- data unload to S3
- copy data from S3 to DynamoDB

The ETL unloads data to S3 in JSON format. It allows to created nested object with dynamic structure and doesn't require additional transformation for uploading to DynamoDB.

In order to increase the speed of data uploading to DynamoDB the ETL uses multiprocessing and multithreading which can be resource-intensive.

Whole part of the flow which uploads data to DynamoDB is wrapped with a common error handler, which is bad practice in general but we use it in order to make sure that the flow doesn't fail with high capacity of DynamoDB table. All potential errors a logged to the Sentry and flow also raises an exception explicitly in the end of the flow.

## How to add new entities for synchronization

If you need to add a new logical synchronization please create the new scope value.
Then if you need to synchronize data from Snowflake the recommended way is define models for your scope in the [consts.py file](https://github.com/theorchard/dim-refresh-etl/blob/master/dim_refresh_etl/flows/dynamo_sync/consts.py) and then implement corresponding queries [queries directory](https://github.com/theorchard/dim-refresh-etl/tree/master/dim_refresh_etl/flows/dynamo_sync/queries) by adding a `.sql` file.(note that the name of a model in `consts` and name of the corresponding SQL file in `queries directory` must match).

## How to run SWF processes

Run the following as separate processes

```sh
# decider
dim-refresh dynamo_sync decider
```

```sh
# worker
dim-refresh dynamo_sync worker
```

```sh
# flow execution
dim-refresh dynamo_sync exec <scope>

ex.
dim-refresh dynamo_sync exec mobile
```