# dim-refresh-etl
Scripts to execute dimension refreshes via Garcon/SWF Workflows

The overall tech design for this project can be found here: https://docs.google.com/document/d/1Fm9xZ8y995eNXySmQo-jAmKBznXJAx5Om74QyhIUpe0/edit

## Install
### Code
```
# clone repo
git clone git@github.com:theorchard/dim-refresh-etl.git
cd dim-refresh-etl

# create a python virtual environment to isolate from global packages
# (you could also pick you interpreter here)
pyvenv env
source env/bin/activate

# install project requirements
pip install -r reqs.pip
pip install -r reqs-test.pip   # if you intend to run tests [CI]

# install the project itself
pip install .            # using pip
python setup.py install  # using setuptools
python setup.py develop  # if you plan to develop and don't want to reinstall on certain modifications
```

### Setup Environment
The environment variables that need to be set for dim-refresh are defined in the .env.shadow

Update these to the appropriate values, copy the file to .env, and then run `source .env`

## Run SWF Processes Manually

Run the following as separate processes

```sh
# decider
dim-refresh decider
```

```sh
# activity worker
dim-refresh worker
```

```sh
# flow execution
dim-refresh exec <dimension>

ex.
dim-refresh exec artist
```
## Testing

```sh
py.test tests
```

#Flow types

* refresh
* geocoding
* [dynamo_sync](https://github.com/theorchard/dim-refresh-etl/blob/master/dim_refresh_etl/flows/dynamo_sync/README.md)