# swf-analytics-aggregation
ETL logic focused on pre-aggregating analytics data from DW.


### Version
1.0.0

## Installation

### Dependencies
- Python 3.4
- PostgreSQL

### Set up a virtual Python environment
For dev environment
```sh
$ git clone git@github.com:theorchard/swf-analytics-aggregation.git .
$ pyvenv ./env
$ python setup.py develop
```

For production environment  
**do NOT run `python setup.py install` for your dev environment**
```sh
$ git clone git@github.com:theorchard/swf-analytics-aggregation.git .
$ pyvenv ./env
$ source env/bin/activate
$ python setup.py install
$ pip install -r reqs.pip
$ pip install -r reqs-test.pip
```

### .env.shadow
.env.shadow is a template environment variable script that exports keys and values necessary for successful connectivity, paths, etc.  You will need to copy this file to `.env` and populate the necessary keys.

```sh
$ cp .env.shadow .env
```

### Environment variables description:

Variables which start with `STREAMS_VENDOR_` or `STREAMS_TRACK_` are responsible for appropriate DynamoDB tables read and write throughput during ETL execution.

### Running the script:
```sh
$ source env/bin/activate
$ source .env
$ garcon decider {flow_name}
$ garcon worker {flow_name}
$ garcon exec {flow_name}

*flow_name is the folder name under analytics_aggregation/flows/
```

## How it works
This project works on top of [Garcon](https://github.com/xethorn/garcon) and Amazon SWF.
 

## Folder structure
```
- conf (1)
- analytics_aggregation
  - flows (2)
    - sos_preaggregation
      - config.py (6)
    ...
  - tasks (3)
  - util (4)
- tests (5)
```

1. conf - Contains supervisord configuration files.
2. flows - Contains workflows. If you need to add a new workflow, create a package here and add the name of your package to the __all__ list in __init__.py.
3. tasks - Generic tasks that are being used among all workflows.
4. util - Generic utility methods being used among all workflows.
5. tests - Unit tests.
6. config.py - Flow specific configs.


## General Conventions
1. All file and folder names are lowercase with "_" as word separator.
2. Workflow specific tasks must reside in /analytics_aggregation/flows/{particular flow}/
3. Prevent having a context dictionary that is too large. Load configuration from config files within tasks if possible. This will simplify code and unit test a lot.
4. Only abstract out utility methods/module/class if it is a very generic task. Such task should be very small and accept very small number of input params.
5. Unless a compelling reason exists (ex. piping between process to avoid writing to disk), try and keep code in Python and use boto instead of subprocess commands calling aws. This makes it easier to unit test the code.
6. Workflows should begin using Python configs instead of yaml.

## Logging
Base flow class contains _on_exception_ method, which captures exceptions and sends them to Sentry.
Production logs can be viewed in loggly.

### Workflow Conventions
1. A workflow for an already loaded day should not achieve idempotency by backing out the data and loading it again (ex. to avoid unnecessary load on Redshift / avoid a lack of analytics when the workflow is rerunning).
2. (Nice to have) Partially run workflows should resume from where they left off. Optionally one can use the [Feed Status](README.md#feed-status)) module to handle this.

### Workflow Status
All workflows should update the status in DynamoDB for a particular day's run (or date range).


## Debugging Tools

`dev` package contains tools for debugging and developments.