# swf-snowflake-views

Introducing swf-snowflake-views, the coolest Python package at the Orchard used for building
materialized views in Snowflake!

This package can be used to make the queries in your application faster! This is accomplished
by splitting your queries into smaller subqueries, and computing the results of computationally-intensive
subqueries beforehand.

Not familiar with materialized views? Don't worry. Materialized views are similar to regular views,
except instead of simply storing references to queries against some data, they actually run those queries
and store the results as new independent database objects. The
[wikipedia article](https://en.wikipedia.org/wiki/Materialized_view) is a good starting resource.

This package is set up to build the views specified in the [config](snowflake_views/config.py).
To add a new view to the repository, simply add the relevant options to that file.

To learn more about the process of creating views (and swf-snowflake-views in particular),
please see the
[Demos 'N Drinks presentation](https://docs.google.com/presentation/d/1QSHmFxOxbOwSa36nWeicOqX06g80c_sbOqgH7dVdy_s/edit).

To learn how materialized views can be applied to a real-world problem, see the
[presentation](https://docs.google.com/presentation/d/1flBQJK0ZnPSIUGwyxZZnlNjKTEEMeOLMYaew9MkvHb4/edit?usp=sharing)
on using pre-aggregation to count listeners.

## Installation

The package can either be installed in a production or development environment.

### Production Environment

To install the package in production, first clone this repository in your current directory.

```sh
$ git clone git@github.com:theorchard/swf-snowflake-views.git
```

Then create a virtual environment `env` into which to install the package and activate it:

```sh
$ python -m venv env
$ source env/bin/activate
```

Next install the dependencies, then the package itself:

```sh
$ pip install -r requirements.txt
$ pip install .
```

Copy the environment shadow file `.env.shadow` to a different file (e.g., `.env`) and fill in
the various values corresponding to your AWS and Snowflake credentials.

```sh
export Environment=dev

export LOGGLY_URI=

export SENTRY_DSN=

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_REGION=us-east-1

export DEV_SWF_DOMAIN=

export SNOWFLAKE_ROLE=
export SNOWFLAKE_WAREHOUSE=
export SNOWFLAKE_DATABASE=
export SNOWFLAKE_SCHEMA=
export SNOWFLAKE_USER=
export SNOWFLAKE_PASSWORD=
export SNOWFLAKE_ACCOUNT=

export SNOWFLAKE_MICROSERVICE_WAREHOUSE=

export SF_CACHE_WARMUP_ATTEMPT_LIMIT=

export INGESTION_STATUS_TABLE=
```

To set them, run:

```sh
$ source .env
```

### Development Environment

The installation instructions are similar to those in production. First fork the repository and
clone the fork:

```sh
$ git clone <my_fork_uri>
```

Set up your virtual environment as in production, then install the package and dependencies:

```sh
$ pip install -r requirements-test.txt
$ pip install -r requirements.txt
$ pip install -e .
```

The additional requirements file `requirements-test.txt` contains dependencies for testing and linting.
Using the `-e` option when installing the package allows changes you make to the package to take effect
immediately, without having to reinstall.

## Create Views

Installing the package gives you access to the command-line utility `garcon`, which is used to
run the flow to create views:

```sh
$ garcon [decider | worker | exec]
```

To see how this works, in one terminal window, run `garcon decider` to set up a decider daemon to
handle activity orchestration for the workflow. Then in another terminal, run `garcon worker`,
which is used to actually process the activities. Finally, kick off the job with `garcon exec`.

To learn more about Garcon (a lightweight library for AWS SWF) on which swf-snowflake-views is based,
please see the [garcon repository](https://github.com/theorchard/ows-garcon).

## Tests

To run the tests, first make sure the test dependencies have been installed (i.e., you've run
`pip install -r requirements-test.txt`). Then run:

```sh
$ py.test tests
```

## Linting Checks

To run linting checks, make sure the test dependencies are installed and run:

```sh
$ flake8
```