# Looker Analytics Reprocessing

This repo allows a user to remove duplicated data from and load missing data into
the Looker Analytics tables.

## Background
Data is inserted into the Looker Analytics tables daily. The `INSERT` statement
that loads data into these tables simply adds anything from `facts.prod.fact_analytics`
that has a processeddaytime greater than the `max(processeddaytime)` in 
the Looker Analytics tables

Issues may occur with this process where data is reprocessed upstream, i.e.
a store delivers data again for a date after we have already loaded data
once for that date. In `fact_analytics`, data for a date that 
is being reprocessed is automatically. However, the Looker Analytics tables
do not get updated. This repo identifies those inconsistencies and resolves them.


## To Set Up

* Run the commands below

```bash
$ python -m venv env
$ source env/bin/activate
$ pip install -r requirements.txt
```
Update the `config.py` file with your credentials


## Run

* Run the command:

`python looker_analytics_reprocessing.py`

Running `looker_analytics_reprocessing.py` takes the following steps:
    1. identify dates/stores that are either incorrectly duplicated
        or missing from the Looker Analytics tables
    2. creates log of dates/stores that are to be deleted or loaded
    3. generate sql to delete and or load incorrect data
    4. execute delete and or load

You will be presented with dates/stores that will be deleted/loaded and
you will need to confirm the delete/load by pressing `y`.

A full list of dates/stores that are deleted/loaded will be added to 
the directory called `change_log`. You can also review the change here 
prior to confirming the delete/load, just locate the file with most recent datestamp