###Description
This repository is responsible for the following Data warehouse tasks:
- Dimension table refresh from art_relation database
- Analytics raw files ingestion
- Monthly ETL for accounting sales data from art_relation database
- Daily ETL for analytics data
- Daily ETL Heat map data
- Daily ETL for social media data
- Scheduling of RED feeds
- Scheduling of analytics raw file ingestion jobs
- Generation of RED feeds (migration from talend to php in progress)
- Generation of Soundscan feeds (migration from talend to php in progress)
- Syncing data warehouse data between Amazon refshift cluster
- Data integrity checks on data warehouse
- Warming cache for analytics

###Code Documentation
https://github.com/theorchard/etl/wiki

###Workflow & UML
https://docs.google.com/a/theorchard.com/document/d/13T-Dve0AGg6-8qMpVRoaFMQkpVq53i9W_zcIOoiYb78/edit?usp=sharing

###Configuration
To set up the repo, you can do the following once:
1). Edit the .env file with your credentials, for example:
     AWSACCESSKEYID="MY@#@#$SECRET@^^^id"
     AWSSECRETACCESSKEY="MYSECRET!!!"
2). In command prompt, type: source .env


##Deprecated Directories and shell scripts
The following directories / shell scripts will be slowly migrated into php:
1). talend (We are going to move away from talend in order to gain more flexibile code maintainence.
2). sql (We are moving away from having a sql wrapper to run the sql for ETL. This is because we want to write functional test and unit test. These sqls will be in some kind of Entities classes)
3). AmazonPipeline (This contains json file for building amazon pipeline. Since we are adapting the messaging pattern, we are moving away from "one-big-pipeline-doing-all" way of ETL.)
4). deploy_xxx scripts (These are shell scripts that deploy talend jobs. Once we have deprecated all talend jobs, these scripts will be deprecated too)

The following directory will no longer be useful
1). Hadoop (there was an old research task to set up our own Hadoop cluster for ETL. But such project dies because we are using Amazon now)
2). ruby (there was a research task to see if ruby can be used in ETL process. But such project dies after we have achieved better performance with Java using multi-threading

##Java directory
A multi-threaded java program was built to get the processed_dig_sales data from art_relations.