cloudalytics
============

Cloudalytics is an ETL process designed to sync Orchard Snowflake data to CloudSearch.

## Configuration

 - Copy `configs/log.shadow.yml` to `configs/log.yml` and populate with a Loggly token
 - Copy `configs/snowflake.shadow.yml` to `configs/showflake.yml` and populate with Snowflake connection data
 - Ensure that the CloudSearch endpoints listed in the `src/Orchard/Models/Redshift/` point to valid CloudSearch *document* endpoints.
 - For populating data from Snowflake adjust odbc.ini file which contains the following lines 
   (please adjust them accordingly to your settings and Snowflake role/schema/database):
   
   [Data Sources]
   testodbc = SnowflakeDSIIDriver
    
   [testodbc]
   Driver      = /./usr/lib64/snowflake/odbc/lib/libSnowflake.so
   Description = Snowflake ODBC
   uid         = USER_NAME
   server      = delphi.us-east-1.snowflakecomputing.com
   role        = DEV_ENGINEERING
   database    = DEV_ENGINEERING
   schema      = TEST_SHEMA
   warehouse   = DEV_OWS_WAREHOUSE
   
## Execution

The shell scripts run the PHP CLI script `scripts/populate_dim_data.php`
representing batched chunks of IDs.  Example:

```
php /var/www/html/scripts/populate_dim_data.php --model \\Orchard\\Models\\Snowflake\\Entity\\Release --from 1 --to 5000
php /var/www/html/scripts/populate_dim_data.php --model \\Orchard\\Models\\Snowflake\\Entity\\Release --from 5001 --to 8868
php /var/www/html/scripts/populate_dim_data.php --model \\Orchard\\Models\\Snowflake\\Entity\\Release --from 8869 --to 8869
php /var/www/html/scripts/populate_dim_data.php --model \\Orchard\\Models\\Snowflake\\Entity\\Artist --from 1 --to 5000
```

### populate_dim_data.php options

- **--model:** The fully namespaced classname to be loaded
- **--from:** The lowest ID (specified in the model) to process in this batch
- **--to:** The highest ID to process in this batch