cloudsearch-etl
===============

CloudSearch ETL is an Extract-Transform-Load process that moves data from the art_relations
database to multiple Amazon CloudSearch instances to create an up-to-date searchable index.

**NOTE:** These are the basic setup instructions.  For more complete diagrams and
deployment instructions, view the wiki:
https://github.com/talskyorchard/cloudsearch-etl/wiki

## Installation Instructions

1. Clone the repo locally
2. Run `composer install`
3. Create the config file: `cp config/cliConfig.yml.shadow config/cliConfig.yml`
4. Create the document config file for all or specific cloudsearch instances:
  1. Each instance only uses the config file containing its name (ex. "test" instance would need config/testConfig.yml)
  2. Copy the appropriate config: `cp config/testConfig.yml.shadow config/testConfig.yml`
5. Set all values in config

## Codeception Setup

1. Create the `tests/_log/` and `tests/_data` directory and ensure it's writeable
2. Make sure your CLI date.timezone is explicitly set in php.ini
3. On Linux / OS X: Run the test suite with:
  1. Functional tests: `./vendor/bin/codecept run -c tests/codeception.yml functional`
  2. Unit tests: `./vendor/bin/codecept run -c tests/codeception.yml unit`

## Amazon AWS Services Setup

Cloudsearch-etl makes use of SQS for job queueing and S3 for storing the actual
cloudsearch documents until they're inserted into CloudSearch.

### Cloudsearch Setup

Each instance needs to be set up as a CloudSearch domain.  There needs to be one
CloudSearch domain per instance config in config/<instanceName>Config.yml.

Fields should be set up one-to-one with the fields in the document portion of the 
instance config:
```yaml
fields:
release_id:
  sql    : releases.release_id
  type   : int
display_upc:
  sql    : releases.display_upc
  type   : text
```

It's important for the field types in config to match the field types in CloudSearch.
Currently there is an additional spreadsheet with all of the relevant settings for
the releases CloudSearch configuration.

### SQS Setup

Queues are automatically created, but need to be configured with a Dead Letter Queue policy
to keep messages from being reprocessed eternally.  Queues like the following:
```YAML
update-queue:
  adapter: SQS
  queue-name: prod-cloudsearch-etl-releases-update-q

document-queue:
  adapter: SQS
  queue-name: prod-cloudsearch-etl-releases-document-q
```

For each instance's update and document queue, a deadletter queue needs to be created, like:
```
prod-cloudsearch-etl-releases-update-deadletter
prod-cloudsearch-etl-releases-document-deadletter
```

Then, in the AWS console, configure each queue to:
 - Use Redrive Policy: x
 - Dead Letter Queue: Dead letter queue name
 - Maximum Recieves: 5


### S3 Setup

Cloudsearch-etl stores its documents in S3.  The S3 bucket name referred to in cliConfig.yml:
```YAML
document-s3-bucket: prod-cloudsearch-etl-documents
```
is not automatically created.  It needs to be created in the AWS console.

As a performance optimization, the ETL code does not delete document objects from S3 after processing them.
In order to prevent processed documents from lingering in S3, it is necessary to set up an auto-expiration
rule on the S3 bucket. The rule needs to be defined to permanently delete objects in the bucket after X number
of days. The specified number of days should be at least as long as the SQS retention period for the document queue
in order to ensure that S3 does not expire a document that is still pending processing in the queue.

## Automated Process Setup

The files in the bin folder need to be set as executable.


### Cron Setup

Cron runs the `bin/seed-full-dataset.php` process that regularly refreshes the 
entire dataset.  To run the script every four days, add a crontab line like this:

```
* * 4 * * bin/seed-full-dataset.php --environment=prod --cloudsearch-instance=releases 2>&1 | logger -p user.error
```

### Supervisord Setup

Supervisord runs the `bin/assemble-cloudsearch-documents.php` and 
`bin/cloudsearch-document-upload.php` processes in parallel and keeps
them alive.

Supervisord will include any config files that are found under the /etc/supervisord.d directory. The supervisord
configuration for `bin/assemble-cloudsearch-documents.php` and `bin/cloudsearch-document-upload.php` needs to be
set up within that directory.

1. Copy the `config/supervisord.conf.shadow` file to a temporary location before editing it.

        $ cp config/supervisord.conf.shadow /tmp/cloudsearch-etl.conf
2. Edit the temporary file, replacing placeholders with the appropriate values.
3. Move the temporary file into the /etc/supervisord.d directory.

        $ sudo mv /tmp/cloudsearch-etl.conf /etc/supervisord.d/cloudsearch-etl.conf