# awal-ingestion-flow-control

This script reads delivery information from RDS and starts ingestion for as many deliveries as the system can handle at the same time. It starts from the oldest delivery that has not been ingested.

## Summary

The awal-ingestion-flow-control is the kick-off script which invokes the step machine found at:

QA:

[qa-awal-ingester-sfn](https://console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:437795906767:stateMachine:qa-awal-ingester-sfn)
| [Terraform Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/ddex-ingester/sfn_awal_ingester.tf)
| [SFN Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/ddex-ingester/sfn-awal-ingester.json)

Prod:

[prod-awal-ingester-sfn](https://console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:437795906767:stateMachine:prod-awal-ingester-sfn)
| [Terraform Definition](https://github.com/theorchard/terraform-infra/blob/master/prod/ddex-ingester/sfn_awal_ingester.tf)
| [SFN Definition](https://github.com/theorchard/terraform-infra/blob/master/prod/ddex-ingester/sfn-awal-ingester.json)

## Detail

### Environment Vars
```
Environment=QA  # PROD
AWS_ACCESS_KEY_ID=<AWS_KEY>
AWS_SECRET_ACCESS_KEY=<AWS_SECRET>

# ########################################

# Environment
APPLICATION_NAME=awal-ingestion-flow-control

# LOGGER_DSN is not needed for OWSLogger 2.x
LOGGER_DSN=
LOGGER_LEVEL='INFO'

# Sentry
export SENTRY_DSN=

# RDS
RDS_RW_USER=ddex-ingester-RW
RDS_DB_NAME=ddex-ingester
RDS_HOST=
RDS_PASSWORD=

# APP SPECIFIC --------------- #
BULK_ASSETS_INGESTER_SFN_ARN=arn:aws:states:us-east-1:437795906767:stateMachine:qa-bulk-assets-ingester-sfn
BULK_ASSETS_INGESTER_SFN_NAME=qa-bulk-assets-ingester-sfn
CONCURRENT_STATE_MACHINE_COUNT=3

# ------- RDS source table
SOURCE_TABLE_NAME=Smithsonian_ASSET_ALL

OUTPUT_S3_BUCKET=qa-bulk-assets-ingester
OUTPUT_S3_LOCATION=bulk-assets
```