# sme-analytics-ingestion-flow-control

This script reads delivery information from RDS and starts ingestion for as many deliveries as the system can handle at the same time. It starts from the oldest delivery that has not been ingested.

## Summary

The sme-analytics-flow-control is the kick-off script which invokes the step machine found at:

QA:

[qa-sme-analytics-ingester-sfn](https://console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:437795906767:stateMachine:qa-sme-analytics-ddex-ingester-sfn)
| [Terraform Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/lambda-ddex-ingester/sfn/sfn_sme_analytics_ddex_ingester.tf)
| [SFN Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/lambda-ddex-ingester/sfn/sfn-sme-analytics-ddex-ingester.json)

## Detail
### Logic testing  
Since the whole ddex ingestion execution logic is written in sql query [SELECT_ALL_N_ROWS_ORDERED](https://github.com/theorchard/lambda-ddex-ingester/blob/master/lambda/sme_analytics_ingestion_flow_control/constants/queries.py)
and this logic should be tested using mysql db.   

For these purposes [ddex_delivery_to_ingest_query_test](https://github.com/theorchard/database/pull/22955/files) table was created. 
New user password :  [AWS SECRET](https://us-east-1.console.aws.amazon.com/secretsmanager/secret?name=qa%2Fddex-ingester-sme-analytics-ingestion-flow-control%2FDDEX-INGESTER-TEST-PASSWORD&region=us-east-1) 
#### Testing query mock :
1) In order to test time related cases we need to mock mysql `now()`. In order to do these `now()` statement is replaced to `STR_TO_DATE(...)` statement : 
```
def mock_myql_db_time(self, current_time: datetime):
    format = "%Y-%m-%d %H:%M:%S"
    musql_format = "%Y-%m-%d %H:%i:%s"

    mock_dattime = f'STR_TO_DATE("{current_time.strftime(format)}", "{musql_format}")'
    queries.SELECT_ALL_N_ROWS_ORDERED = queries.SELECT_ALL_N_ROWS_ORDERED.replace(
        'now()', mock_dattime)
```

2)Also tests are executed on top of ddex_delivery_to_ingest_query_test table : 
```
    @classmethod
    def setUpClass(cls):
        queries.SELECT_ALL_N_ROWS_ORDERED = queries.SELECT_ALL_N_ROWS_ORDERED.replace(
            'ddex_ingester.ddex_delivery_to_ingest',
            'ddex_ingester.ddex_delivery_to_ingest_query_test')

        queries.UPDATE_ROW_STATUS = queries.UPDATE_ROW_STATUS.replace(
            'ddex_ingester.ddex_delivery_to_ingest',
            'ddex_ingester.ddex_delivery_to_ingest_query_test')
```



### Environment Vars
```
Environment=QA  # PROD
AWS_ACCESS_KEY_ID=<AWS_KEY>
AWS_SECRET_ACCESS_KEY=<AWS_SECRET>

# ########################################

# Environment
APPLICATION_NAME=sme-analytics-flow-control

# RDS
RDS_RW_USER=ddex-ingester-RW
RDS_DB_NAME=ddex-ingester
RDS_HOST=
RDS_PASSWORD=

# APP SPECIFIC --------------- #
BULK_ASSETS_INGESTER_SFN_ARN=arn:aws:states:us-east-1:437795906767:stateMachine:qa-sme-analytics-ddex-ingester-sfn
BULK_ASSETS_INGESTER_SFN_NAME=qa-sme-analytics-ddex-ingester-sfn
CONCURRENT_STATE_MACHINE_COUNT=3
```