# lambda-bulk-asset-ingester
This repo contains a lambda orcehstration which allows usess to send asset files to ows-assets from an S3 bucket via
GraphQL queries. The metadata for the assets must already be ingested into the Orchard database. At the time of
development this is performed via the [`bulk-upload-feeder`](https://github.com/theorchard/bulk-upload-feeder),
but metadata could be ingested via other means.

## Dockerized Lambda project
This is an early attempt at using
[Container Image Support](https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/)
for AWS Lambda functions. All development occuring locally should be done within containers. The initial implementation
contained in this repository was derived from
[`lambda-asset-fingerprinting`](https://github.com/theorchard/lambda-asset-fingerprinting) and
[`lambda-abacus`](https://github.com/theorchard/lambda-abacus/)

## Orchestration Diagram
![bulk-asset-ingester Orchestration Diagram](./docs/images/diagram_2021-05-27.png)

More info in [State Machine Diagram](#state-machine-diagram) below.

## Logging
Successful operations at the end of the orchestration are logged to:

`INTEGRATION.<env>.CATALOG_INGESTION`
and
`INTEGRATION.<env>.CATALOG_INGESTION_ACTION`

Example join query to get logging output:

`<fully_qualified.source_table>` is the source table used to generate the input file.

`execution_name_1` is set to the name of the sfn execution.

```
-- ----------- Bulk-Assets-Ingester Catalog Logging - QA

-- Name of State Machine invocation:
SET execution_name_1 ='BULK_<FILE_NAME>-<SHORTUUID>';


-- Join across tables
SELECT ci.STATE_MACHINE_EXECUTION_NAME,
	   RESULT,
	   ci.TIMESTAMP AS "START_TIME",
	   cact."TIMESTAMP" AS "END_TIME",
	   TIMESTAMPDIFF('min', ci.TIMESTAMP, cact."TIMESTAMP") AS MINS_UNTIL_INGESTED,
	   FOREIGN_REL_ID,
	   ORCH_UPC,
	   TRACK,
	   PROJECT_ID,
	   ORCH_RELEASE_ID,
	   vendor_id,
	   MESSAGE
FROM
	<fully_qualified.source_table> aca
LEFT JOIN INTEGRATION.QA.CATALOG_INGESTION_ACTION cact
	ON orch_upc = cact.upc
	AND aca.TRACK = ZEROIFNULL(TRACK_SEQUENCE_NUMBER)
	AND aca.VOLUME = ZEROIFNULL(TRACK_VOLUME_NUMBER)
LEFT JOIN INTEGRATION.QA.CATALOG_INGESTION ci
	USING (STATE_MACHINE_EXECUTION_NAME)
WHERE
	cact.STATE_MACHINE_EXECUTION_NAME in ($execution_name_1)
ORDER BY END_TIME DESC;


-- Edge case fails
SELECT *
  FROM INTEGRATION.QA.CATALOG_INGESTION_ACTION cact
  WHERE cact.STATE_MACHINE_EXECUTION_NAME
    IN ($execution_name_1)
  AND RESULT <> 'Success'
;


-- Sanity check sfn is present
SELECT *
  FROM INTEGRATION.QA.CATALOG_INGESTION
  WHERE STATE_MACHINE_EXECUTION_NAME
    IN ($execution_name_1)
;
```

## Incomplete Elements

### Invocation Script on Scheduler - [WIP here](https://github.com/theorchard/bulk-asset-uploader)
A script which dumps the contents of an asset ingestion table to a JSON payload for the coordinating SFN.

Currently, the invocation of the orchestration can be achieved with SQL similar to the following ***PROVIDED YOU ARE A
MEMBER OF TH E `<env>-bulk-assets-ingester-team` IAM GROUP***:

```
-- Bulk-Asset Trigger JSON Drop

-- Drop file into trigger bucket
COPY INTO 's3://qa-bulk-assets-ingester/bulk-assets/TEST_INGESTION_RUN_TRIGGER_FILE-2021-04-23'
FROM (
	SELECT array_agg(object_construct(*)) WITHIN GROUP(ORDER BY ORCH_UPC asc, "VOLUME" asc, TRACK asc)
	FROM (
		SELECT *
		FROM <fully_qualified.source_table>
		ORDER BY ORCH_UPC ASC
		LIMIT 3
		OFFSET 0
	))
CREDENTIALS = (
    aws_key_id='<PERSONAL_AWS_KEY>'
    aws_secret_key='<PERSONAL_AWS_SECRET>'
  )
FILE_FORMAT = (
 	TYPE=JSON
 	COMPRESSION=NONE
 )
;
```

`<fully_qualified.source_table>` is the source table used to generate the input file, as defined below:
```
-- Input table DDL

create or replace TABLE INPUT_TABLE_NAME (
	FOREIGN_REL_ID NUMBER(20,0),  -- a provided original unique key -- can be upc.
	VOLUME NUMBER(1,0),
	TRACK NUMBER(1,0),
	FILE_NAME VARCHAR(16777216), -- S3 filename without path, bucket
	PATH VARCHAR(16777216), -- S3 path without filename, bucket
	UPLOAD_DATE DATE,
	ORCH_UPC NUMBER(20,0),
	ORCH_RELEASE_ID NUMBER(38,0),
	ORCH_PROJECT_CODE VARCHAR(256),
	VENDOR_ID NUMBER(38,0),
	SESSION_ID VARCHAR(1),  -- arbitrary
	ASSET_TYPE VARCHAR(5),  -- enum ('cover'/'audio')
	FILE_SIZE NUMBER(38,0),  -- optional data
	S3_BUCKET VARCHAR(13) -- bucket name alone
);
```


## Lambda Details
The following section describes the purpose of each contained lambda, with inputs and outputs. The flow of an asset
should follow the order of the lambdas from top to bottom.

N.B. The first lambda -- `trigger-state-machine` -- executes outside of the SFN lambda flow, as it triggers the
execution of each state machine.

### [`trigger-sfn`](lambda/trigger_sfn) (SFN Invocation)
This lambda is used to trigger the state machine with a pre-determined execution name. Although a state machine can be
triggered directly via EventBridge (a CloudWatch event), when done as such, the execution name is auto-generated. By
intercepting the CoudWatch event via this lambda, we can exert some control over the invocation and execution of the
state machine. The input provided to this lambda is generated in practice by executing the
[`bulk-uploader`](bulk_uploader) script on scheduler, which places a file in an S3 bucket.

#### example input
The input will be a rather robust payload from a CloudWatch event. The relevant portion is detailed below.
```
{
  .
  .
  .
  "detail-type": "AWS API Call via CloudTrail",
  "detail": {
    "requestParameters": {
      "bucketName": "qa-bulk-assets-ingester",
      "key": "bulk-assets/ambition_test_set_1_0_0_0.json"
    }
  }
  .
  .
  .
}
```
or the new event type:
```
{
  .
  .
  .
  "detail-type": "Object Created",
  "detail": {
    "bucket": {
      "name": "qa-bulk-assets-ingester"
    },
    "object": {
        "key": "bulk-assets/ambition_test_set_1_0_0_0.json"
    }
  }
  .
  .
  .
}
```

#### example output
In the below example, `<execution_name>` would be something like:
`BULK_ASSET_COVER_ALL_2021-04-16_1_0_0_0-9423da68-1528-4d53-9f54-72918c9ae018`, where `ALL_2021-04-16_1_0_0_0` is
the name of the source file (which itself is named after the generating source table), followed by a shortuuid
generated at the lambda's runtime. Additionally, a correlation_id is generated here, and passed through the entire
orchestration, and all subsequent external service calls (i.e. GraphQL, Datadog, etc.)
```
arn:aws:states:us-east-1:437795906767:execution:qa-bulk-assets-ingester-sfn:<execution_name>
```
----
The following lambdas execute within each state machine invocation.

### [`parse-bulk-assets`](lambda/parse_bulk_assets)
This lambda receives the location of an JSON file from an S3 `putObject` invocation, parses the number of items, and
creates an index of items. This pattern circumvents the input payload size limitations for AWS Step Functions
executions.
[See here](https://docs.aws.amazon.com/step-functions/latest/dg/limits.html#service-limits-state-machine-executions).
This index list, along with the key and bucket of the input file is passed to a map within the
[bulk-asset-ingester sfn](https://github.com/lealvona/terraform-infra/blob/INT-884_Add_lambda-bulk-asset-ingester/qa/bulk-assets-ingester/sfn-bulk-assets-ingester.json#L38)
which in turn calls [`lookup-orchard-product`](lambda/pre_process_artwork).

#### example input

```
{
    "bucket": "bucket-name",
    "key": "lambda-bulk-asset-ingester/example_table_name_drop_file_0_0_0.json",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```
#### curl (docker)
```
curl --location \
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
            "bucket": "bucket-name",
            "key": "lambda-bulk-asset-ingester/asset_ingestion_rows_0_0_0.json",
            "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
            "state_machine_name": "<env>-bulk-assets-ingester-sfn",
            "correlation_id": "<uuid4>"
        }'
```

#### example output

```
{
    "key": "lambda-bulk-asset-ingester/asset_ingestion_rows_0_0_0.json"
    "bucket": "bucket-name",
    "items": [0, 1, 2, 3, 4, 5],
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`lookup-product`](lambda/lookup_product)
This lambda receives the bucket and key of the input JSON document, and an associated index from the mapping task which
invokes it. It then finds the item within the input JSON document matching the index, and queries `ows-product`
via GraphQL to grab the relevant metadata. (It currently uses `upc` as the key for the GraphQL query, but can easily
be extended to use `product_id` to protect against edge cases where only the upc is problematic or not provided.) This
metadata is passed along to the rest of the orchestration.

#### example input

```
{
    "bucket": "bucket-name",
    "key": "lambda-bulk-asset-ingester/asset_ingestion_rows_0_0_0.json"
    "item_index": 1,
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```
#### curl (docker)
```
curl --location \
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
       "bucket": "bucket-name",
       "key": "lambda-bulk-asset-ingester/asset_ingestion_rows_0_0_0.json"
       "item_index": 1,
       "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
       "state_machine_name": "<env>-bulk-assets-ingester-sfn",
       "correlation_id": "<uuid4>"
     }'

```

#### example output -- Artwork

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Beware the Wolf",
        "artwork": {
            "bucket": "source-asset-drop",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": null
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### example output -- Audio

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Beware the Wolf",
        "track": {
            "bucket": "source-asset-drop",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": null
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        },
        "vendor_id": 9808
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`pre-process-art`](lambda/pre_process_art)
This lambda receives the metadata and location for a single artwork asset. It checks the artwork for basic validity
across the following metrics:

* aspect ratio
* minimum resolution
* color mode

If the artwork asset does not meet the minimum requirements for delivery to `ows-assets`, it attempts to repair the
image before passing the payload to the next lambda in the orchestration, [`handle-artwork`](lambda/handle_artwork).

#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "source-asset-drop",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": null
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "vendor_id": 34176,
            "subaccount_id": null,
            "release_name": "Beware the Wolf",
            "project_code": "886446392011",
            "project_name": "Beware the Wolf",
            "project_id": 654321,
            "artwork": {
                "bucket": "source-asset-drop",
                "key": "123456789012/123456789012.tif",
                "filename": "123456789012.tif",
                "ows_assets_filename": null
            }
        },
        "asset_type": "cover",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```

#### example output

```
N.B. artwork bucket changes.
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": null
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`handle-artwork`](lambda/handle_artwork)
This lambda attempts to deliver the artwork asset to `ows-assets` via GraphQL queries.

#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": null
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "vendor_id": 34176,
            "subaccount_id": null,
            "release_name": "Beware the Wolf",
            "project_code": "886446392011",
            "project_name": "Beware the Wolf",
            "project_id": 654321,
            "artwork": {
                "bucket": "staging-bucket",
                "key": "123456789012/123456789012.tif",
                "filename": "123456789012.tif",
                "ows_assets_filename": null
            }
        },
        "asset_type": "cover",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```

#### example output

```
N.B. ows_assets_filename is now a uuid as received from GraphQL method `create_asset_token()`
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`poll-art-status`](lambda/poll_art_status)
This lambda polls `ows-assets` upon delivery of an artwork asset, and attempts to receive notification of successful
completion of ingestion.

#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "vendor_id": 34176,
            "subaccount_id": null,
            "release_name": "Beware the Wolf",
            "project_code": "886446392011",
            "project_name": "Beware the Wolf",
            "project_id": 654321,
            "artwork": {
                "bucket": "staging-bucket",
                "key": "123456789012/123456789012.tif",
                "filename": "123456789012.tif",
                "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
            }
        },
        "asset_type": "cover",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```


#### example failure output - will retry as defined in [SFN def](#state-machine-definition)
```
{
    "errorMessage": "Artwork asset is not valid",
    "errorType": "ArtworkException",
    "stackTrace": [
        "  File \"/var/task/index.py\", line 25, in handler\n    check_artwork_valid(\n",
        "  File \"/var/task/index.py\", line 51, in check_artwork_valid\n    raise ArtworkException(\n"
    ]
}
```

#### example fatal failure output - will proceed to error handling in [SFN diagram](#state-machine-diagram)
```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>",
    "errors": {
        "Error": "Lambda.ArtworkFatalException",
        "Cause": "Asset is no good, bro."
    }
}
```

#### example success output

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`pre-process-audio`](lambda/pre_process_audio)

This lambda receives the metadata and location for a single audio asset. That means track, not release. It checks the
asset extension for validity, and checks the waveform is not in the BAD_FORMATS constant defined in
[`constants/audio.py`](lambda/pre_process_audio/constants/audio.py). If the audio asset has a bad encoding format,
this lambda attempts to repair the audio stream before passing the payload to the next lambda in the orchestration,
[`handle-audio`](lambda/handle_audio).

#### Transcoding
The function [audio_spec_check()](lambda/pre_process_audio/src/index.py#L63) contains all the transcoding logic
performed on audio assets as they travel through the system. The transcoding is aggressive, but can be controlled via
the use of the audio constants at [audio.py](lambda/pre_process_audio/constants/audio.py), or by short-circuiting the
method outright.


#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "track": {
            "bucket": "source-asset-drop",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": null
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        },
        "vendor_id": 9808
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "project_code": "TST00002",
            "project_id": 654321,
            "project_name": "Project Name",
            "release_name": "Release Name",
            "subaccount_id": null,
            "vendor_id": 1234,
            "track": {
                "bucket": "source-asset-drop",
                "key": "123456789012/123456789012.wav",
                "filename": "123456789012.wav",
                "ows_assets_filename": null
                "isrc": "TEST12100201",
                "track_name": "The Track Name",
                "track_sequence_number": 1,
                "track_volume_number": 1,
                "tuid": "34916592"
            }
        },
        "asset_type": "audio",
        "state_machine_name": "sfn_name",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```

#### example output

```
N.B. artwork bucket changes.
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "vendor_id": 1234,
        "track": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": null
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        }
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`handle-audio`](lambda/handle_audio)

This lambda attempts to deliver the audio asset to `ows-assets` via GraphQL queries.

#### example input
```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "vendor_id": 1234,
        "track": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": null
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        }
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "project_code": "TST00002",
            "project_id": 654321,
            "project_name": "Project Name",
            "release_name": "Release Name",
            "subaccount_id": null,
            "vendor_id": 1234,
            "track": {
                "bucket": "staging-bucket",
                "key": "123456789012/123456789012.wav",
                "filename": "123456789012.wav",
                "ows_assets_filename": null
                "isrc": "TEST12100201",
                "track_name": "The Track Name",
                "track_sequence_number": 1,
                "track_volume_number": 1,
                "tuid": "34916592"
            }
        },
        "asset_type": "audio",
        "state_machine_name": "sfn_name",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```

#### example output

```
N.B. ows_assets_filename is now a uuid as received from GraphQL method `create_asset_token()`
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "vendor_id": 1234,
        "track": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.wav"
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        }
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`poll-audio-status`](lambda/poll_audio_status)

This lambda polls `ows-assets` upon delivery of an audio asset, and attempts to receive notification of successful
completion of ingestion.

#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "vendor_id": 1234,
        "track": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.wav"
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        }
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
            "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "project_code": "TST00002",
            "project_id": 654321,
            "project_name": "Project Name",
            "release_name": "Release Name",
            "subaccount_id": null,
            "vendor_id": 1234,
            "track": {
                "bucket": "staging-bucket",
                "key": "123456789012/123456789012.wav",
                "filename": "123456789012.wav",
                "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.wav"
                "isrc": "TEST12100201",
                "track_name": "The Track Name",
                "track_sequence_number": 1,
                "track_volume_number": 1,
                "tuid": "34916592"
            }
        },
        "asset_type": "audio",
        "state_machine_name": "sfn_name",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>"
    }'
```

#### example failure output - will retry as defined in [SFN def](#state-machine-definition)

```
{
    "errorMessage": "Audio asset is not valid",
    "errorType": "AudioException",
    "stackTrace": [
        "  File \"/var/task/index.py\", line 25, in handler\n    check_artwork_valid(\n",
        "  File \"/var/task/index.py\", line 51, in check_artwork_valid\n    raise ArtworkException(\n"
    ]
}
```

Unlike artwork, there is no fatal case exception. The lambda will simply fail on retries in the state machine.

#### example success output
```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "project_code": "TST00002",
        "project_id": 654321,
        "project_name": "Project Name",
        "release_name": "Release Name",
        "subaccount_id": null,
        "vendor_id": 1234,
        "track": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.wav",
            "filename": "123456789012.wav",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.wav"
            "isrc": "TEST12100201",
            "track_name": "The Track Name",
            "track_sequence_number": 1,
            "track_volume_number": 1,
            "tuid": "34916592"
        }
    },
    "asset_type": "audio",
    "state_machine_name": "sfn_name",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

### [`log-asset-error`](lambda/log-asset-error)

This receives an error payload from a failed step in the state machine execution, and logs
these otherwise disappearing errors to the same location as all other catalog ingestion actions.

#### example input

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>",
    "errors": {
        "Error": "Lambda.TooManyRequestsException",
        "Cause": "Rate Exceeded. (Service: AWSLambda; Status Code: 429; Error Code: TooManyRequestsException; Request ID: dd5a16d6-ec19-4ff5-aaff-aac51a60f1c4; Proxy: null)"
    }
}
```

#### curl (docker)

```
curl --location
     --request POST 'http://localhost:9000/2015-03-31/functions/function/invocations' \
     --header 'Content-Type: application/json' \
     --data-raw '{
        "product": {
            "product_id": 123456,
            "upc": 123456789012,
            "vendor_id": 34176,
            "subaccount_id": null,
            "release_name": "Beware the Wolf",
            "project_code": "886446392011",
            "project_name": "Beware the Wolf",
            "project_id": 654321,
            "artwork": {
                "bucket": "staging-bucket",
                "key": "123456789012/123456789012.tif",
                "filename": "123456789012.tif",
                "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
            }
        },
        "asset_type": "cover",
        "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
        "state_machine_name": "<env>-bulk-assets-ingester-sfn",
        "correlation_id": "<uuid4>",
        "errors": {
            "Error": "Lambda.TooManyRequestsException",
            "Cause": "Rate Exceeded. (Service: AWSLambda; Status Code: 429; Error Code: TooManyRequestsException; Request ID: dd5a16d6-ec19-4ff5-aaff-aac51a60f1c4; Proxy: null)"
        }
    }'
```

#### example success output

```
{
    "product": {
        "product_id": 123456,
        "upc": 123456789012,
        "vendor_id": 34176,
        "subaccount_id": null,
        "release_name": "Beware the Wolf",
        "project_code": "886446392011",
        "project_name": "Beware the Wolf",
        "project_id": 654321,
        "artwork": {
            "bucket": "staging-bucket",
            "key": "123456789012/123456789012.tif",
            "filename": "123456789012.tif",
            "ows_assets_filename": "2ba9f569_b4ed_4f63_8aa3_f883f04ef932.tif"
        }
    },
    "asset_type": "cover",
    "execution_name": "BULK_EXAMPLE_TABLE_NAME_2021-01-01_0_0_0-<shortuuid>",
    "state_machine_name": "<env>-bulk-assets-ingester-sfn",
    "correlation_id": "<uuid4>"
}
```

## Docker Instructions
### Setup
* Install [docker](https://docs.docker.com/get-docker/)
* Clone this repository
* Navigate to the lambda you want to work on in `lambda/`

### Running

#### Start Container
```
$ docker-compose up --build function
```

#### Execute Function
Use the HTTP client of your choice. The body of the request is the `event` passed into the function.

```
curl --request POST \
  --url http://localhost:9000/2015-03-31/functions/function/invocations \
  --header 'Content-Type: application/json' \
  --data '{
	"num": 2
}'
```

#### Load Changes
The `src` directory and `config.py` are mounted into the container via the `docker-compose.yaml` configuration. This
means you don't need to re-build the entire container to see changes. **However** the container will need to be
restarted. This process should be as quick as restarting uwsgi when making changes to a web application.
```
$ docker-compose restart function
```
:zap: You should add this command to a post-write hook in your preferred editor to automatically restart.

### Linting and Testing

#### Run Container
```
$ docker-compose up --build lint-and-test
```

#### Accessing Host IP services
`.env` Files will be processed by Docker when you build the container.

```
environment:
      - Environment=${Environment}
      - GRAPHQL_GATEWAY_URL=${GRAPHQL_GATEWAY_URL}
      ...etc
```
In your .env file, set the URL of any services you will be running locally to `docker.for.mac.host.internal`.

e.g.
```
GRAPHQL_GATEWAY_URL=http://docker.for.mac.host.internal:8080/graphql
```

#### Set pytest Options
Sometimes you may want to customize how tests run using [pytest options](https://docs.pytest.org/en/stable/usage.html)
when doing development locally, only running a subset of tests using `-k` for example. Simply edit `TEST_ARGS` value
in `docker-compose.yaml`, **without** quotes, to set the options you need.

:warning: Careful not to commit changes

#### Skip Linting
When developing locally, you might not want to worry about linting when working to make tests pass. Simple edit the
`SKIP_LINT` value in `docker-compose.yaml` to be `1`.

:warning: Careful not to commit changes

#### Capture Exit Code
Unless the container itself crashes, or is killed, the exit code will be `0`. When the exit code of `lint-and-test.sh`
script matters, a pull-request builder for example, run tests with the following flags.
```
$ docker-compose up --exit-code-from lint-and-test --abort-on-container-exit --build lint-and-test
```

### Deploying

#### RTFM
Read this: https://docs.google.com/document/d/1HahCwZ-kHKvjJMQtj-SE4IK7aPphYaul4gNTznXuvFs/edit#

***Don't not read it.***

#### Create ECR Container Repository
Make sure that you have created the appropriate ***ECR Container Repository*** by running the build pipeline
[here](https://pipeline.theorchard.io/job/create-lambda-ecr-repository/) BEFORE YOU APPLY THE ASSOCIATED TERRAFORM.

Failure to do so will result in a half-completed terraform application, which cannot easily be overwritten without
DevOps intervention (and the associated annoyance that creates.)

#### Terraform lambda module
Use lambda module version 1.1.1 (or higher). Set the `use_container_image` directive to `true`

example:
```
module "lambda_parse_bulk_assets" {
  source = "git@github.com:theorchard/terraform-lambda.git?ref=1.1.1"

  environment              = var.environment
  lambda_name              = "lambda-${var.service_name}-parse-bulk-assets"
  lambda_description       = "Handles parsing of JSON files dumped from Bulk Asset DB into a defined structure for SFN"
  use_container_image      = true

  lambda_function_environment_variables = {
    Environment           = var.environment
    LOGGER_DSN            = var.logger_dsn
    SENTRY_DSN            = module.sentry_parse_bulk_assets_lambda.sentry_key_dsn_public_output
  }

  lambda_function_timeout                        = var.lambda_default_timeout
  vpc_security_group_ids                         = var.vpc_security_group_ids
  vpc_subnet_ids                                 = var.vpc_subnet_ids
  lambda_function_reserved_concurrent_executions = var.lambda_concurrent_executions
  datadog_enabled                                = var.datadog_enabled
}
```

#### A Note About Lambda Name Length
Terraform / Atlantis has a hard limit of >=80 characters for some fields, and >50 characters for others.

Be sparing when naming your lambdas, as the full production name of some entities can explode when all the
extra elements are prepended and appended.

`lambda-abc` will rapidly give rise to a sentry element of `prod-lambda-abc-sentry` when following certain naming
conventions.

`lambda-{service}-check-out-file`

with a `{service}` name of "bulk-file-tester"

will rapidly become

`prod-lambda-bulk-file-tester-check-out-files-sentry`

etc.

## Appendix I - State Machine Info
Links and related info to the over-arching SFN orchestration

### State Machine Diagram

![<env>-bulk-assets-ingester-sfn State Machine Diagram](./docs/images/state_machine_definition_diagram_2021-05-27.png)

### State Machine Definition

QA:

State machine at [qa-bulk-assets-ingester-sfn](https://console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:437795906767:stateMachine:qa-bulk-assets-ingester-sfn)
| [Terraform Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/bulk-assets-ingester/sfn_bulk_assets_ingester.tf)
| [SFN Definition](https://github.com/theorchard/terraform-infra/blob/master/qa/bulk-assets-ingester/sfn-bulk-assets-ingester.json)

Prod:

State machine at [prod-bulk-assets-ingester-sfn](https://console.aws.amazon.com/states/home?region=us-east-1#/statemachines/view/arn:aws:states:us-east-1:437795906767:stateMachine:prod-bulk-assets-ingester-sfn)
| [Terraform Definition](https://github.com/theorchard/terraform-infra/blob/master/prod/bulk-assets-ingester/sfn_bulk_assets_ingester.tf)
| [SFN Definition](https://github.com/theorchard/terraform-infra/blob/master/prod/bulk-assets-ingester/sfn-bulk-assets-ingester.json)