# DynamoDB Sink Connector

## Description

This type of the Kafka Connector pulls data form Kafka topic and write data to a DynamoDB table.
The connector is based on Apache Camel framework. More information about the connector can be found [here](https://camel.apache.org/camel-kafka-connector/next/reference/connectors/camel-aws-ddb-sink-kafka-sink-connector.html).

## Local testing

### Running the connector

The easiest way to test the connector locally is to use dev Kafka cluster and dev AWS Dynamodb account. So before running the connector make sure that you have access to those services. Dev Kafka cluster is available under VPN connection. For accessing Dev AWS you should have a dedicated AWS account.

After that you need to copy `.env.shadow` to `.env` and fill in the values. Then run
```
awsume dev
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 086679231553.dkr.ecr.us-east-1.amazonaws.com
docker-compose up --build -d
```

If all is configured correctly you should see the following logs:

```
Woohoo! Kafka Connect is up!
Generate the connector config...
Creating Kafka Connect dynamodb_sink connector
Curl HTTP status code is 200
dynamodb_sink connector successfully updated
dynamodb_sink connector started
```

and at this point you can see the connector status and its config by the following links:

- http://localhost:8083/connectors/dynamodb_sink/status

- http://localhost:8083/connectors/dynamodb_sink/config

### Kafka messages

Kafka message value should be in JSON format. The connector will try to parse the message and write it to the DynamoDB table. The value must contain `Partition Key` and `Sort Key` (if necessary) fields according to the DynamoDB table definition.


### Schema Registry

The connector can be configured to use AVRO schema for the Kafka messages.
For a proper configuration you need to do the following steps:
- Before sending messages you need to register schema for the topic. Here is a minimal schema example:
```json
{
  "type": "record",
  "name": "mySchemaName",
  "namespace": "com.orchard.kafka.data.highway",
  "fields": [
    {
      "name": "filename",
      "type": "string",
      "doc": "Filename value."
    },
    {
      "name": "myProperty",
      "type": "int",
      "doc": "Test property."
    }
  ]
}
```

Registered schemas on dev environment can be found here: https://dev-schema-registry.dev.theorchard.io/schemas

- The schema must be registered under the following name: `<my-topic>-value`. The connector will automatically discover the schema and use it for the messages.
- Additionally to the basic configuration set the following env variables
  - SCHEMA_TYPE=AVRO
  - SCHEMA_REGISTRY_URL=https://dev-schema-registry.dev.theorchard.io


## Production Deploy

### Terraform example

TBD

### AWS resources access

By default, the AWS resources access is automatically managed by STS component included in this package.

### Jenkins Pipeline

For initial pipeline build use this Jeknins Job https://pipeline.theorchard.io/job/kafka-infra-jobs-builder/