# OpenSearch Sink Connector

## Description

This type of the Kafka Connector pulls data from a Kafka Topic and writes it into OpenSearch.

For ElasticSearch, please use the available [ElasticSearch connector](../elasticsearch_sink/).

### OpenSearch access configuration

For local testing `OPENSEARCH_USERNAME` and `OPENSEARCH_PASSWORD` can be set in `docker-compose.yml` file or in `.env` file and run 

```
awsume dev
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 086679231553.dkr.ecr.us-east-1.amazonaws.com
docker-compose up --build -d
```


For the instances deployed to AWS we use Lenses' Secret Provider. This extension allows the connector to get secrets from AWS Secrets Manager. Secret values are used for the following config params:

- connection.username
- connection.password

the values are provided with this format `${aws:<SECRET_NAME>:<SECRET_KEY>}` where `<SECRET_NAME>` is composed of `"<Environment>/<SERVICE_NAME>/<SECRETS_MANAGER_OPENSEARCH_SECRET_NAME>/credentials"` env variables and `<SECRET_KEY>` is `username` or `password` accordingly.

Also, for access AWS secrets manager from local machine please provide temp credentials via `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, ` AWS_SESSION_TOKEN` env variables.

### Connector flags
The following environment variables can be set to change the configuration of the connector:

- `OPENSEARCH_KEY_IGNORE`: By default (false) the connector will use the Kafka message key as the document ID to be set in OpenSearch, which is useful for `upsert` operations. Setting this property to `true` the document IDs will be autogenerated.
- `TRANSFORMS_RENAMETOPIC_REPLACEMENT`: The name of the index or alias in OpenSearch to sink the messages in. This is required as by default the connector would use the Kafka Topic name as Index name.
- `WRITE_METHOD`: `upsert` or `insert`. For upserts please keep `OPENSEARCH_KEY_IGNORE` as **false**.


### Running it locally

To Start the server:

```sh
cp .env.shadow .env
# add your credentials and configurations in the .env file
docker-compose up
```


### References
- [Aiven OpenSearch Connector for Apache Kafka](https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/tree/main)