--- sidebar_position: 3 --- # Deploying a Connector instance ## Prerequisites 1. An existing [Kafka Topic created with Terraform](../kafka-basics/events-and-topics.mdx#infrastructure-setup) 2. Pick an existing connector from the [list of available ones](./available-connectors/connectors-table). 3. `[Optional, Sink only]` An existing DLQ Kafka Topic. ## 1. Configure a new Connect Cluster with Terraform In [terraform-infra](https://github.com/theorchard/terraform-infra), under a directory for your prefered AWS Account and environment, create a new set of resources. As a live example, you can check out this [Snowflake Sink connector in Terraform](https://github.com/theorchard/terraform-infra/tree/master/prod/kafka-infra/snowflake_sink_abacus). ### Variables The following is a list of the minimum variables specific to the connector that are used across the resources in Terraform: - **cluster_type**: The prefix defining this as a Kafka Connect cluster, set as `kc`. - **connector_type**: Defines the type of connector using a mix of if it is a Sink or a Source, and the target of the integration. In our example, this is `sfsink`. - **connector_name**: A descriptive unique name for this connector type, usually the name of the data being targeted. In our example, for "Sales File Delivery" this is `sfd`. ### Locals Set the following locals to simplify their use across resources: - **service_name**: `--` - **ecs_cluster_name**: `-` - **ecr_image_name**: `-` :::info[Resource name character limit] Some resources part of this deployment have a strict name character limit of **32 characters**, to work around it is best to try and shorten the cluster_type to **`kc`** to allow the connector type and name to be more descriptive. ::: ### Fargate Environment Using the latest release of the [Terraform-Fargate module](https://github.com/theorchard/terraform-fargate), with emphasis on the following properties: - **service_name**: Unique name for the fargate service, use `service_name` local variable. - **non_ecr_image**: This is the Connector's ECR image on AWS, check the Connector docs for the right value and set it as: `.dkr.ecr..amazonaws.com/${local.ecr_image_name}:latest` - **desired_task_count**: number of [Kafka Connect Cluster Workers](./intro.mdx#workers) - **health_check_path**: Set to `/connectors/${local.service_name}` - **environment_variables**: Besides the ones defined in the Connector's docs, the following must be set in all connectors: - **MAX_TASKS**: Number of conenctor tasks in a cluster. For **Sink** connectors, this number cannot be greater than the number of partitions in the configured Topic. - **CONNECT_GROUP_UNIQUE_IDENTIFIER**: This must always be set to the `ecs_cluster_name`. - **CONNECTOR_NAME**: Set to the local var `service_name`. - **CONNECT_BOOTSTRAP_SERVERS**: Kafka Bootstrap Server addresses. - **SERVICE_NAME**: Set to the local var `service_name`. - **KAFKA_TOPICS**: List of Kafka Topics that the connector should use. ### Additional Security Group Rules #### Distributed mode Add a new INGRESS AWS Security Group rule to the security group id from the Fargate task (`module.your_fargate_task.fargate_security_group_id`) to allow tasks to communicate with each other in distributed mode, for TCP port `8083`. #### AKHQ cross-account If your connector is being deployed in it's own AWS account (not LEGACY QA/PROD), you will need an additional SG INGRESS Rule for port `443` on TCP allowing the IPs from The Orchard legcy account, like in [this example](https://github.com/theorchard/terraform-infra/blob/master/permissions-platform/qa/kafka-pp/connectors/source/dynamodb-streams/pp-identity/main.tf#L163-L174). ### Fargate Service Dashboard and Monitors This is optional, but highly recommended as the minimum monitoring setup for a connector. Using the latest version of the service [Terraform-Datadog Kafka-Connector module](https://github.com/theorchard/terraform-datadog/tree/master/modules/kafka_connector). ## 2. Add the new connector to CI/CD for deployments Adding a new Kafka Connect service for deployment is easy, just add your service name and related Project (which connector it's using) to the `SERVICES` mapping in [kafka-connect's Jenkinsfile](https://github.com/theorchard/kafka-connect/blob/master/Jenkinsfile). Once added you'll be able to trigger manual deployments through [this Jenkins job](https://pipeline.theorchard.io/job/theorchard/job/kafka-connect/job/master/). :::tip The pipeline will have to be run after new changes are applied to the Fargate resource in terraform, which includes environment variables in terraform-infra. This is because the Terraform Fargate module just creates a new Fargate Task Definition, and the deployment pipeline updates the task to start using the latest definition. ::: ## 3. Add the new Connector to AKHQ To have visibility over the Connector status and current configuration in AKHQ you can add a new item to the AKHQ configuration [like in this example](https://github.com/theorchard/terraform-infra/blob/e631bde48e4f526e98d45aed2e16a4f325d024ef/prod/kafka-infra/akhq/main.tf#L78-L79). ![A Connector in AKHQ](./img/akhq-connector.png) ### Cross Account CNAME If your connector is being deployed in an account that's not The Orchard AWS Legacy QA/PROD account, add a CNAME mapping before adding the connector to the AKHQ configuration, like [in here](https://github.com/theorchard/terraform-infra/pull/17470/files).