# Debezium MySQL Source ## Usage There are two options to run the connector: 1) a local connector to run against Kafka in an existing environment (like dev) defined in `docker-compose.yml`, using your own `.env` 2) a fully dockerized environment with a basic setup to test the connector defined in `docker-compose-local.yml`, using a provided `.env.local` file Before building the connector, you will need to set up access to the Shared AWS Account's ECR repository. ```sh awsume prod ``` If you haven't [set up the ecr-credential-helper with Docker](https://www.notion.so/AWS-Access-f841b9dd815d4443a80e96a86c92cd2f?pvs=4#7a6624c58c6642abaecd804e5f25c820), you will also have to run this manually: ```sh aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 086679231553.dkr.ecr.us-east-1.amazonaws.com ``` Copy `.env.shadow` to `.env` (or use `.env.local` for the dockerized env) and configure the connector options more info in [the Connector Docs](https://kdh-docs.qaorch.com/docs/kafka-connect/available-connectors/debezium-mysql-source#configuration-properties). ### Dockerized Run the following commands to bring up all the services with docker compose: ```sh docker compose -f docker-compose-local.yml up --build -d # Follow the logs for the connector only docker compose -f docker-compose-local.yml logs debezium-mysql-source -f ``` ### Connector only Run the following commands to bring up a test database and connector: ```sh docker compose -f docker-compose.yml up build -d ``` If all is configured correctly you should see the following logs: ``` 2024-07-22 11:20:51 Mon Jul 22 05:50:51 UTC 2024: debezium_mysql_source connector started 2024-07-22 11:20:57 Jul 22, 2024 5:50:57 AM com.github.shyiko.mysql.binlog.BinaryLogClient connect 2024-07-22 11:20:57 INFO: Connected to test-db:3306 at 3f05ad8a-47ed-11ef-a764-0242ac130002:1-12 (sid:1, cid:8) ``` and at this point you can see the connector status and its config by the following links: - http://localhost:8084/connectors/debezium_mysql_source/tasks/0/status - http://localhost:8084/connectors/debezium_mysql_source/config > Note that docker-compose will attempt to bind to ports 3306, 9095 and 8084 on your local machine and will fail if they are already in use. ### Topic names By default, the MySQL connector writes change events for all of the INSERT, UPDATE, and DELETE operations that occur in a table to a single Apache Kafka topic that is specific to that table. The connector uses the following convention to name change event topics: `topicPrefix.databaseName.tableName` Eg ``` art_relations.releases > will write change events to topic > cdc.art_relations.releases art_relations.artist_info > will write change events to topic > cdc.art_relations.artist_info ``` If you want to change it from underscore/snake case to camel case then use the `CHANGE_TOPIC_CASE_FROM` env as: ``` export CHANGE_TOPIC_CASE_FROM=LOWER_UNDERSCORE export CHANGE_TOPIC_CASE_TO=LOWER_CAMEL ``` Eg: ``` art_relations.releases > will write change events to topic > cdc.artRelations.releases art_relations.artist_info > will write change events to topic > cdc.artRelations.artistInfo ``` > In docker-compose JMX Metrics is available on config: localhost:9095 with unauthenticated access. Refer [this sample app](https://github.com/cstroe/java-jmx-in-docker-sample-app/blob/master/README.md) for more details. Refer [this](https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-monitoring) to see what metrics are available to us. ### Auto topic.creation > The `topic.creation.default.*` configurations are only applicable to the table level topics that are created by the connector. If you want to turn off auto topic creation, you will also have to do the following changes: 1) Turn off at broker level with config `"topic.creation.enable":"false"` 2) set `"topic.creation.enable": false,` in Debezium config 3) set `env CONNECT_TOPIC_CREATION_ENABLE=false` in Debezium connector docker file or env or terraform. 4) manually/terraform the topic using the debezium connector's naming convention. Connector wont throw error or log anything if the topic does not exist when it starts. It will simply ignore those messages till the topic is created. ### Connector upgrade > Note: After updating to confluentinc/cp-kafka-connect-base:7.6.1, the kafka-connect-transform-common results in some error in connector logs. They are from for following classes: AdjustPrecisionAndScale, HeaderToField, SetMaximumPrecision, CloudEventsConverter.We are not using these transformations so ignoring the errors and raising it [as in issue](https://github.com/jcustenborder/kafka-connect-transform-common/issues/111) with their team. > Note: We are on connector 3.0.6 since most of our DBs are have been upgraded to Mysql V8.0.x ### Database Requirements #### Enabling the binlog Debezium work is based on the MySQL binlog. The binlog must be enabled and the `binlog_format` must be set to `ROW`. How to enable the binlog for MySQL can be found [here](https://debezium.io/documentation/reference/connectors/mysql.html#enable-mysql-binlog). How to enable the binlog for AWS RDS MySQL can be found [here](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_LogAccess.MySQL.BinaryFormat.html). #### Granting Privileges The user that Debezium uses to connect to the database must have the following privileges: ```sql CREATE USER 'kafka_connect'@'*' IDENTIFIED BY 'kafka_connect'; GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'kafka_connect' IDENTIFIED BY 'kafka_connect'; FLUSH PRIVILEGES; ``` More information on the required privileges can be found [here](https://debezium.io/documentation/reference/connectors/mysql.html#mysql-creating-user). ### Deployed Connector Configure the adapter for the database you wish to use via terraform. [Example](https://github.com/theorchard/terraform-infra/blob/master/qa/kafka-infra/debezium_source/main.tf#L17-L125) ## Links - [Confluent Docs](https://docs.confluent.io/debezium-connect-mysql-source/current/overview.html) - [Debezium Docs](https://debezium.io/documentation/reference/3.0/connectors/mysql.html)