# Schema-Registry

## Dockerfile

We use the `latest` tag of the `confluentinc/cp-schema-registry` image.
We don't want to pin the specific `x.y.z` version to avoid stale versioning, and major versions like `x` or `x.y` do not exist for that image.

## Entrypoint script

This bash script configures environment variables, launches the `upload-schemas.sh` script in the background, and starts the schema-registry.

It starts schema-registry using the `exec` command to pass the `PID 1` and all the signals (like SIG_TERM) directly to schema-registry. Otherwise, the schema-registry process may be stopped abnormally.

## Upload-Schemas script

This bash script waits for the schema-registry API endpoint to be available, and then starts uploading schema definition and config files from shared and environment-specific directories.

Schema config gets uploaded before the schema definition to allow changing schema compatibility level before any schema changes. This is useful when applying backward-incompatible changes.

## Local environment

This is a local fully isolated environment.
It uses schema definition files from `schemas/local`.

The environment exposes the following ports:

- **8081** for schema-registry API
- **8080** for AKHQ UI
- **9092** for broker

Start the environment:

```bash
docker-compose -f docker-compose-local.yml up -d --build
```

Fetch schema-registry logs:

```bash
docker-compose -f docker-compose-local.yml logs -f schema-registry
```

Fetch schema-registry logs for schema upload entries:

```bash
docker-compose -f docker-compose-local.yml logs -f schema-registry | grep -F 'upload-schemas.sh'
```

Stop the environment:

```bash
docker-compose -f docker-compose-local.yml down
```

## Local/Dev environment

This environment uses the MSK cluster on Dev, and local schema-registry with local AKHQ.
It uses schema definition files from `schemas/local`.

To isolate this environment from others, the `local-<username>` prefix is added.
The `<username>` will be replaced with your local account name.

The environment exposes the next ports:

- **8081** for schema-registry API
- **8080** for AKHQ UI
- **9092** for broker

Start the environment:

```bash
docker-compose -f docker-compose-dev.yml up -d --build
```

Fetch schema-registry logs:

```bash
docker-compose -f docker-compose-dev.yml logs -f schema-registry
```

Fetch schema-registry logs for schema upload entries:

```bash
docker-compose -f docker-compose-dev.yml logs -f schema-registry | grep -F 'upload-schemas.sh'
```

Stop the environment:

```bash
docker-compose -f docker-compose-dev.yml down
```

## Schema workflow

Schema files located in `schemas/shared` directory are shared for all environments.
Files inside `dev`, `qa`, `prod` and `local` are specific for each environment.

### Adding a schema

To add a schema for a topic, create a directory inside one of the `schemas/shared` or `schemas/<environment>` directories.
The directory name should be using the next format: `<topic>-key` if you want to use schema for keys of the particular topic, or `<topic>-value` if you want to use schema for values of that topic. Usually, you will need only the `<topic>-value` schema.

After creating the directory, add 2 files to it: `schema.json` with schema definition and `config.json` with schema config.
You can use this https://json-schema-validator.herokuapp.com/avro.jsp validator page to validate your schema json. That can help identity errors before you even start your docker compose to test changes.

Example of `schema.json`:

```json
{
  "type": "record",
  "name": "ExampleSchema",
  "namespace": "orchard.example",
  "fields": [
    {
      "name": "message",
      "type": "string"
    }
  ]
}
```

Example of `config.json`:

```json
{
  "compatibility": "BACKWARD"
}
```

### Push changes to Dev

For schema development/iteration, you can push your changes to the dev schema registry using the following cURL:

```bash
curl --location --request POST 'https://dev-schema-registry.dev.theorchard.io/subjects/<topic>-value/versions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "schema": "{\"type\":\"record\",\"name\":\"review\",\"namespace\":\"product\",\"fields\":[{\"name\":\"operation\",\"type\":{\"type\":\"record\",\"name\":\"operationInformation\",\"fields\":[{\"name\":\"type\",\"type\":{\"type\":\"enum\",\"name\":\"operationOptions\",\"symbols\":[\"create\",\"update\",\"delete\"]},\"doc\":\"Operation type.\"},{\"name\":\"context\",\"type\":[\"null\",{\"type\":\"enum\",\"name\":\"operationContextOptions\",\"doc\":\"Additional information about event.\",\"symbols\":[\"new\",\"copy\",\"update\",\"submit\",\"unsubmit\",\"approve\"]}],\"default\":null},{\"name\":\"timestamp\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"timestamp of the operation.\"}]},\"doc\":\"Operation information\"},{\"name\":\"payload\",\"type\":{\"type\":\"record\",\"name\":\"payloadInformation\",\"fields\":[{\"name\":\"product_id\",\"type\":\"int\",\"doc\":\"Product unique identifier generated by art_relations database.\"},{\"name\":\"upc\",\"type\":[\"null\",\"string\"],\"doc\":\"Universal Product Code.\"},{\"name\":\"product_name\",\"type\":[\"null\",\"string\"],\"doc\":\"Product name provided by user.\"},{\"name\":\"project_id\",\"type\":[\"null\",\"int\"],\"doc\":\"Project id of the product.\"}]},\"doc\":\"Payload information.\"}]}",
    "schemaType": "AVRO"
}'
```
Replace schema with your modified schema. It is escaped, but if you run the schema-registry/akhq stack as described in the docker-compose section for `docker-compose-local.yml`, you can copy the schema value from http://localhost:8081/schemas.