# KSQLDB server

## Dockerfile

We use the `latest` tag of the `confluentinc/ksqldb-server` image.
We don't want to pin the specific `x.y.z` version to avoid stale versioning, and major versions like `x` or `x.y` do not exist for that image.

## Entrypoint script

This bash script configures environment variables, launches the `upload-queries.sh` script in the background, and starts the ksqldb-server.

It starts ksqldb-server using the `exec` command to pass the `PID 1` and all the signals (like SIG_TERM) directly to ksqldb-server. Otherwise, the ksqldb-server process may be stopped abnormally.

## Upload-Queries script

This bash script waits for the ksqldb-server API endpoint to be available, and then starts uploading query files from shared and environment-specific directories.

You can set `UPLOAD_QUERIES_DEBUG` environment variable to `true` to print the content of queries that are being sent to ksqldb-server.

## Local environment

This is a local fully isolated environment.
It uses query files from `queries/local`.

The environment exposes the next ports:

- **8088** for ksqldb-server API
- **8081** for schema-registry API
- **8080** for AKHQ UI

Start the environment:

```bash
docker-compose -f docker-compose-local.yml up -d --build
```

Fetch ksqldb-server logs:

```bash
docker-compose -f docker-compose-local.yml logs -f ksqldb-server
```

Fetch ksqldb-server logs for query upload entries:

```bash
docker-compose -f docker-compose-local.yml logs -f ksqldb-server | grep -F 'upload-queries.sh'
```

Stop the environment:

```bash
docker-compose -f docker-compose-local.yml down
```

You can use a 3-node Kafka cluster for local setup with `docker-compose-local-3node.yml` file. But this setup has high memory consumption.

## Local/Dev environment

This environment uses the MSK cluster on Dev, and local ksqldb-server with local AKHQ.
It uses query files from `queries/local`.

To isolate this environment from others, the `local-<username>` prefix is added.
The `<username>` will be replaced with your local account name.

The environment exposes the next ports:

- **8088** for ksqldb-server API
- **8081** for schema-registry API
- **8080** for AKHQ UI

Start the environment:

```bash
docker-compose -f docker-compose-dev.yml up -d --build
```

Fetch ksqldb-server logs:

```bash
docker-compose -f docker-compose-dev.yml logs -f ksqldb-server
```

Fetch ksqldb-server logs for query upload entries:

```bash
docker-compose -f docker-compose-dev.yml logs -f ksqldb-server | grep -F 'upload-queries.sh'
```

Stop the environment:

```bash
docker-compose -f docker-compose-dev.yml down
```

## Query workflow

Query files located in `queries/shared` directory are shared for all environments.
Files inside `dev`, `qa`, `prod` and `local` are specific for each environment.

### Adding a query

To add a query, create a file inside one of the `queries/shared` or `queries/<environment>` directories, while following the next restrictions:

- Use `*.ksql` for file names
- Group multiple stream statements in one file
- Do not put comments in your queries
- Environment-specific queries should be placed in the corresponding directories
- Shared queries will be executed before the environment-specific ones

## Required env variables for building the Docker image

Variable | Description
------------ | -------------
GITHUB_USERNAME | Github username for access token
GITHUB_TOKEN | Github personal access token with `read:packages` scope

## Required env variables for QA and Prod

Variable | Description
------------ | -------------
KSQL_BOOTSTRAP_SERVERS | A list of Kafka bootstrap servers
KSQL_GROUP_UNIQUE_IDENTIFIER | Unique identifier for server group, e.g. `qa_orchard`
KSQL_KSQL_SCHEMA_REGISTRY_URL | Schema registry URL and port

## Production usage

Passing secrets as build arguments is insecure. The best way is to pass them as `secrets` using [Docker Buildkit](https://docs.docker.com/develop/develop-images/build_enhancements/#new-docker-build-secret-information).

```bash
export DOCKER_BUILDKIT=1

docker build -t ksqldb-server --progress=plain --no-cache --secret id=github,src=github.secret .
```

## Authentication

Jetty authentication support has not been implemented. Use [nginx-auth-proxy](https://github.com/theorchard/nginx-auth-proxy/) instead.