# Docker Compose for Maxwell, Kinesis Listener, and Lambda Function

This folder contains a Docker Compose configuration for setting up and running the following services locally:

1. **Maxwell**: Connects to a MySQL database and sends logs to a Kinesis stream.
2. **Kinesis Listener**: Listens to events in the Kinesis stream and invokes a Lambda function.
3. **Function**: A local version of the Lambda function that processes logs by inserting them into Snowflake tables.

## Prerequisites

1. Install Docker and Docker Compose.
2. Ensure you have the `awsume` CLI installed for managing AWS credentials.
3. Export your AWS credentials using `awsume <profile_name>` (e.g., `awsume dev-role`). Replace `<profile_name>` with your AWS profile name.
4. Ensure you have access to the shared ECR repo via `awsume`.
5. Copy the `.env.shadow` file to `.env` and update the required values:
   ```bash
   cp .env.shadow .env
   ```

### Environment Variables

Update the following variables in the `.env` file:

- **General Configuration**:
  - `SENTRY_DSN`
- **AWS Configuration**:
  - `AWS_REGION` (e.g., `us-east-1`)
- **Snowflake Configuration**:
  - `SNOWFLAKE_USER`
  - `SNOWFLAKE_ACCOUNT`
  - `SNOWFLAKE_ROLE`
  - `SNOWFLAKE_WAREHOUSE`
  - `SNOWFLAKE_DATABASE`
  - `SNOWFLAKE_SCHEMA`
  - `SNOWFLAKE_PRIVATE_KEY_LOCATION` (set this to the container path where the private key is mounted, e.g., `/var/task/private_key.pem`)
  - `SNOWFLAKE_PRIVATE_KEY_PASSPHRASE`
- **Kinesis Configuration**:
  - `KINESIS_STREAM_NAME`
  - `LAMBDA_ENDPOINT` (e.g., `http://function:8080/2015-03-31/functions/function/invocations`)
- **Maxwell Configuration**:
  - `MAXWELL_KINESIS_STREAM`
  - `MAXWELL_MYSQL_HOST`
  - `MAXWELL_MYSQL_USER`
  - `MAXWELL_MYSQL_PASSWORD`
  - `MAXWELL_MYSQL_SCHEMA_DATABASE`

### Important Notes

1. The values for `KINESIS_STREAM_NAME` and `MAXWELL_KINESIS_STREAM` must be the same for the services to function correctly.
2. You can update `INCLUDE_EXPRESSIONS` and `TABLES_TO_LOG` in the .env file to include additional tables. These values should be comma-separated.

## Adding Snowflake Private Key

To securely use your Snowflake private key, you need to mount it as a volume in the `function` service.

1. Update the `docker-compose.yml` file to include the private key volume:
   ```yaml
   services:
     function:
       volumes:
         - <local_machine_path>:<container_path>
   ```
   Replace `<local_machine_path>` with the full path to your private key on your local machine and `<container_path>` with the path inside the container (e.g., `/var/task/private_key.pem`).

2. Ensure that the `SNOWFLAKE_PRIVATE_KEY_LOCATION` in the `.env` file matches the container path.

## Running the Services Locally

1. Export your AWS credentials using `awsume`:
   ```bash
   awsume dev-role
   ```
   Replace `dev-role` with your specific AWS profile name.

2. Start the services using Docker Compose:
   ```bash
   docker compose up -d function
   ```
   This will automatically start all dependent services (Maxwell, Kinesis Listener, and Function).

3. Verify that the services are running:
   ```bash
   docker compose ps
   ```

## Stopping the Services

To stop all services, run:
```bash
   docker compose down
```

## Verifying the Setup

To verify that the setup works:
1. Check the logs for each service using `docker compose logs <service_name>`.
2. Ensure logs from Maxwell are being sent to the Kinesis stream.
3. Confirm that the Kinesis Listener invokes the Lambda function correctly.
4. Verify that logs are being processed and inserted into the Snowflake database.

## Troubleshooting
- Ensure that `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are correctly exported or configured in `.env`.
- Verify that the private key file is accessible and correctly mounted in the `function` service.
- Check that `KINESIS_STREAM_NAME` and `MAXWELL_KINESIS_STREAM` match in the .env file.
- Update `INCLUDE_EXPRESSIONS` and `TABLES_TO_LOG` to include the required tables for logging.