# KMS encryption issues and how to face them

## KMS encryption issues

1. Data becomes inaccessible when the corresponding KMS key is deleted:

    - logs in S3 (possibly affected: EMR)
    - logs in CloudWatch (affected: Airflow)
    - objects in S3 uploaded from a managed service (possibly affected: EMR, Airflow)

2. Inter-service communication:

    - S3 bucket uses `KMS-Key-1` as default SSE-KMS encryption. To read objects which were uploaded without encryption method being specified, services need explicit access to `KMS-Key-1`.
    - EMR uses `KMS-Key-2` for encryption at rest, and uploads objects to the bucket encrypted by `KMS-Key-2`.
    - Airflow uses `KMS-Key-3` for encryption at rest, and is unable to read objects from the bucket without explicit access to `KMS-Key-2`.

3. KMS key monthly price + key rotations.

4. GSIRT alerts on KMS keys scheduled for deletion.

## KMS authorization

1. IAM principal's behalf: S3 SSE-KMS (authorized by IAM or KMS key policy).
2. Service IAM role: Airflow, EMR (authorized by IAM or KMS policy).
3. Service principal: CloudWatch Logs (authorized by KMS key policy).

## Issues remediation

1. Forwarding logs to DataDog with DataDog Archive enabled for long-term storage.
2. Use a shared KMS key per `application_family`.
3. Avoid KMS key deletion at all.

## Shared KMS keys solution

1. Create `terraform-kms-key` module:

    - KMS key alias is `alias/{environment}-{service_name}-{application_family}`.
    - Default `service_name` is `shared`, can be customized to have several KMS keys per `application_family`.
    - IAM policies to encrypt/decrypt and decrypt only access.
    - Encrypt/Decrypt policy permits grants to AWS services.
    - Grants CloudWatch Logs service encrypt/decrypt access.

2. Create `terraform-infra` project per each application family to host KMS keys:

    - Can be placed in `terraform-infra/{environment}/kms/{application_family}`.

3. Update modules to use this approach instead of embedded KMS keys:

    - Add `kms_key_name` variable to specify KMS key name with `shared` as the default.
    - Add data source to fetch KMS key alias to get corresponding KMS key ID.
    - Add data source to fetch on the policies depending on the required access.
    - Attach the policy to the service role.
    - Additionally encrypt CloudWatch log groups with KMS key.
    - Modules to update in the first place: `terraform-s3`, `terraform-lambda`, `terraform-airflow`, `terraform-emr`.

## Pros and Cons

Pros:

- KMS keys are tagged with `application_family` so the teams will get access to _their_ keys without any additional actions.
- CloudWatch Logs KMS encryption may help to avoid SME security findings triggered.
- Inter-service access will work out of the box, as the services use the same key.

Cons:

- Possibility to reach KMS key API request limit and get throttled:

  - The most probable service to be affected by this is S3, which can easily be remediated by enabling `bucket_key`.

- Module update have breaking changes:

  - Not critical for `terraform-emr` as it gets re-created each time.
  - Not critical for `terraform-airflow` as the only reason to bump the module version is to get newer Airflow, which anyway requires environment to be torn down.
  - Not critical for `terraform-lambda` at all.
  - May have data loss impact on `terraform-s3`.