# Terraform Standards (from terraform-infra)

All Terraform code must follow the conventions established in the `terraform-infra` monorepo (`../../../terraform-infra/`). When Resonance Engine infrastructure is finalized, it will live under `dev/resonance-engine/` in that repo. During development, we mirror the same patterns here.

## Required File Structure

Every Terraform configuration must include:
- `main.tf` — Terraform settings, backend, provider declarations, data sources
- `variables.tf` — Input variables
- `versions.tf` — **Required** — Exact Terraform version (not ranges)

Optional:
- `outputs.tf` — Only when downstream consumers need resource ARNs/IDs
- `imports.tf` — Resource import blocks for migrations
- `moved.tf` — Resource move blocks for refactoring

For large projects with 5+ lambdas, create subdirectories per lambda to prevent slow plan/apply.

## Backend Configuration

```hcl
terraform {
  backend "s3" {
    bucket  = "dev-orcd-terraform-state"
    key     = "dev/resonance-engine/terraform.tfstate"
    region  = "us-east-1"
    encrypt = true
  }
}
```

## Provider & Default Tags

In the `terraform-infra` monorepo, projects use the shared `terraform-default-tags` module. This repo **does not** use that module (it requires a specific repo path); instead, it mirrors the same behavior via `local.default_tags`:

```hcl
locals {
  default_tags = {
    environment                 = var.environment
    application_family          = var.application_family
    service_name                = var.service_name
    terraformed                 = "true"
    terraform_github_repository = "terraform-infra"
    terraform_github_path       = "dev/resonance-engine"
  }
}

provider "aws" {
  region = var.aws_region

  default_tags {
    tags = local.default_tags
  }
}
```

When this config migrates to `terraform-infra`, switch to `module "default_tags"`.

## Standard Variables

Every configuration must define:

```hcl
variable "aws_region" {
  default     = "us-east-1"
  description = "AWS region"
}

variable "environment" {
  description = "Available values: dev, qa, prod."
  default     = "dev"
}

variable "application_family" {
  default = "resonance-engine"
}

variable "service_name" {
  default = "resonance-engine"
}
```

## versions.tf

Use exact Terraform version (not ranges). Use the latest version:

```hcl
terraform {
  required_version = "1.12.0"
  required_providers {
    aws = {
      source = "hashicorp/aws"
    }
  }
}
```

## Naming Conventions

- **IAM policies**: `AWSSERVICE-${var.environment}-${var.service_name}-access_type` (e.g., `S3-dev-resonance-engine-RW`)
- **Other resources**: `${var.environment}-${var.service_name}-${optional_suffix}`
- **Lambda names**: Must begin with `lambda-` (e.g., `lambda-resonance-manifest-parser`)

## Lambda Module

Use the Orchard's Lambda module (`terraform-lambda@5.2.1` for container images):

```hcl
module "lambda_manifest_parser" {
  source = "git@github.com:theorchard/terraform-lambda.git//?ref=5.2.1"

  environment        = var.environment
  application_family = var.application_family
  lambda_name        = "lambda-resonance-manifest-parser"
  lambda_description = "Parses DDB export manifest and sends file paths to SQS"

  use_container_image        = true
  container_image_custom_uri = "${aws_ecr_repository.manifest_parser.repository_url}:latest"

  vpc_enabled    = true
  vpc_id         = module.vpc_info.vpc_id
  vpc_subnet_ids = module.vpc_info.default_private_subnet_ids

  lambda_function_timeout                        = var.manifest_parser_timeout
  lambda_function_memory_size                    = var.manifest_parser_memory_size
  lambda_function_reserved_concurrent_executions = var.manifest_parser_concurrency

  # Datadog disabled in dev — enable for prod
  datadog_enabled          = false
  datadog_advanced_enabled = false

  lambda_function_environment_variables = {
    ENVIRONMENT       = var.environment
    SQS_QUEUE_URL     = module.manifest_files_queue.queue_url
    DDB_EXPORT_BUCKET = var.ddb_export_bucket_name
  }

  # IAM policies attached as inline role policies in iam.tf
  # (generic-engineer-role lacks iam:CreatePolicy for managed policies)

  zappa_s3_policy_enabled = false
}

# Inline role policies — attach to the module's execution role
resource "aws_iam_role_policy" "manifest_parser_s3_read" {
  name   = "S3-${var.environment}-${var.service_name}-read-ddb-export"
  role   = module.lambda_manifest_parser.lambda_role_id
  policy = data.aws_iam_policy_document.s3_read_ddb_export.json
}
```

Key parameters:
- **IAM policies**: Use `aws_iam_role_policy` (inline) attached to `module.*.lambda_role_id` — the dev `generic-engineer-role` lacks `iam:CreatePolicy` for managed policies
- **`container_image_custom_uri`**: ECR repo URL + tag (image changes are ignored by the module — deployments happen outside Terraform)
- **`vpc_enabled`**: Must be explicitly `true` for VPC integration
- **Module outputs**: `lambda_name`, `lambda_arn`, `lambda_invoke_arn`, `lambda_role_id`, `lambda_role_arn`

## IAM Pattern

Use `aws_iam_policy_document` data sources for policies (not inline JSON):

```hcl
data "aws_iam_policy_document" "lambda_assume_role" {
  statement {
    effect = "Allow"
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
    actions = ["sts:AssumeRole"]
  }
}

resource "aws_iam_role" "collector_worker_role" {
  name               = "${var.environment}-${var.service_name}-collector-worker-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume_role.json
}
```

## Snowflake Sink Connector Pattern (Fargate)

Follows org-standard pattern from `terraform-infra/prod/kafka-infra/snowflake_sink*/`:

- **Module**: `terraform-fargate` (v5.5.4+) running Kafka Connect
- **Connector config** (environment variables):
  - `SNOWFLAKE_INGESTION_METHOD = "SNOWPIPE_STREAMING"`
  - `KAFKA_TOPICS = "resonance-engine.spotify-data"`
  - `SNOWFLAKE_TOPIC_TABLE_MAP = "resonance-engine.spotify-data:TABLE_NAME"`
  - `BUFFER_COUNT_RECORDS = 10000`, `BUFFER_SIZE_BYTES = 5000000`, `BUFFER_FLUSH_TIME = 240`
  - `DLQ_TOPIC_NAME = "dlq.resonance-engine"`
- **Secrets**: Snowflake private key + passphrase via `terraform-secrets-manager`
- **Health check**: Port 8083 (Kafka Connect REST API)
- **Monitoring**: `terraform-datadog` kafka_connector module
- **Service naming**: `kc-sfsink-resonance-engine`

## Key Conventions

- **Always use data sources** for external resource references — never hardcode IDs or ARNs
- Use `data.aws_caller_identity.current` for account ID
- Use `for_each` instead of `count` for multiple similar resources
- Use `module "vpc_info"` from `terraform-vpc-info` for VPC lookups
- Use `module "default_tags"` for consistent tagging on all resources
- Modules must use exact version pinning (no ranges), e.g.: `?ref=1.2.3`
- Run `terraform fmt` before committing
- Follow EditorConfig: 2-space indentation, Unix line endings (LF), UTF-8, 80-char line limit

## Security Requirements

- Never commit secrets, tokens, or keys
- Encrypt everything using AWS-managed or CMK KMS keys
- No plaintext secrets — use Secrets Manager (`terraform-secrets-manager` module)
- Follow least privilege — never use wildcards in IAM
- Run Checkov scans and resolve warnings

## CI/CD (terraform-infra)

- Atlantis handles `terraform plan` and `terraform apply` via PR comments
- Jenkins pipeline runs Checkov security scans
- PRs should be atomic (single directory, max 10 services, single environment)
- Never merge until Atlantis has successfully applied
- Always commit to dev first, practice deployments before higher environments