# Ephemeral Environments -- TRD

## Status

**Draft** | 2026-03-22 | Author: Michael Rojas

## Overview

PR-triggered ephemeral environments for ows-coda on QA infrastructure. Developers comment on GitHub PRs to create, destroy, extend, and list short-lived copies of the application. Each environment gets its own ECS Fargate service and MySQL database schema on the shared QA RDS cluster, sharing Redis, S3, and downstream service connections with the main QA deployment.

The Terraform infrastructure already exists (`terraform-infra/qa/ows-coda`) with workspace-based isolation. This system wraps it with a Jenkins pipeline (`Jenkinsfile.ephemeral`) that automates the full lifecycle -- from PR comment to running service -- with gating on test results, TTL-based expiry, capacity management, and nightly cleanup.

## Goals

1. **Comment-triggered lifecycle** -- Developers create/destroy environments by commenting `ephemeral deploy` / `ephemeral destroy` on PRs. No Jenkins UI, no shell access, no Terraform knowledge required.
2. **Quality gate** -- Environments are only created after lint, typecheck, unit tests, and functional tests pass on the merged code.
3. **Auto-teardown** -- All environments expire via configurable TTL (default 7 days). Nightly cron reaps expired environments. Phase 2 adds auto-destroy on PR merge.
4. **Multi-PR support (Phase 2)** -- Combine multiple PRs into a single environment for integration testing before any of them merge.
5. **DB automation** -- Eliminate manual SQL. Jenkins creates/drops databases and runs Prisma migrations automatically.
6. **Capacity management** -- Global cap of 10 concurrent environments with visibility into active environments via `ephemeral list`.
7. **Generic prefix** -- The `ephemeral` command prefix is service-agnostic, designed for adoption by other teams.

## Architecture

### Lifecycle

```mermaid
stateDiagram-v2
    [*] --> Parse: PR comment / manual trigger / cron

    Parse --> CheckCapacity: action = create
    Parse --> List: action = list
    Parse --> Cleanup: action = cleanup
    Parse --> TerraformDestroy: action = destroy

    CheckCapacity --> DuplicateCheck: capacity OK (< 10)
    CheckCapacity --> [*]: at capacity (>= 10), fail

    DuplicateCheck --> MergePR: env does not exist
    DuplicateCheck --> [*]: env exists, fail

    MergePR --> RunChecks: merge successful
    MergePR --> [*]: merge conflict, fail

    RunChecks --> BuildPush: tests pass
    RunChecks --> [*]: tests fail, no deploy

    BuildPush --> TerraformApply: image pushed to ECR
    TerraformApply --> DBSetup: Fargate service created
    DBSetup --> TagNotify: DB created + migrations run
    TagNotify --> [*]: URL posted on PR

    TerraformDestroy --> DBDrop: terraform destroy
    DBDrop --> Notify: database dropped
    Notify --> [*]: confirmation posted on PR

    List --> [*]: table posted on PR

    Cleanup --> [*]: expired envs destroyed
```

### Full lifecycle (create to cleanup)

```
create ──> [update*] ──> destroy
                           ^
                     TTL reaper (nightly cron, ~3am)
```

An environment can be updated any number of times (re-merge, re-test, re-deploy) before being destroyed either manually or by TTL expiry.

### What each ephemeral environment provisions

- **Fargate service** (`qa-ows-coda-{name}`) -- own ALB, security group, IAM role, DNS record
- **Database** (`coda_{name}` schema on shared RDS) -- own data, shared cluster
- **Security group rules** -- ephemeral Fargate to shared RDS + Redis ingress

### What is shared from main QA

- RDS cluster (Aurora MySQL)
- ElastiCache Redis
- S3 uploads bucket
- Secrets Manager entries
- Downstream service URLs (ows-grass, ows-abacus-account, ows-royalties, etc.)

### Component diagram

```mermaid
graph LR
    subgraph "GitHub"
        PR[PR Comment]
    end

    subgraph "Jenkins"
        JF[Jenkinsfile.ephemeral]
    end

    subgraph "AWS QA Account"
        subgraph "Shared Infrastructure"
            RDS[(Aurora MySQL)]
            Redis[(ElastiCache Redis)]
            S3[(S3 Bucket)]
            ECR[(ECR Registry)]
        end

        subgraph "Ephemeral env: pr-42"
            ALB1[ALB]
            FG1[Fargate Service<br/>512 CPU / 1GB RAM]
            DB1[(coda_pr_42 schema)]
        end

        subgraph "Ephemeral env: pr-56"
            ALB2[ALB]
            FG2[Fargate Service<br/>512 CPU / 1GB RAM]
            DB2[(coda_pr_56 schema)]
        end

        subgraph "Main QA"
            MFGALB[ALB]
            MFG[Fargate Service<br/>1024 CPU / 2GB RAM]
            MDB[(coda schema)]
        end
    end

    PR --> JF
    JF --> ECR
    JF -->|terraform apply| ALB1
    JF -->|terraform apply| ALB2
    FG1 --> RDS
    FG1 --> Redis
    FG1 --> S3
    FG2 --> RDS
    FG2 --> Redis
    FG2 --> S3
    MFG --> RDS
    MFG --> Redis
    MFG --> S3
    DB1 -.->|schema on| RDS
    DB2 -.->|schema on| RDS
    MDB -.->|schema on| RDS
```

### Ephemeral sizing vs. main QA

| Resource       | Main QA | Ephemeral |
| -------------- | ------- | --------- |
| Fargate CPU    | 1024    | 512       |
| Fargate memory | 2048 MB | 1024 MB   |
| Desired tasks  | 2       | 1         |
| Max tasks      | 4       | 2         |
| Min tasks      | 2       | 1         |

## Detailed Design

### PR comment triggers

| Comment                | Internal action    | Effect                                   |
| ---------------------- | ------------------ | ---------------------------------------- |
| `ephemeral deploy`     | `create`           | Create with default TTL (7 days)         |
| `ephemeral deploy 15d` | `create`           | Create with custom TTL                   |
| `ephemeral extend 7d`  | `extend` (Phase 2) | Set expiry to now + 7 days (no stacking) |
| `ephemeral destroy`    | `destroy`          | Tear down immediately                    |
| `ephemeral list`       | `list`             | Post table of all active environments    |

Jenkins `issueCommentTrigger` regex:

```groovy
issueCommentTrigger('.*ephemeral\\s+(deploy|destroy|list)(\\s+\\d+d)?.*')
```

TTL choices: 1d, 7d (default), 15d, 30d. The `extend` command sets expiry to `now + Nd`, not `current_expiry + Nd` -- multiple extends do not stack.

TTL is stored as an `ttl_expires_at` AWS tag (ISO 8601 timestamp) on the ECS service. The nightly cleanup job queries for expired tags.

### Jenkins pipeline structure

`Jenkinsfile.ephemeral` is a standalone declarative pipeline registered as a separate Jenkins job with its own build history. The main `Jenkinsfile` is not modified.

**Parameters:**

```groovy
ACTION:   choice ['create', 'destroy', 'list', 'cleanup']
PR_ID:    string (e.g. "42")
ENV_NAME: string (auto-derived as "pr-{id}" for single-PR)
TTL:      choice ['7d', '1d', '15d', '30d']
```

**Triggers:**

```groovy
issueCommentTrigger('.*ephemeral\\s+(deploy|destroy|list)(\\s+\\d+d)?.*')
cron('H 3 * * *')  // nightly cleanup at ~3am
```

**Pipeline options:** `disableConcurrentBuilds()` prevents parallel Terraform runs and race conditions on shared state.

**Stage layout:**

```
Parse               -- always: determine action, validate inputs
Check Capacity      -- create only: fail if >= 10 environments
Duplicate Check     -- create only: fail if workspace already exists
Merge PR            -- create/update: fetch PR branch, merge onto master
Run Checks          -- create/update: lint, typecheck, unit + functional tests
Build & Push        -- create/update: suiteAppBuild + dockerToEcr
Terraform           -- create: create-ephemeral.sh / destroy: destroy-ephemeral.sh
DB Setup            -- create: CREATE DATABASE + prisma migrate / destroy: DROP DATABASE
Tag & Notify        -- all: update AWS tags, post GitHub comments
List                -- list: query tagged services, post table
Cleanup             -- cleanup: reap expired TTLs
```

Each stage uses `when { expression { ... } }` to gate on the parsed action.

### Terraform resources

The existing `terraform-infra/qa/ows-coda` module uses workspace-based isolation (`var.instance`). Each ephemeral environment is a Terraform workspace that provisions:

- ECS Fargate service and task definition
- Application Load Balancer + target group + listener
- Security groups (Fargate to RDS, Fargate to Redis)
- IAM task role and execution role
- Route 53 DNS record (`qa-ows-coda-{name}.theorchard.io`)

Shell scripts in `terraform-infra/qa/ows-coda/scripts/`:

- **`create-ephemeral.sh {name} [--ttl-days N] [--pr-numbers IDS] [--created-by USER]`** -- creates workspace, runs `terraform apply`, tags ECS service with lifecycle metadata
- **`destroy-ephemeral.sh {name} [--db-host HOST --db-user USER --db-pass PASS]`** -- runs `terraform destroy`, deletes workspace, optionally drops database

### Database setup

Two MySQL users:

- **`coda_ephemeral`** -- dedicated DDL user for CREATE/DROP DATABASE. Password stored in Secrets Manager (`qa/ows-coda/CODA_DB_EPHEMERAL_PASS`). Wildcard grant on `coda_%` pattern -- cannot touch the main `coda` database.
- **`coda_svc`** -- existing application user for runtime queries and Prisma migrations. Needs a one-time wildcard grant update to access `coda_%` databases.

**Create flow:**

```bash
# 1. Create database (as coda_ephemeral)
mysql -h "${RDS_HOST}" -u coda_ephemeral -p"${CODA_DB_EPHEMERAL_PASS}" \
  -e "CREATE DATABASE IF NOT EXISTS \`coda_${ENV_NAME}\`;"

# 2. Run migrations (as coda_svc via the deploy Docker image)
docker run --rm \
  -e DATABASE_URL="mysql://coda_svc:${CODA_DB_PASS}@${RDS_HOST}:3306/coda_${ENV_NAME}" \
  ${ECR_IMAGE}:${COMMIT_SHA} \
  sh -c "pnpm --filter @coda/db prisma migrate deploy"
```

**Destroy flow:**

```bash
mysql -h "${RDS_HOST}" -u coda_ephemeral -p"${CODA_DB_EPHEMERAL_PASS}" \
  -e "DROP DATABASE IF EXISTS \`coda_${ENV_NAME}\`;"
```

Prisma migrations are idempotent -- each migration is tracked in a `_prisma_migrations` table. Updates re-run `prisma migrate deploy` to apply only new migrations.

### TTL management

- Default: 7 days from creation
- `ttl_expires_at` AWS tag (ISO 8601) on the ECS service
- `extend` resets to `now + Nd` (absolute, not relative to current expiry)
- Nightly cleanup cron at ~3am queries all `ephemeral=true` tagged services and destroys those past expiry
- Environments without `ttl_expires_at` (manually created) are skipped by cleanup and shown as "no TTL (manual)" in list output

### Cleanup cron (nightly)

```
1. List all ECS services tagged ephemeral=true + service_name=ows-coda
2. For each where now > ttl_expires_at: run full Destroy flow
3. Skip environments with no ttl_expires_at tag (manually created)
4. Skip environments that failed during create (no auto-reap of partial state)
5. Post expiry notification on associated PRs
6. Log summary; Phase 2 adds Slack notification to #coda-devs
```

### AWS resource tagging

| Tag              | Value              | Purpose                            |
| ---------------- | ------------------ | ---------------------------------- |
| `ephemeral`      | `true`             | Discovery (set by Terraform)       |
| `instance`       | e.g., `pr-42`      | Identification (set by Terraform)  |
| `service_name`   | `ows-coda`         | Service scoping (set by Terraform) |
| `ttl_expires_at` | ISO 8601 timestamp | TTL enforcement (set by Jenkins)   |
| `ttl_days`       | e.g., `7`          | Display (set by Jenkins)           |
| `pr_numbers`     | e.g., `42,56,78`   | PR association (set by Jenkins)    |
| `created_by`     | GitHub username    | Accountability (set by Jenkins)    |

### Capacity management

- Global cap: 10 concurrent ephemeral environments
- Enforcement: before `terraform apply`, query ECS services tagged `ephemeral=true` via the Resource Groups Tagging API. If count >= 10, fail with `ephemeral list` output showing all active environments.
- `ephemeral list` output example:

```
Active ephemeral environments (3/10):

| Name    | URL                                        | PRs       | Expires    | Created by |
|---------|--------------------------------------------|-----------|------------|------------|
| pr-42   | https://qa-ows-coda-pr-42.theorchard.io    | #42       | 2026-03-28 | mrojas     |
| pr-56   | https://qa-ows-coda-pr-56.theorchard.io    | #56       | 2026-03-25 | jdoe       |
| feat-ui | https://qa-ows-coda-feat-ui.theorchard.io  | #78, #91  | 2026-04-01 | mrojas     |
```

### Multi-PR merge (Phase 2)

For integration testing across multiple PRs:

1. Developer triggers via Jenkins job UI with `prs="42,56,78"` and `name="feat-auth"`
2. Jenkins fetches all PR branches and merges them sequentially onto `master`
3. On merge conflict: fail fast, identify conflicting PR + files
4. On success: run full test suite on merged result, build, deploy
5. URL comment posted on all constituent PRs

### Environment name validation

Names must match `^[a-z][a-z0-9-]{0,20}$` (lowercase alphanumeric with hyphens, 1-21 characters, starting with a letter). The 21-character limit avoids exceeding AWS resource name limits (e.g., 64-char IAM role names with the `qa-ows-coda-` prefix).

## Alternatives Explored

### Why Jenkins PR comments vs. GitHub Actions?

The existing CI/CD pipeline runs on Jenkins. The organization's infrastructure automation (Terraform, ECR pushes, Secrets Manager access, IAM role assumptions) is all wired through Jenkins shared libraries (`suiteAppBuild`, `dockerToEcr`, `withSecrets`, `withEcr`). Building this on GitHub Actions would require reimplementing all of that integration and maintaining a parallel CI system. Jenkins `issueCommentTrigger` provides the same comment-driven UX with zero new infrastructure.

### Why shared RDS vs. separate RDS instances?

Separate RDS instances would provide complete isolation but at roughly $60-100/month per instance (Aurora MySQL). With 10 concurrent environments, that would add $600-1,000/month. Shared RDS with per-environment schemas (`coda_{name}`) provides sufficient isolation for QA testing at zero incremental RDS cost. The `coda_ephemeral` user's wildcard grant pattern (`coda_%`) ensures ephemeral operations cannot touch the main `coda` database.

### Why Fargate vs. EKS?

The production and QA deployments already run on ECS Fargate. The Terraform modules, IAM roles, deployment scripts, and monitoring are all built for Fargate. EKS would require new infrastructure, new deployment tooling, and new operational knowledge. Fargate also provides simpler per-environment isolation (each environment is its own ECS service) without needing namespace management.

### Why workspace-based Terraform isolation vs. separate state files?

Terraform workspaces are already used by the existing `create-ephemeral.sh` / `destroy-ephemeral.sh` scripts. Each workspace gets its own state file, and the `var.instance` variable drives resource naming. This pattern is proven and requires no changes to the Terraform module itself.

### Why `disableConcurrentBuilds()` vs. Terraform state locking?

Terraform state locking (e.g., DynamoDB) would allow parallel creates of different environments. However, the Jenkins job also shares the workspace filesystem, ECR push context, and GitHub API calls. Serial execution via `disableConcurrentBuilds()` is simpler and sufficient given the expected volume (a few creates/destroys per day, not per minute).

## Cost Analysis

### Per-environment cost breakdown

| Resource                                | Unit cost      | Notes                                                           |
| --------------------------------------- | -------------- | --------------------------------------------------------------- |
| Application Load Balancer               | ~$16/month     | $0.0225/hour fixed + LCU charges (minimal for QA traffic)       |
| Fargate task (512 CPU, 1GB RAM, 1 task) | ~$3.50/month   | $0.04048/vCPU/hour + $0.004445/GB/hour, 0.5 vCPU                |
| RDS (shared)                            | $0 incremental | Shared Aurora cluster; ephemeral schemas add negligible storage |
| Redis (shared)                          | $0 incremental | Shared ElastiCache cluster                                      |
| S3 (shared)                             | $0 incremental | Shared bucket                                                   |
| DNS (Route 53)                          | ~$0.50/month   | Per hosted zone record                                          |
| **Per-environment total**               | **~$20/month** |                                                                 |

### Monthly cost scenarios

| Scenario            | Concurrent envs     | Monthly cost    | Assumptions                                                |
| ------------------- | ------------------- | --------------- | ---------------------------------------------------------- |
| **Worst case**      | 10 (cap full, 24/7) | **~$195/month** | All 10 slots occupied continuously for 30 days             |
| **Heavy usage**     | 5-7 average         | ~$120/month     | Active development sprint, environments live for full TTL  |
| **Realistic usage** | 3-5 average         | **~$100/month** | Most environments destroyed before TTL, typical team usage |
| **Light usage**     | 1-2 average         | ~$40/month      | Off-sprint, occasional PR previews                         |

The ALB is the dominant cost driver at ~$16/month per environment ($160/month at full capacity). Fargate compute is relatively cheap at the reduced ephemeral sizing.

### 10-environment cap rationale

The cap of 10 balances cost control with team productivity:

- **Cost ceiling:** $195/month worst case is well within QA infrastructure budgets
- **Team size:** The ows-coda team has ~5 active developers; 10 slots provides 2 environments per developer
- **ALB limits:** Each environment requires its own ALB; AWS account limits and VPC subnet capacity are factors
- **Practical usage:** With 7-day default TTLs and nightly cleanup, utilization rarely approaches the cap

The cap is a constant (`MAX_EPHEMERAL_ENVS = 10`) in the Jenkinsfile and can be adjusted without infrastructure changes.

### Cost mitigation strategies

- **TTL enforcement:** Default 7-day TTL prevents zombie environments. Nightly cleanup reaps expired environments automatically.
- **Reduced sizing:** Ephemeral environments run at half the CPU/memory of main QA (512/1024 vs. 1024/2048) with a single task instead of two.
- **Shared infrastructure:** RDS, Redis, and S3 are shared with main QA -- no incremental cost for these services.
- **Phase 2 auto-destroy:** Single-PR environments will be auto-destroyed on PR merge, reducing the average lifespan well below TTL.

## Performance Analysis

### Deployment time (create)

| Stage                                      | Estimated duration | Notes                                    |
| ------------------------------------------ | ------------------ | ---------------------------------------- |
| Parse + Capacity Check                     | ~10 seconds        | API calls to Resource Groups Tagging API |
| Merge PR onto master                       | ~15 seconds        | Git fetch + merge                        |
| Run Checks (lint + typecheck + unit)       | ~3-5 minutes       | Docker Compose, parallelized             |
| Run Checks (functional tests)              | ~2-3 minutes       | Docker Compose with MySQL                |
| Build & Push (suiteAppBuild + dockerToEcr) | ~3-5 minutes       | Client build + Docker build + ECR push   |
| Terraform apply                            | ~3-5 minutes       | ALB, ECS service, security groups, DNS   |
| DB Setup (CREATE + migrate)                | ~1-2 minutes       | Database creation + Prisma migration     |
| Tag & Notify                               | ~10 seconds        | AWS CLI tagging + GitHub API             |
| **Total create time**                      | **~12-20 minutes** |                                          |

### Teardown time (destroy)

| Stage                  | Estimated duration | Notes                                  |
| ---------------------- | ------------------ | -------------------------------------- |
| Terraform destroy      | ~3-5 minutes       | ALB, ECS service, security groups, DNS |
| DB drop                | ~5 seconds         | Single SQL statement                   |
| Notify                 | ~5 seconds         | GitHub API                             |
| **Total destroy time** | **~3-5 minutes**   |                                        |

### Update time (Phase 2)

Same as create minus capacity check, duplicate check, and DB creation. Terraform apply is faster for updates (incremental). Estimated ~8-15 minutes.

### Nightly cleanup time

Sequential destruction of expired environments. With `disableConcurrentBuilds()`, each environment takes ~3-5 minutes to destroy. Worst case (10 expired environments) takes ~30-50 minutes. Scheduled at 3am, so no impact on developer workflows.

## Scaling Characteristics

### Current design: 10 concurrent environments

- **Hard cap** enforced at the Jenkins pipeline level (`MAX_EPHEMERAL_ENVS = 10`)
- **Capacity check** runs before every create, querying ECS services via Resource Groups Tagging API
- **At limit behavior:** Create fails with a list of all active environments and a message to destroy unused ones or wait for TTL expiry

### What happens beyond 10

The cap is a pipeline-level constant, not an infrastructure limit. Raising it requires:

1. Changing `MAX_EPHEMERAL_ENVS` in `Jenkinsfile.ephemeral`
2. Verifying VPC subnet capacity (each ALB needs IPs in at least 2 AZs)
3. Verifying AWS service quotas (ECS services per cluster, ALBs per region, security groups per VPC)
4. Accepting the linear cost increase (~$20/month per additional environment)

### Serial execution constraint

`disableConcurrentBuilds()` means all ephemeral operations are serialized. At current volume (a few operations per day), this is not a bottleneck. If adoption grows significantly:

- Creates/destroys queue behind each other
- A create (~15 min) blocks all other operations for its duration
- Mitigation: Terraform state locking (DynamoDB) would allow parallel operations on different environments, but adds complexity

### Shared RDS capacity

All ephemeral databases share the QA Aurora cluster. At 10 concurrent environments with QA-level traffic:

- Connection pool: each Fargate task opens a connection pool. 10 tasks add ~10 connection pools to the shared cluster.
- Storage: ephemeral schemas are small (migrations + test data). Negligible impact on RDS storage.
- CPU/IOPS: QA traffic is light. 10 ephemeral environments with developer-level traffic do not meaningfully impact the shared cluster.

## Breakdown Points and Mitigations

### Orphaned environments

**Risk:** A create succeeds but the TTL tag is not applied (e.g., AWS CLI tagging fails after Terraform apply). The environment runs indefinitely with no automatic cleanup.

**Mitigation:** Tagging is done inside `create-ephemeral.sh` immediately after `terraform apply`. If tagging fails, the script logs a warning but the environment still runs. `ephemeral list` shows it as "no TTL (manual)". The team can manually destroy it. Phase 2 could add a "maximum age" safety net (destroy any ephemeral environment older than 30 days regardless of TTL).

### Failed creates (partial state)

**Risk:** Terraform apply fails midway, leaving partial resources (e.g., ALB created but no ECS service). Subsequent create attempts fail ("workspace already exists").

**Mitigation:** The Terraform workspace is preserved on failure (not deleted) so `ephemeral destroy` can find and remove partial resources. The failure comment directs the user to run `ephemeral destroy`. Nightly cleanup does NOT auto-reap failed creates to avoid destroying an environment someone is actively debugging.

### RDS unreachable from Jenkins

**Risk:** Terraform apply succeeds (Fargate service created) but Jenkins cannot reach the shared RDS to create the database. The Fargate service starts but fails health checks.

**Mitigation:** DB Setup failure is non-fatal to the pipeline. The failure comment includes manual SQL commands (existing `create-ephemeral.sh` behavior). `ephemeral update` (Phase 2) re-runs migrations. The user can also run `ephemeral destroy` and retry.

### TTL expiry race condition

**Risk:** A developer is actively using an environment when the nightly cleanup runs and destroys it because the TTL expired.

**Mitigation:** Default TTL is 7 days (generous for PR review). Developers can run `ephemeral extend 7d` to reset the clock. Cleanup runs at 3am when active use is unlikely. The destroy posts a comment on the PR notifying the developer.

### Duplicate create attempts

**Risk:** A developer comments `ephemeral deploy` twice, or two developers comment on the same PR.

**Mitigation:** The Duplicate Check stage detects existing Terraform workspaces and ECS services with the same name. It fails with a message: "Environment already exists -- use `ephemeral destroy` first." `disableConcurrentBuilds()` prevents race conditions between two simultaneous create attempts.

### Concurrent Terraform operations

**Risk:** Two pipeline runs modify Terraform state simultaneously, causing state corruption.

**Mitigation:** `disableConcurrentBuilds()` ensures only one pipeline run at a time. Builds queue. This also prevents Terraform state lock conflicts.

### PR force-pushed after deploy

**Risk:** A developer pushes new commits to a PR after deploying. The ephemeral environment runs stale code.

**Mitigation:** Not auto-detected in Phase 1. The deploy comment includes the commit SHA and a note to run `ephemeral update` after pushing. Phase 2 adds the `update` action.

### Merge conflicts in multi-PR environments

**Risk (Phase 2):** Multiple PR branches conflict when merged onto master.

**Mitigation:** The merge step fails fast, identifying the conflicting PR and files. A comment is posted on the requesting PR with the conflict details. No partial deploy occurs.

## Decision Log

| Decision                                                    | Rationale                                                                                                                                                         | Date       |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| Jenkins PR comments over GitHub Actions                     | Existing CI/CD is Jenkins; shared libraries (suiteAppBuild, dockerToEcr, withSecrets) already handle Terraform, ECR, Secrets Manager                              | 2026-03-22 |
| Shared RDS with per-env schemas over separate RDS instances | $0 incremental cost vs. $60-100/month per instance; sufficient isolation for QA                                                                                   | 2026-03-22 |
| Fargate over EKS                                            | Matches existing production/QA deployment model; no new infrastructure or operational knowledge                                                                   | 2026-03-22 |
| Terraform workspaces over separate state files              | Already used by existing scripts; `var.instance` drives naming                                                                                                    | 2026-03-22 |
| `disableConcurrentBuilds()` over Terraform state locking    | Simpler; sufficient for expected volume; prevents filesystem and API races too                                                                                    | 2026-03-22 |
| 10-environment cap                                          | Balances cost (~$195/mo worst case) with team capacity (~5 devs, 2 envs each)                                                                                     | 2026-03-22 |
| Dedicated `coda_ephemeral` DB user over reusing `coda_svc`  | Separation of concerns; DDL operations (CREATE/DROP DATABASE) are distinct from application operations; wildcard grant `coda_%` cannot touch main `coda` database | 2026-03-22 |
| TTL as AWS tag vs. external database/config                 | No additional infrastructure; tags are queryable via Resource Groups Tagging API; visible in AWS console                                                          | 2026-03-22 |
| Nightly cleanup skips failed creates                        | Avoids destroying environments someone is actively debugging; manual `ephemeral destroy` required                                                                 | 2026-03-22 |
| Half-size Fargate (512 CPU / 1GB RAM) for ephemeral         | Sufficient for QA-level traffic; reduces cost; matches the "preview, don't load test" use case                                                                    | 2026-03-22 |

## Dependencies

| Dependency                                     | Type               | Notes                                                                           |
| ---------------------------------------------- | ------------------ | ------------------------------------------------------------------------------- |
| **Terraform** (1.14+)                          | Infrastructure     | Workspace-based isolation in `terraform-infra/qa/ows-coda`                      |
| **Jenkins**                                    | CI/CD              | Declarative pipeline with `issueCommentTrigger`, cron, shared libraries         |
| **Jenkins shared libraries**                   | CI/CD              | `suiteAppBuild`, `dockerToEcr`, `withSecrets`, `withEcr`                        |
| **Shared QA RDS** (Aurora MySQL)               | Database           | Hosts all ephemeral schemas; must be reachable from Jenkins for DB setup        |
| **ECR**                                        | Container registry | Stores Docker images tagged with commit SHA                                     |
| **ECS Fargate**                                | Compute            | Runs the application; existing task definitions parameterized by `var.instance` |
| **AWS Secrets Manager**                        | Secrets            | Stores `CODA_DB_EPHEMERAL_PASS` and existing secrets                            |
| **GitHub API**                                 | Notifications      | PR comment posting via API token (`github_api_token` credential)                |
| **Route 53**                                   | DNS                | `qa-ows-coda-{name}.theorchard.io` records                                      |
| **`prod-jenkins-aws-pipeline-agent` IAM role** | Auth               | Must have permissions for ECS, EC2, IAM, ELB in QA account                      |
| **terraform-infra repo**                       | Cross-repo         | Shell scripts and Terraform module; changes must be merged first                |

## Testing Strategy

### Pipeline verification (post-deployment)

| Test                                 | Method                                        | Validates                                                           |
| ------------------------------------ | --------------------------------------------- | ------------------------------------------------------------------- |
| V1: `ephemeral list` on empty state  | PR comment                                    | Pipeline triggers, AWS API query, "No active environments" response |
| V2: `ephemeral deploy` end-to-end    | PR comment                                    | Full create flow: merge, test, build, Terraform, DB, notify         |
| V3: Environment health check         | Browser / curl                                | Fargate service running, app responds, DB connected                 |
| V4: `ephemeral list` with active env | PR comment                                    | Environment appears in table with correct metadata                  |
| V5: `ephemeral destroy`              | PR comment                                    | Full teardown: Terraform destroy, DB drop, PR notification          |
| V6: `ephemeral list` after destroy   | PR comment                                    | Environment no longer listed                                        |
| V7: Nightly cleanup (empty)          | Manual cron trigger                           | No errors on empty environment list                                 |
| V8: TTL expiry reaping               | Set `ttl_expires_at` to past, trigger cleanup | Expired environment destroyed, PR notified                          |

### Edge case testing

| Test                 | Method                                        | Validates                                            |
| -------------------- | --------------------------------------------- | ---------------------------------------------------- |
| Duplicate create     | Comment `ephemeral deploy` twice              | "Environment already exists" error, no partial state |
| Destroy non-existent | Comment `ephemeral destroy` on PR with no env | "No ephemeral environment found" error               |
| At capacity          | Create 10 environments, attempt 11th          | "At capacity" error with list output                 |
| Merge conflict       | Deploy PR with known conflict against master  | Clear error identifying conflicting files            |
| Test failure         | Deploy PR with failing tests                  | "Checks failed" comment, no deploy                   |

### Ongoing validation

- Nightly cleanup cron runs automatically and logs results
- Jenkins build history provides audit trail of all ephemeral operations
- AWS resource tags provide discoverability via AWS console and CLI

## Rollout Plan

### Phase 1: MVP (current scope)

**Deliverables:**

- `Jenkinsfile.ephemeral` with `create`, `destroy`, `list`, `cleanup` actions
- Comment triggers: `ephemeral deploy`, `ephemeral destroy`, `ephemeral list`
- TTL tagging + nightly cleanup (TTL expiry only)
- Capacity check (max 10)
- GitHub comment notifications on PRs
- Single-PR only
- DB automation: CREATE/DROP database + Prisma migrations

**Manual one-time setup required:**

1. Create Jenkins pipeline job (Type: Pipeline, script path: `Jenkinsfile.ephemeral`, enable comment trigger + cron)
2. Create `coda_ephemeral` MySQL user on shared QA RDS cluster:
   ```sql
   CREATE USER 'coda_ephemeral'@'%' IDENTIFIED BY '<generated-password>';
   GRANT ALL PRIVILEGES ON `coda\_%`.* TO 'coda_ephemeral'@'%';
   FLUSH PRIVILEGES;
   ```
3. Update `coda_svc` wildcard grants:
   ```sql
   GRANT ALL PRIVILEGES ON `coda\_%`.* TO 'coda_svc'@'%';
   FLUSH PRIVILEGES;
   ```
4. Store `coda_ephemeral` password in Secrets Manager (`qa/ows-coda/CODA_DB_EPHEMERAL_PASS`)
5. PR and merge terraform-infra changes (add secret to Terraform module, update shell scripts)
6. Verify `prod-jenkins-aws-pipeline-agent` IAM role has sufficient permissions for QA ECS/EC2/IAM/ELB operations

**Cross-repo changes:**

| Repo              | File                                       | Change                                                                                    |
| ----------------- | ------------------------------------------ | ----------------------------------------------------------------------------------------- |
| `terraform-infra` | `qa/ows-coda/main.tf`                      | Add `CODA_DB_EPHEMERAL_PASS` to secrets for_each                                          |
| `terraform-infra` | `qa/ows-coda/scripts/create-ephemeral.sh`  | Add `--ttl-days`, `--pr-numbers`, `--created-by` flags and AWS tagging                    |
| `terraform-infra` | `qa/ows-coda/scripts/destroy-ephemeral.sh` | Add `--db-host/user/pass` flags and `-auto-approve`                                       |
| `ows-coda`        | `Jenkinsfile.ephemeral`                    | Full pipeline (Parse, Capacity, Merge, Test, Build, Terraform, DB, Notify, List, Cleanup) |

### Phase 2: Multi-PR and polish

- Multi-PR support: merge multiple branches, validate, deploy via Jenkins job UI
- `ephemeral update` action (rebuild + redeploy to existing environment)
- `ephemeral extend` action
- Auto-destroy on single-PR merge (nightly cleanup checks if associated PR has been merged)
- Slack notifications to #coda-devs

### Phase 3: Adoption

- Extract reusable patterns for other teams
- Document the `var.instance` / `is_ephemeral` Terraform pattern as a template
- Consider a shared Jenkins library step (`ephemeralDeploy`) if multiple repos adopt

## Open Questions

1. **IAM role permissions:** Does `prod-jenkins-aws-pipeline-agent` have sufficient permissions for `terraform apply/destroy` in QA (ECS, EC2 security groups, IAM, ELB)? If not, a scoped `qa-ows-coda-ephemeral-terraform-role` needs to be created. This must be determined during Phase 1 setup.

2. **VPC subnet capacity:** How many ALBs can the QA VPC subnets support? Each ephemeral ALB requires IPs in at least 2 AZs. At 10 environments, this is 20+ IPs dedicated to ephemeral ALBs.

3. **Jenkins agent disk space:** Docker builds and ECR pushes accumulate layers. Is there sufficient disk space on the Jenkins agent, and is `cleanWs()` sufficient for cleanup?

4. **Prisma migration compatibility:** If a PR adds a migration that depends on data that only exists in the main QA database, the ephemeral environment will have an empty database post-migration. Should there be a seed data step?

5. **Downstream service compatibility:** Ephemeral environments share downstream QA services (ows-grass, ows-abacus-account). If an ephemeral PR changes API contracts with these services, the ephemeral environment may fail. Is this acceptable, or should there be a way to point at different downstream URLs?

6. **SSL certificates:** Do the ephemeral DNS records (`qa-ows-coda-{name}.theorchard.io`) need individual certificates, or is there a wildcard certificate (`*.theorchard.io`) on the ALBs?

7. **Cost monitoring:** Should there be an AWS Budget alert for ephemeral environment spend, or is the 10-environment cap sufficient cost control?