# Jenkins Playwright Pipeline Dashboard — Design Spec

> **Status: IMPLEMENTED** — See implementation plan for details.

## Overview

A local-first dashboard that monitors Jenkins pipelines containing Playwright E2E tests across an organization. It polls Jenkins for pipeline/stage statuses and uses a three-color system to surface E2E test health at a glance:

- **Green** — all stages passed
- **Red** — failed on an E2E test stage (extracts URLs from console output)
- **Yellow** — failed on a non-E2E stage, or E2E stage failed due to upstream cascade

## Context

- **Users:** Single user (personal tool)
- **Scale:** 20+ pipelines with high activity
- **Auth:** Jenkins API token (username + token)
- **No webhooks** — local-first, all data pulled via polling
- **URL extraction** from Playwright stage console output is in scope for v1

## Architecture

```
┌─────────────────┐     ┌──────────────────┐       ┌─────────────┐
│  Jenkins API     │◄────│  Poller Service   │────►│  SQLite DB  │
│  (pipelines,     │     │  (Node.js)        │     │  (Drizzle)  │
│   stages, logs)  │     │  - polls on cron  │     │             │
└─────────────────┘     │  - parses logs    │      └──────┬──────┘
                         └──────────────────┘             │
                         ┌──────────────────┐             │
                         │  REST API Server  │◄───────────┘
                         │  (Express)        │
                         └────────┬─────────┘
                                  │
                         ┌────────▼─────────┐
                         │  React Frontend   │
                         │  (Vite + React)   │
                         └──────────────────┘
```

### Components

1. **Poller Service** — Runs on a configurable interval (default 30s). For each tracked pipeline, fetches the latest run and its stages via Jenkins REST API. When a run fails on the Playwright stage, fetches console output and extracts URLs.

2. **SQLite Database (Drizzle ORM)** — Stores pipeline runs, stage statuses, and extracted URLs. Drizzle migrations manage schema changes.

3. **REST API (Express)** — Serves current pipeline statuses to the frontend. See API Endpoints section below.

4. **React Frontend (Vite)** — Single-page dashboard with auto-refreshing card grid.

5. **Pipeline Discovery (separate script)** — Uses `gh` CLI to scan org repos for Jenkinsfiles containing `playwrightTest` calls.

## Data Model

### pipelines
| Column | Type | Description |
|--------|------|-------------|
| id | integer PK | Auto-increment |
| name | text | Pipeline name (e.g. "repo-name/branch") |
| jenkins_url | text UNIQUE | Full Jenkins job URL |
| last_polled_at | integer | Timestamp of last poll |
| created_at | integer | Timestamp of creation |

### runs
| Column | Type | Description |
|--------|------|-------------|
| id | integer PK | Auto-increment |
| pipeline_id | integer FK | References pipelines.id |
| build_number | integer | Jenkins build number |
| status | text | "green", "red", "yellow", or "building" |
| started_at | integer | Build start timestamp |
| duration_ms | integer | Build duration |
| fetched_at | integer | When we fetched this data |

**Constraints:** Unique on `(pipeline_id, build_number)`.

**Status derivation:**
- `green` — all stages passed
- `red` — an E2E stage (identified by `is_playwright` flag) failed, AND no non-E2E stage failed before it. A pipeline can have multiple E2E stages; if any one fails genuinely, the run is red.
- `yellow` — a non-E2E stage failed, OR an E2E stage's failure was caused by an upstream non-E2E stage failure (cascaded failure detection)
- `building` — build is still in progress (detected from run-level `IN_PROGRESS` status from `wfapi/runs`, not just stages, since Jenkins only returns stages that have started)

**Cascaded failure detection:** When Jenkins marks a non-E2E stage as FAILED before the E2E stage, all subsequent stages (including E2E) are also marked FAILED even though they never ran. The poller detects this by checking if any non-E2E stage failed before the first E2E failure, and treats such cases as yellow.

**Jenkins status values:** Jenkins uses `FAILED` (not `FAILURE`) for stage statuses. The poller accepts both. Stage status is stored as free text (not an enum) to handle any Jenkins status including `UNSTABLE`.

**In-progress build handling:** When the latest build is in progress, the API's `latestRun` returns the most recent **completed** run's data (status, stages, URLs) with an additional `isBuilding: true` flag. The frontend renders the previous result's color with a blue pulsing border and "BUILDING" label. If no completed run exists yet, `latestRun` is `null` with `isBuilding: true`.

### stages
| Column | Type | Description |
|--------|------|-------------|
| id | integer PK | Auto-increment |
| run_id | integer FK | References runs.id |
| name | text | Stage name |
| status | text | Any Jenkins status (SUCCESS, FAILED, IN_PROGRESS, NOT_EXECUTED, UNSTABLE, etc.) |
| duration_ms | integer | Stage duration |
| is_playwright | integer | 1 if this is an E2E test stage (a run can have multiple) |

### repo_scans (discovery cache)
| Column | Type | Description |
|--------|------|-------------|
| id | integer PK | Auto-increment |
| repo_name | text UNIQUE | GitHub repo name |
| has_playwright_test | integer | 1 if Jenkinsfile contains playwrightTest |
| scanned_at | integer | Timestamp of last scan |

### extracted_urls
| Column | Type | Description |
|--------|------|-------------|
| id | integer PK | Auto-increment |
| run_id | integer FK | References runs.id |
| stage_id | integer FK | References stages.id |
| url | text | Extracted URL from console output |
| label | text | Inferred context (e.g. "report", "trace") |
| created_at | integer | Timestamp |

## Jenkins API Endpoints Used

The poller uses the Jenkins Pipeline REST API (workflow API):

1. **Get latest builds:** `GET /job/{org}/job/{repo}/job/{branch}/wfapi/runs` — returns recent runs with status and stage info. The run-level `status` field (`IN_PROGRESS`, `SUCCESS`, etc.) is used to detect in-progress builds.
2. **Get stages for a run:** `GET /job/{org}/job/{repo}/job/{branch}/{buildNumber}/wfapi/describe` — returns stages with names, statuses, and durations. Note: only stages that have started are included.
3. **Get console output:** `GET /job/{org}/job/{repo}/job/{branch}/{buildNumber}/consoleText` — plain text console log (only fetched for red runs)

Jenkins jobs are organized under `/job/{org}/job/{repo}/job/{branch}` (org-level folder structure).

## Polling Strategy

- **Staggered polling** — Pipelines are polled in round-robin fashion, evenly spaced across the interval (e.g., 25 pipelines / 30s = ~1 request per 1.2s)
- **Smart skip** — If build number hasn't changed since last poll and Jenkins reports the run as complete, skip fetching stages/logs. Re-polls if Jenkins reports `IN_PROGRESS`.
- **Last polled timestamp** — Updated on every poll cycle (not just when new builds are found)
- **Lazy console fetch** — Only fetch console output when a run fails on the Playwright stage
- **Configurable interval** — Default 30s, set in config
- **Error handling** — On Jenkins errors (unreachable, 5xx, rate limit), log the error and skip to the next pipeline. After 3 consecutive failures for a pipeline, back off exponentially (double the wait, max 5 minutes). Connection status shown in the UI header.

## E2E Stage Identification

Two separate patterns are used:

- **`stagePattern`** (default: `"E2E"`) — Matched as a case-insensitive **substring** against each stage's `name` from the Jenkins Pipeline Stages API. For example, `"E2E"` matches stages named `"E2E Tests"`, `"E2E Permissions Tests"`, `"E2E Prod Tests"`. The `is_playwright` flag is set on matching stages during poll ingestion.
- **`jenkinsfilePattern`** (default: `"playwrightTest"`) — Used by the discovery script to identify repos that contain Playwright E2E tests by scanning Jenkinsfile content for this pattern.

## Data Retention

Keep the latest 50 runs per pipeline. On each poll cycle, prune older runs (and their associated stages and extracted_urls) to keep the database from growing unbounded.

## API Endpoints

### `GET /api/pipelines`
Returns all tracked pipelines with their latest run status.

```json
{
  "pipelines": [
    {
      "id": 1,
      "name": "auth-service/main",
      "jenkinsUrl": "https://jenkins.example.com/job/auth-service/job/main",
      "latestRun": {
        "buildNumber": 142,
        "status": "red",
        "failedStages": ["playwrightTest QA", "playwrightTest Staging"],
        "startedAt": 1710600000,
        "durationMs": 120000,
        "extractedUrls": [
          { "url": "https://playwright.theorchard.io/api/bucket/prod/report/...", "label": "Playwright Report Link", "stageName": "playwrightTest QA" }
        ],
        "isBuilding": false
      }
    }
  ],
  "lastPolledAt": 1710600030,
  "healthy": true
}
```

### `GET /api/pipelines/:id/runs`
Returns recent runs for a specific pipeline (for future history view).

```json
{
  "runs": [
    {
      "buildNumber": 142,
      "status": "red",
      "startedAt": 1710600000,
      "durationMs": 120000,
      "stages": [
        { "name": "Build", "status": "SUCCESS", "durationMs": 30000 },
        { "name": "playwrightTest", "status": "FAILURE", "durationMs": 90000, "isPlaywright": true }
      ],
      "extractedUrls": [{ "url": "https://...", "label": "report" }]
    }
  ]
}
```

## Pipeline Discovery

- Separate script run via `pnpm discover` (or system cron)
- Uses `gh repo list <org>` to enumerate repos
- Checks each repo for a Jenkinsfile via `gh api`
- Parses Jenkinsfile content for `playwrightTest` calls (configured via `jenkinsfilePattern`)
- Upserts matching pipelines into the database
- For multibranch pipelines: queries Jenkins for available branches via `/job/{org}/job/{repo}/api/json`. Only adds `master` branch pipelines (PR/feature branches are skipped).
- **Discovery caching** — Scan results are cached in the `repo_scans` SQLite table with a 24-hour TTL. On subsequent runs, repos already scanned within the TTL are skipped (no GitHub API call). Repos with `playwrightTest` still check Jenkins for new branches. Use `pnpm discover -- --force` to bypass the cache.
- **Environment** — Loads `.env` file via dotenv for Jenkins credentials
- Decoupled from the main server process

## URL Extraction

- Triggered only for red (Playwright-failed) runs
- Fetches console log via Jenkins API: `GET /job/{name}/{build}/consoleText`
- Regex-based extraction for URLs in output
- Extracts URLs that follow known label patterns in console output, e.g. lines like:
  `Playwright Report Link: https://playwright.theorchard.io/api/bucket/prod/report/...`
- Default extraction: matches lines containing a label followed by a URL (e.g. `Label: https://...`)
- The label text before the URL is stored in `extracted_urls.label`
- Additional URL patterns can be added via `urlPatterns` config array (regexes)
- Results stored in `extracted_urls` table

## Frontend

### Stack
- Vite + React + TypeScript + Tailwind CSS

### Dashboard (single page)
- **Header** — App name, last poll timestamp, connection status
- **Filter bar** — Tabs: All / Failing (red) / Other Issues (yellow) / Passing (green) with counts
- **Card grid** — Responsive grid, sorted: red → yellow → green
- **Each card shows:**
  - Pipeline name (repo/branch)
  - Color-coded status dot + border
  - Failed stage name (if any)
  - Extracted URLs as clickable links (red cards only)
  - Build number + relative time
- **Auto-refresh** — Polls REST API every 10s
- **Click card** — Opens Jenkins build page in new tab
- **Dark theme** — Dark background with color-coded cards

## Configuration

```json
{
  "jenkins": {
    "url": "https://jenkins.example.com",
    "user": "your-username",
    "token": "your-api-token"
  },
  "polling": {
    "intervalSeconds": 30
  },
  "playwright": {
    "stagePattern": "E2E",
    "jenkinsfilePattern": "playwrightTest",
    "urlPatterns": ["Playwright Report Link:\\s*(https?://\\S+)"]
  },
  "github": {
    "org": "your-org"
  }
}
```

Jenkins credentials can also be set via environment variables: `JENKINS_URL`, `JENKINS_USER`, `JENKINS_TOKEN`. A `.env` file in the project root is loaded automatically via dotenv.

## Project Structure

```
dash/
├── package.json
├── drizzle.config.ts
├── config.json
├── server/
│   ├── index.ts                 # Express server + starts poller
│   ├── db/
│   │   ├── schema.ts            # Drizzle schema
│   │   └── migrations/          # Drizzle migrations
│   ├── poller/
│   │   ├── index.ts             # Poll scheduler (setInterval, staggered)
│   │   ├── jenkins-client.ts    # Jenkins API calls
│   │   └── url-extractor.ts     # URL extraction from console logs
│   ├── routes/
│   │   └── pipelines.ts         # REST API routes
│   └── discover.ts              # Pipeline discovery (uses gh CLI)
├── client/
│   ├── index.html
│   ├── src/
│   │   ├── App.tsx
│   │   ├── components/
│   │   │   ├── Dashboard.tsx     # Main grid + filters
│   │   │   ├── PipelineCard.tsx  # Individual pipeline card
│   │   │   └── FilterBar.tsx     # Status filter tabs
│   │   ├── hooks/
│   │   │   └── usePipelines.ts   # API polling hook
│   │   └── types.ts
│   └── vite.config.ts
└── data/
    └── dash.db                  # SQLite database file
```

### Package.json Scripts
- `pnpm dev` — uses `concurrently` to start both server and Vite dev server in parallel
- `pnpm dev:server` — server only (Express on port 3001)
- `pnpm dev:client` — Vite dev server only (port 5173, proxies `/api` to port 3001)
- `pnpm discover` — run pipeline discovery (`--force` to bypass cache)
- `pnpm db:generate` — generate Drizzle migrations
- `pnpm db:migrate` — run Drizzle migrations
- `pnpm test` — run all tests (Vitest)
- `pnpm test:watch` — run tests in watch mode